Google Cloud “GCP” native NixOS images build
How to build NixOS Google Cloud VM images using Google Cloud VM
·
Join the DZone community and get the full member experience.
Join For Free
I am using NixOs as a Development environment that is running on Google Cloud VM.
NixOs official Web site does not provide an official GCP image. On the NixOs Wiki page, you can find an article how you can build your GCP NixOs Image that didn’t work well in the past but now seems all build image issue was resolved and now works well. NixOS Wiki page article also provides two GCP storage that you can find outdated GCP VM images that seem not actively maintained anymore.
- nixos-images > gs://nixos-images
- nixos-cloud-images > gs://nixos-images
In the below "HowTo," we will cover how you can build your images in GCP by using Google Cloud VM and store the images in your Google storage bucket.
Preparing the Google Cloud Environment.
Before we can start, you will need to have a Google Cloud Account.
We will create a dedicated GCP Project with VM that we will use to build updated NixOs images from the latest nixpkgs builds.
Prerequisites
Before starting
In the 1st step, you will need to install Google Cloud SDK Depending on the OS or Linux Distribution, and please follow installation instructions.
Gcloud Authentication
Once installation is complete, you will need to authenticate CLI to be able to access your google cloud resources.
~> gcloud auth login
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32542940657.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=skHTaEGrhJSYIzDIwt4phrhSzUm97t&prompt=consent&access_type=offline&code_challenge=EcgLC0aZhpefFkL7k6ep-8lvJ1Og8NgCvs9VllOE5lQ&code_challenge_method=S256
Enter verification code:
GCP Project Creation
Once authenticated, you should be able to access all cloud resources in your google cloud account.
If you are using your account for the first time, Google will create a default Project for your account. We will ignore the default project and will make a dedicated project to build NioxOs images.
GCP New Project Creation
We will create a dedicated GCP project. I use the name for the project mudrii-nixos.
~> gcloud projects create mudrii-nixos
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/mudrii-nixos].
Waiting for [operations/cp.8533728690478532386] to finish...done. Enabling service [cloudapis.googleapis.com] on project [mudrii-nixos]...
Operation "operations/acf.p2-237810657129-fa1f0fa7-3015-46ff-ab52-f92d7bd3f8df" finished successfully.
Get details on the created project:
~> gcloud projects describe mudrii-nixos
createTime: '2021-09-06T08:57:16.001Z'
lifecycleState: ACTIVE
name: mudrii-nixos
projectId: mudrii-nixos
projectNumber: '111111111111'
~> gcloud projects list
PROJECT_ID NAME PROJECT_NUMBER
mudrii-nixos mudrii-nixos 111111111111
Configuring Gcloud CLI
Once a new project is created, it is a good practice to add the newly created project as default for Gcloud CLI. Every command executed with Gcloud of Gsutils will perform on the newly created project.
~> gcloud config set project mudrii-nixos
Updated property [core/project].
Configuring Project Billing
One more step is needed to make the GCP project usable; we will need to add billing to the project.
Listing the Available Billing Accounts
~> gcloud alpha billing accounts list
ACCOUNT_ID NAME OPEN MASTER_ACCOUNT_ID
ZZZZZ-ZZZZZZZ-ZZZZZZ MyBill True
Associating the Billing Account With the Project
~> gcloud alpha billing accounts projects link mudrii-nixos --billing-account=ZZZZZ-ZZZZZZZ-ZZZZZZ
billingAccountName: billingAccounts/ZZZZZ-ZZZZZZZ-ZZZZZZ
billingEnabled: true
name: projects/mudrii-nixos/billingInfo
projectId: mudrii-nixos
Verifying the Billing Account Association
~> gcloud alpha billing accounts projects list --billing-account=ZZZZZZ-ZZZZZZ-ZZZZZZ
PROJECT_ID BILLING_ACCOUNT_ID BILLING_ENABLED
mudrii-nixos ZZZZZ-ZZZZZZZ-ZZZZZZ True
Creating and Configuring Google Storage Bucket
The next step is to create a Google storage bucket to store generated nixos GCP VM images.
~> gsutil mb gs://nixos-images-gcp gsutil mb gs://nixos-images-gcp~> gsutil du -s -h gs://nixos-images-gcp
I decided to grant read access to external users to the generated images. If you do not intend to share generated images, you can skip the below step.
~> gsutil iam ch allUsers:objectViewer gs://nixos-images-gcp
Creating and Configuring GCP Networking
By default, Google will create a global VPC and subnets for every region and firewall rules once a project is created.
To make it clean, I decided to create a separate network stack with a subnet and firewall to control the resources better.
Before creating a network, a good practice to remove all automated default network VPC, subnets, and firewalls.
Listing the Available Network Resource
List VPC:
~> gcloud compute networks list
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
default AUTO REGIONAL
Listing Subnets
~> gcloud compute networks subnets list
NAME REGION NETWORK RANGE
default us-central1 default 10.128.0.0/20
default europe-west1 default 10.132.0.0/20
default us-west1 default 10.138.0.0/20
default asia-east1 default 10.140.0.0/20
default us-east1 default 10.142.0.0/20
default asia-northeast1 default 10.146.0.0/20
default asia-southeast1 default 10.148.0.0/20
default us-east4 default 10.150.0.0/20
default australia-southeast1 default 10.152.0.0/20
default europe-west2 default 10.154.0.0/20
default europe-west3 default 10.156.0.0/20
default southamerica-east1 default 10.158.0.0/20
default asia-south1 default 10.160.0.0/20
default northamerica-northeast1 default 10.162.0.0/20
default europe-west4 default 10.164.0.0/20
default europe-north1 default 10.166.0.0/20
default us-west2 default 10.168.0.0/20
default asia-east2 default 10.170.0.0/20
default europe-west6 default 10.172.0.0/20
default asia-northeast2 default 10.174.0.0/20
default asia-northeast3 default 10.178.0.0/20
default us-west3 default 10.180.0.0/20
default us-west4 default 10.182.0.0/20
default asia-southeast2 default 10.184.0.0/20
default europe-central2 default 10.186.0.0/20
default northamerica-northeast2 default 10.188.0.0/20
default asia-south2 default 10.190.0.0/20
default australia-southeast2 default 10.192.0.0/20
Listing Firewalls
~> gcloud compute firewall-rules list
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
default-allow-icmp default INGRESS 65534 icmp False
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp False
default-allow-rdp default INGRESS 65534 tcp:3389 False
default-allow-ssh default INGRESS 65534 tcp:22 False
To show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.
Removing the Default Created Network Resource
Before removing VPC, we need to remove the firewall first, and once all firewall rules have been removed, we can remove the default VPC.
Removing the Default Firewall
~> gcloud compute firewall-rules delete default-allow-internal --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-internal].~> gcloud compute firewall-rules delete default-allow-icmp --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-icmp].~> gcloud compute firewall-rules delete default-allow-rdp --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-rdp].~> gcloud compute firewall-rules delete default-allow-ssh --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-ssh].
Removing Default VPC
~> gcloud compute networks delete default --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/networks/default].
Creating Networking Resources
We cleared all default networks stuck, and now we can create a new network stuck.
Creating a VPC:
Now, we are ready to create a custom VPC.
~> gcloud compute networks create vpc-nixos --subnet-mode=custom
Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/networks/vpc-nixos].
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
vpc-nixos CUSTOM REGIONAL
Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:
Creating a Custom Subnet
In the below example, I use a narrow IP range in Asia-southeast1 “Singapore” region. As I will run a single instance that will generate nixos images, I do not intend to run anything else.
~> gcloud compute networks subnets create subnet-sg --network=vpc-nixos --range=192.168.1.0/24 --region asia-southeast1
Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/regions/asia-southeast1/subnetworks/subnet-sg].
NAME REGION NETWORK RANGE
subnet-sg asia-southeast1 vpc-nixos 192.168.1.0/24
Creating a Firewall Rule to Allow SSH
In order to connect to remote VM over ssh, we need to open a firewall on port 22.
~> gcloud compute --project=mudrii-nixos firewall-rules create allow-ssh --direction=INGRESS --priority=1000 --network=vpc-nixos --action=ALLOW --rules=tcp:22 --target-tags=allow-ssh
Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/allow-ssh].
Creating firewall...done.
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
allow-ssh vpc-nixos INGRESS 1000 tcp:22 False
Adding an Image to Your VM Images Repository to be Able to Create NixOs VM in GCP
To create VM with NixOS installed, we need to have a VM image in our GCP images repositories.
In the below example, I am using one of the images I created previously locate in gs://nixos-images-gcp/
Note: make sure you specify your project ID --project=mudrii-nixos
~> gcloud compute images create nixos-21-05-2873 \
--source-uri=gs://nixos-images-gcp/nixos-image-21.05.2873.6bfe71f2a4e-x86_64-linux.raw.tar.gz \
--description=nixos-image-21.05.2873.6bfe71f2a4e \
--family=nixos \
--project=mudrii-nixos
List available images in your image repository.
~> gcloud compute images list | grep nixos
nixos-image-21-05-2873-6bfe71f2a4e-x86-64-linux mudrii-nixos nixos-image-21-05 READY
Creating NixOS VM
Our network stuck is up a ready we can create a dedicated VM to build nixos GCP images.
Here are a few points to mention:
- VM will be created in the same region where we set our subnet.
--zone=asia-southeast1-b --machine-type=n2d-standard-4VM AMD EMYC 4 vCPU and 16 GB RAM--metadata=enable-oslogin=trueallow you to ssh into the VM with your Gcloud account--tags=allow-sshAdded tag to the VM to allow open port ssh we configured in the firewall--boot-disk-size=60GBsize of the Disk you may select a lower capacity I find 30 works well too--boot-disk-type=pd-ssdThis is the fastest option local SSD will speed up significantly image generation
~> gcloud beta compute --project=mudrii-nixos instances create nixos-base --zone=asia-southeast1-b --machine-type=n2d-standard-4 --subnet=subnet-sg --network-tier=PREMIUM --metadata=enable-oslogin=true --tags=allow-ssh --image=nixos-21-05-2873 --image-project=mudrii-nixos --boot-disk-size=60GB --boot-disk-type=pd-ssd --boot-disk-device-name=nixos-base
Created [https://www.googleapis.com/compute/beta/projects/mudrii-nixos/zones/asia-southeast1-b/instances/nixos-base].
WARNING: Some requests generated warnings:
- Disk size: '60 GB' is larger than image size: '3 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
nixos-base asia-southeast1-b n2d-standard-4 192.168.1.2 35.188.151.168 RUNNING
SSH into Remote
We can ssh directly to the newly created VM with the gcloud command:
~> gcloud compute ssh --project=mudrii-nixos nixos-base
Building the NixOS Image
Once SSH gets into the newly created NixOs Google VM, we can start preparing to build the NixoS CGP image.
Authenticate from VM into Google Cloud
First, we need to become root.
~> sudo -i
We need access to google SDK to upload newly created images into the google bucket created in the previous steps. To authenticate, we need google SDK, and instead of installing, we will use nix-shell.
[root@nixos-base:~]# nix-shell -p google-cloud-sdk git
Once we have access to the Gcloud binary, we can auth into the google cloud.
[nix-shell:~]# gcloud auth login[nix-shell:~]# gcloud projects list PROJECT_ID NAME PROJECT_NUMBER mudrii-nixos mudrii-nixos 111111111111[nix-shell:~]# gcloud config set project mudrii-nixos Updated property [core/project].[nix-shell:~]# gcloud config configurations list NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION default True my_email@gmail.com mudrii-nixos
To validate authentication, you can check if you have access to your Gcloud bucket created in the previous step.
Note: make sure you add BOTO_CONFIG=/dev/null or you will get an error during the image build.
[nix-shell:~]# export BOTO_CONFIG=/dev/null
[nix-shell:~]# gsutil ls -l gs://nixos-images-gcp
Update NixOS to the Latest Version
To create an image with the latest stable NixOs is essential to update the existing version with the latest version.
Check existing NixOs version and Linux Kernel.
[nix-shell:~]# nixos-version 21.05.3001.12eb1d16ae3 (Okapi)[nix-shell:~]# uname -a Linux nixos-base.asia-southeast1-b.c.mudrii-nixos.internal 5.10.62 #1-NixOS SMP Fri Sep 3 08:09:31 UTC 2021 x86_64 GNU/Linux
Updating NixOs to the Latest Version
Update 1st nix-channels:
[nix-shell:~]# nix-channel --list nixos https://nixos.org/channels/nixos-21.05[nix-shell:~]# nix-channel --update unpacking channels... created 1 symlinks in user environment
Update the first NixOS:
[nix-shell:~]# nixos-rebuild switch
building Nix...
building the system configuration...
these derivations will be built:
...
..
.
[nix-shell:~]# nixos-version
21.05.3021.8b0b81dab17 (Okapi)
Or, if the version is outdated significantly, run the following:
[nix-shell:~]# nixos-rebuild switch --upgrade
NixOs Cleanup
Before attempting image creation, it is a good idea to do a cleanup and do a garbage collection on existing updates and repair any sha inconsistency that may cause failed image build.
[nix-shell:~]# nix-collect-garbage -d removing old generations of profile /nix/var/nix/profiles/system removing generation 4 removing old generations of profile /nix/var/nix/profiles/per-user/root/channels removing generation 5 finding garbage collector roots... removing stale link from ... .. . deleting '/nix/store/trash' deleting unused links... note: currently hard linking saves 21.60 MiB 167 store paths deleted, 1129.80 MiB freed[nix-shell:~]# nix-store --gc finding garbage collector roots... deleting garbage... deleting '/nix/store/trash' deleting unused links... note: currently hard linking saves 21.60 MiB 0 store paths deleted, 0.00 MiB freed[nix-shell:~]# nix-store --repair --verify --check-contents reading the Nix store... checking path existence... checking hashes... path '/nix/store/kacsvbh8qjl28izy5g7a8p96z6xdvnik-google-cloud-sdk-340.0.0' was modified! expected hash 'sha256:0dvxzzklaswx0d2svx0nzjilqfmgd2dxffi7hcbz89p7r6w1jab0', got 'sha256:1wpsb25jajbqvsw29jy073w0cr994005xb22pb9p8mkinn9vx8hp' copying path '/nix/store/kacsvbh8qjl28izy5g7a8p96z6xdvnik-google-cloud-sdk-340.0.0' from 'https://cache.nixos.org'...[nix-shell:~]# nix-store --optimise 430.37 MiB freed by hard-linking 47107 files
NixOs GCP Image Build
To build a GCP nixos image, we need to clone the nixpkgs repository where build scripts is located:
[nix-shell:~]# git clone https://github.com/NixOS/nixpkgs.git --depth 1
Cloning into 'nixpkgs'...
remote: Enumerating objects: 45763, done.
remote: Counting objects: 100% (45763/45763), done.
remote: Compressing objects: 100% (29848/29848), done.
remote: Total 45763 (delta 1593), reused 38844 (delta 1293), pack-reused 0
Receiving objects: 100% (45763/45763), 30.46 MiB | 14.19 MiB/s, done.
Resolving deltas: 100% (1593/1593), done.
Updating files: 100% (27800/27800), done.
The final step is to run a script that will build a GCP nixos image based on the underlying system version.
Note: Make sure you specify your GCP storage bucket name in BUCKET_NAME=
[nix-shell:~]# BUCKET_NAME=nixos-images-gcp nixpkgs/nixos/maintainers/scripts/gce/create-gce.sh
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev
copying path '/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
...
..
.
/nix/store/ii2h0jqwfzmzdc6lxyfmg4ia5726r6g6-google-compute-image
gs://nixos-images-gcp/nixos-image-21.05.3021.8b0b81dab17-x86_64-linux.raw.tar.gz
NixOs Image Build Verification
Once the image is created and uploaded into Google storage bucket and in the image repository, we can verify by the following:
[nix-shell:~]# gsutil ls -l gs://nixos-images-gcp
434150003 2021-09-12T03:18:26Z gs://nixos-images-gcp/nixos-image-21.05.3021.8b0b81dab17-x86_64-linux.raw.tar.gz
Verify Image repository:
[nix-shell:~]# gcloud compute images list | grep nixos
nixos-image-21-05-3021-8b0b81dab17-x86-64-linux mudrii-nixos nixos-image-21-05 READY
Post-Build Cleanup
It is good to remove the nixpkgs repository to minimize space and speed on the next build and stop VM to reduce the cost on your cloud bill.
[nix-shell:~]# ls -la total 44 drwx------ 8 root root 4096 Sep 13 11:02 . drwxr-xr-x 17 root root 4096 Sep 6 10:55 .. -rw------- 1 root root 2538 Sep 10 10:27 .bash_history drwxr-xr-x 3 root root 4096 Sep 6 11:07 .cache drwxr-xr-x 3 root root 4096 Sep 6 11:07 .config lrwxrwxrwx 1 root root 64 Sep 13 11:02 gce -> /nix/store/ii2h0jqwfzmzdc6lxyfmg4ia5726r6g6-google-compute-image drwxr-xr-x 3 root root 4096 Sep 6 11:48 .gsutil -rw-r--r-- 1 root root 45 Aug 25 05:05 .nix-channels drwx------ 2 root root 4096 Sep 13 10:30 .nix-defexpr drwxr-xr-x 9 root root 4096 Sep 13 10:55 nixpkgs drwx------ 2 root root 4096 Sep 6 10:55 .ssh[nix-shell:~]# rm -rf ~/nixpkgs[nix-shell:~]# sudo shutdown now Connection to 35.198.250.170 closed by remote host. Connection to 35.198.250.170 closed. ERROR: (gcloud.compute.ssh) [/run/current-system/sw/bin/ssh] exited with return code [255].
Fin
Next time, when you want to create a new VM on Google Cloud, you can specify in custom images your latest nixos image. Make sure you add --metadata=enable-oslogin=true to allow you to ssh into the VM.
Opinions expressed by DZone contributors are their own.
Comments