NixOS and Home Manager Update With Nix Systemd Services

In previous articles, we described how we deploy native flake NixOs as a development environment and Home Manager to manage user’s specific packages, configurations, and services.

Context

All developers in the project are running as primary Dev environment NixOs VM on Google Cloud. Our development team is divided into two streams.

• Infrastructure/Cloud provisioning and configuration development
• Application development

The Problem We Try to Solve 

Only Infra/Cloud Dev team has access to the root “wheel.” Still, because we selected to run our development environment on the NixOS application, development team members can install any application or service for their account without the need for root privileges.

As we use flake for NixOs deployment, we tightly control flake.lock file allows us to maintain consistency between developers in the package version we deploy OS-wide and nix Home Manager.

As only Infra/Cloud Dev team can update flake.lock files that refer to Github SHA commit/revisions in nixpkgs.

Ex. flake.lock file:

{
  "nodes": {
    "home-manager": {
      "inputs": {
        "nixpkgs": [
          "nixpkgs"
        ]
      },
      "locked": {
        "lastModified": 1631573611,
        "narHash": "sha256-u2E/wstadWNcn6vOIoK1xY86QPOzzBZQfT1FbePfdaI=",
        "owner": "nix-community",
        "repo": "home-manager",
        "rev": "7d9ba15214004c979d2c8733f8be12ce6502cf8a",
        "type": "github"
      },
      "original": {
        "owner": "nix-community",
        "ref": "release-21.05",
        "repo": "home-manager",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1631545603,
        "narHash": "sha256-aT2UoEOnlEBpMRZKBLu/OBsDNTZQS48scjcL936VSLI=",
        "owner": "nixos",
        "repo": "nixpkgs",
        "rev": "b3083bc6933eb7fa4ee7bd4802e9f72b56f3e654",
        "type": "github"
      },
      "original": {
        "owner": "nixos",
        "ref": "nixos-21.05",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "nixpkgs-unstable": {
      "locked": {
        "lastModified": 1631470189,
        "narHash": "sha256-hkUPYlpNOY9nbG1ByRin9NzPAYnPtwq/nGxO/DoeZd0=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "364b5555ee04bf61ee0075a3adab4c9351a8d38c",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
        "ref": "nixos-unstable",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "root": {
      "inputs": {
        "home-manager": "home-manager",
        "nixpkgs": "nixpkgs",
        "nixpkgs-unstable": "nixpkgs-unstable"
      }
    }
  },
  "root": "root",
  "version": 7
}

We need to find a solution where the Infra/Cloud Dev team updates the latest flake.lock file that may have security or new package version updates.

Once the git repository was updated and changes pushed, we needed a mechanism to propagate the changes to the rest of the Google Cloud VM fleet running NixOs.

We find the pull model most robust and easy to implement using native Linux Systemd scheduling, and all this is directly integrated into our nix configuration system.

Solution

We have two separate managed components, one is for managing global OS configuration, and one is for the Nix Home manager. We need two different Systemd services to manage updates for every component update separately. Ideally, our solution should support nix natively as we do not want to diverge too much from the nix configuration language.

NixOs System-Wide Update Systemd Service

We have few options:

1. We can use a separate git repository and integrate it directly into /etc/nixos/flake.nix
2. You can find the example here: GitHub NixOs SYS Auto Update Repo.
3. Option two is to have direct integration with the existing nix configuration.

By having a service directory in the main repo and define with nix expression:

{config, pkgs, lib, ...}:

let
  cfg = config.services.nixos-auto-update;
  name = "nixos-auto-update";
in

with lib;

{
 options.services.nixos-auto-update = with types;{
   enable = mkEnableOption "Auto update NixOS onboot/weekly";
   gitPackage = mkOption {
     type = package;
     default = pkgs.git;
   };
 };

 config =
   let
     cfg = config.services.nixos-auto-update;
     gitPath = "${cfg.gitPackage}/bin/git";
     mkStartScript = name: pkgs.writeShellScript "${name}.sh" ''
       set -euo pipefail
       PATH=${makeBinPath (with pkgs; [ git ])}
       export NIX_PATH="nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos:nixos-config=/etc/nixos/configuration.nix"
       cd /etc/nixos/
       ${gitPath} pull origin master
       ${config.system.build.nixos-rebuild}/bin/nixos-rebuild switch --flake '/etc/nixos/#nixtst' --impure
     '';
   in
   mkIf cfg.enable {
     systemd.services.nixos-auto-update = {
       description = "Auto update NixOS onboot/weekly";      
       wantedBy = [ "default.target" ];
       serviceConfig = {
         ExecStart = "${mkStartScript name}";
       };      
     };

     systemd.timers.nixos-auto-update = {
       description = "Timer for auto update NixOS"; 
       wantedBy = [ "timers.target" ];
       timerConfig = {
         OnBootSec = "5m"; # first run 5min after boot up
         #OnUnitActiveSec = "1w"; # run weekly
       };      
     };
  };
}

Note: Just to make sure to import service.nix file into the following: 

/etc/nixos/configuration.nix imports = [./services/nixos-auto-update.nix];  and declare in services services.nixos-auto-update.enable = true;

NixOs Home Manager Systemd Service Update

As for the nix Home Manager, we will define the same as for OS-wide update:

1. We can use a separate git repository and integrate it directly into ~/.config/nixpkgs
2. You can find the example here: GitHub NixOs HM Auto Update Repo.
3. Option two is to have direct integration with the existing nix configuration.

By having a service directory in the main repo and define with nix expression:

{config, pkgs, lib, ...}:

let
  cfg = config.services.nixos-hm-auto-update;
  name = "nixos-hm-auto-update";
in

with lib;

{
 options.services.nixos-hm-auto-update = with types; {
   enable = mkEnableOption "Auto update NixOS Home manager onboot/weekly";
   gitPackage = mkOption {
     type = package;
     default = pkgs.git;
   };
 };

 config =
   let
     gitPath = "${cfg.gitPackage}/bin/git";
     mkStartScript = name: pkgs.writeShellScript "${name}.sh" ''
       set -euo pipefail
       PATH=/run/current-system/sw/bin:
       cd ~/.config/nixpkgs
       ${gitPath} pull origin main
       $HOME/.nix-profile/bin/home-manager switch --flake ".#$USER" -v
     '';
   in
   mkIf cfg.enable {
     systemd.user.services.nixos-hm-auto-update = {
       Unit = {
         Description = "Auto update NixOS Home Manager onboot/weekly";
       };
       Install = {
         WantedBy = [ "default.target" ];
       };
       Service = {       
         ExecStart = "${mkStartScript name}";
       };      
     };

     systemd.user.timers.nixos-hm-auto-update = {
       Install = {
         WantedBy = [ "timers.target" ];
       };
       Timer = {
         OnBootSec = "10m"; # first run 10min after boot up
         #OnUnitActiveSec = "1w"; # run weekly
       };      
     };
  };
}

Note: Just to make sure to import service.nix file into your 

~/.config/nixpkgs/users.$USER/home.nix imports = [../../services/nixos-hm-auto-update.nix]; and declare in services services.nixos-hm-auto-update.enable = true;

Note: solution and code were developed by one of my colleagues in the project.

Verification

Once services have been deployed, we can see if services started and completed successfully.

Note: System update service will run after 5 min once os started.

~> systemctl status nixos-auto-update
● nixos-auto-update.service - Auto update NixOS onboot/weekly
     Loaded: loaded (/nix/store/l9am9p9kb7ym6djli9zqxxfpm4r86nvr-unit-nixos-auto-update.service/nixos-auto-update.service; enabled; vendor preset: enab>
     Active: inactive (dead) since Wed 2021-09-15 14:56:20 +08; 54s ago
TriggeredBy: ● nixos-auto-update.timer
    Process: 1525 ExecStart=/nix/store/kvbs7rq26r1crjr61nls91im0pbqf0lj-nixos-auto-update.sh (code=exited, status=0/SUCCESS)
   Main PID: 1525 (code=exited, status=0/SUCCESS)
         IP: 4.6K in, 2.4K out
        CPU: 5.458s

Sep 15 14:56:20 nixtst kvbs7rq26r1crjr61nls91im0pbqf0lj-nixos-auto-update.sh[1584]: setting up /etc...
Sep 15 14:56:20 nixtst kvbs7rq26r1crjr61nls91im0pbqf0lj-nixos-auto-update.sh[1563]: reloading user units for mudrii...
Sep 15 14:56:20 nixtst su[1714]: Successful su for mudrii by root
Sep 15 14:56:20 nixtst su[1714]: pam_unix(su:session): session opened for user mudrii(uid=1001) by (uid=0)
Sep 15 14:56:20 nixtst su[1714]: pam_unix(su:session): session closed for user mudrii
Sep 15 14:56:20 nixtst kvbs7rq26r1crjr61nls91im0pbqf0lj-nixos-auto-update.sh[1563]: setting up tmpfiles
Sep 15 14:56:20 nixtst kvbs7rq26r1crjr61nls91im0pbqf0lj-nixos-auto-update.sh[1563]: the following new units were started: logrotate.service
Sep 15 14:56:20 nixtst nixos[1563]: finished switching to system configuration /nix/store/f01hww03nd76ipnyc1azjymwvy14sf75-nixos-system-nixtst-21.05.20>
Sep 15 14:56:20 nixtst systemd[1]: nixos-auto-update.service: Succeeded.
Sep 15 14:56:20 nixtst systemd[1]: nixos-auto-update.service: Consumed 5.458s CPU time, received 4.6K IP traffic, sent 2.4K IP traffic.

Note: Nix Home Manager update service will run after 10 min once os started to eliminate any conflict with os serviced scheduler.

~> systemctl status nixos-hm-auto-update --user
● nixos-hm-auto-update.service - Auto update NixOS home manager onboot/weekly
     Loaded: loaded (/nix/store/962p2c97cbysfzqqm5fw9g60jh3kx8bm-home-manager-files/.config/systemd/user/nixos-hm-auto-update.service; enabled; vendor >
     Active: inactive (dead) since Wed 2021-09-15 15:33:05 +08; 59s ago
TriggeredBy: ● nixos-hm-auto-update.timer
    Process: 2063 ExecStart=/nix/store/7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh (code=exited, status=0/SUCCESS)
   Main PID: 2063 (code=exited, status=0/SUCCESS)
        CPU: 666ms

Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2214]: '/home/mudrii/.config/fish/functions/fish_greeting.fish' -> '/ni>
Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2217]: '/home/mudrii/.config/git/config' -> '/nix/store/962p2c97cbysfzq>
Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2238]: '/home/mudrii/.config/systemd/user/nixos-hm-auto-update.service'>
Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2244]: '/home/mudrii/.config/systemd/user/default.target.wants/nixos-hm>
Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2265]: '/home/mudrii/.local/share/fish/home-manager_generated_completio>
Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2274]: '/home/mudrii/.gnupg/scdaemon.conf' -> '/nix/store/962p2c97cbysf>
Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2277]: '/home/mudrii/.gnupg/gpg.conf' -> '/nix/store/962p2c97cbysfzqqm5>
Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2076]: Activating onFilesChange
Sep 15 15:33:05 nixtst 7hdczx9za1xfn5v7822zqn15i1qsx6ll-nixos-hm-auto-update.sh[2076]: Activating reloadSystemd
Sep 15 15:33:05 nixtst systemd[1238]: nixos-hm-auto-update.service: Succeeded

Hope this helped!