Can we build it?

So at this point, my home-lab consists of the following:

• an approved-packages repository, which is the primary package source for my nix-configs
• a local network nix cache server
• a remote nix build server (which is also the cache server, so the cache is fed)
• a NixOS powered ThinkPad development laptop

What do I want to do next? Well my current pain point is starting to be the approved-packages repository. I have to keep it up to date for one thing. Once I update it, I have to then update any repos that depend on it. Currently, that’s just my nix-configs, but obviously I will build others.

I want to automate this as much as possible, so that I don’t increase the burden with every new repo I create.

So I figured I could build a github workflow to update the approved-packages repo on a schedule. Then I could run the same update on my nix-config repo some time after the approved-packages repo update had completed, and it would do the same thing. I didn’t need one workflow to trigger the other. Nothing complicated. Just one workflow, shared.

I rarely have a unique problem, nor solution for that matter. I figured that someone else will likely have solved this already, and so I did a quick search for it. The update-flake-lock action from Determinate Systems looked like a good option to me. This action creates a branch, updates the flake.lock file, then raises a PR. Not bad for now.

Let’s talk about flakes (again)

A flake is a special Nix file that takes in some inputs (usually other flakes), and (often but not always) exposes some outputs. In the case of my approved-packages repo, it takes in the nixpkgs and unstable flakes from the Nix packages repo, and exposes just the packages I want to allow out to my consuming flakes - currently just my nix-config.

However, a flake is pinned to the commit hash versions of its inputs. This is deliberate and important, as this is how flakes guarantee reproducibility.

This means that my approved-packages are always pointing to the same version of the nixpkgs though. If I want to allow access to more recent versions, I need to update my flakes in the approved-packages. Then after that, I also have to update my consumer repo (my nix-config) to pull in the updated version of the approved-packages. A faff, certainly. A pain, eventually.

There are a lot of great benefits I’m seeing with my approved-packages, but there is an overhead too. Though it’s an overhead that is inherently automatable. How am I going to do that? I’m going to use the infrastructure I’m building to build the infrastructure. Ouroboros!

Github workflows, baby!

The repo is hosted in Github, and I pay a tiny fee each month so that I can play with the various toys. I’d only need to run this once a week really, as I don’t need to be on the bleeding edge of the packages. So where’s the problem?

Honestly, there is no problem with this, except that I do actually have a bigger plan. I keep saying this home-lab was an accident, but this isn’t quite true. I’m building towards a fully automated, self-healing infrastructure. To get there, I need to solve the foundational problem of dependency management. The accident part is that I had expected to be building this using cloud platforms and things. For various reasons, it just became obvious that to control costs and technical decisions, I would have to own the whole process.

So what am I going to need then? Well, I know I’m going to need to build my own apps, packages, tools, and things eventually. Having Nix based systems is an enormous boon to not only to development, but also to building solutions.

While there is some benefit to using Nix in a Github runner, there is much greater benefit in using Nix in a self-hosted Github runner. If it’s on my network, it could use my cache server. Whenever I build locally, the cache is refreshed which will help the runner builds. Similarly, the runner builds would also refresh the cache, which would help my local builds. This is mighty.

It’s agreed then. I need some self-hosted Github runners. Time for another google, a chat with Dave, and an obligatory search on Amazon for another ThinkCentre.

We’re using self-hosted Github runners where I currently work, and so I’m acutely aware of the potential maintenance overhead they can be. We use containers and images, and maintain an internal repo for the image to be built from. Each month or so, we update the image, then push it out to the environments. It’s a fair bit of effort, and I really didn’t want more of the same at home.

Fortunately for me, I had a better option.

I use Nix, by the way

Nix is more than just a package manager. When running NixOS itself, you’re able to define your entire system declaratively. Complex solutions can be defined in modules, then consumed by your system configs in a very readable manner, and with a very low cognitive load. In addition to that, Nix has a fervant community that build modules for the system to make the setup of such things very easy.

What this means for me is, in its simplest form, all I have to do is write services.github-runner.my-runner.enable and I have a runner.

Obviously, I will want to customise the runner so that it has things such as jq, curl, gh on there etc. And I know I’m going to want more than one runner, so I’ll define my runner in its own module too. Here is my final github-runner module:

1
{
2
  name,
3
  labels,
4
  max-memory,
5
  approved-packages,
6
  approvedLib,
7
  isLinux ? false,
8
}: {
9
  config,
10
  pkgs,
11
  lib,
12
  inputs,
13
  ...
14
}:
15
{
16
  services.github-runners.${name} = {
17
    enable = true;
18
    url = "https://github.com/machinology";
19
    tokenFile = config.sops.secrets.github-runner-token.path;
20
    inherit name;
21
    extraLabels = labels;
22

23
    # Use unstable github-runner to avoid deprecated version
24
    package = approvedLib.requireApproved "github-runner";
25

26
    serviceOverrides =
27
      lib.optionalAttrs isLinux {
28
        MemoryMax = max-memory;
29
      }
30
      // lib.optionalAttrs (!isLinux) {
31
        SessionCreate = true;
32
      };
33

34
    # Replace the runner if the token changes
35
    replace = true;
36

37
    # Run as a dedicated user
38
    user = "github-runner";
39

40
    nodeRuntimes = [
41
      "node24"
42
    ];
43

44
    extraPackages = approvedLib.approvedList [
45
      "nodejs"
46
      "nodejs_latest"
47
      "git"
48
      "just"
49
      "jq"
50
      "yq-go"
51
      "curl"
52
      "gnupg"
53
      "gh"
54
      "cocogitto"
55
      "go-latest"
56
      "actionlint"
57
      "pinact"
58
      "shellcheck"
59
      "yamlfmt"
60
    ];
61
    ephemeral = true;
62
  };
63
}
64
// (lib.optionalAttrs isLinux {
65
  users.users.github-runner.extraGroups = ["docker"];
66

67
  virtualisation.docker = {
68
    enable = true;
69
    daemon.settings = {
70
      insecure-registries = [
71
        "registry.k3s.machinology.internal:30500"
72
      ];
73
    };
74
  };
75
})

To consume that module, I have another module that pulls them in and names them:

1
{
2
  approved-packages,
3
  approvedLib,
4
  ...
5
}: {
6
  imports = [
7
    (import ./github-runner.nix {
8
      inherit approved-packages approvedLib;
9
      name = "mach-runner-1";
10
      isLinux = true;
11
      labels = ["nix" "nixos"];
12
      max-memory = "10G";
13
    })
14
    (import ./github-runner.nix {
15
      inherit approved-packages approvedLib;
16
      name = "mach-runner-2";
17
      isLinux = true;
18
      labels = ["nix" "nixos"];
19
      max-memory = "10G";
20
    })
21
    (import ./github-runner.nix {
22
      inherit approved-packages approvedLib;
23
      name = "mach-runner-3";
24
      isLinux = true;
25
      labels = ["nix" "nixos"];
26
      max-memory = "10G";
27
    })
28
  ];
29

30
  users.users.github-runner = {
31
    isSystemUser = true;
32
    group = "github-runner";
33
    home = "/var/lib/github-runner";
34
    createHome = true;
35
  };
36

37
  users.groups.github-runner = {};
38

39
  nix.settings.trusted-users = ["github-runner"];
40

41
  sops.secrets.github-runner-token = {
42
    sopsFile = ../../secrets/github-runners.yaml;
43
  };
44
}

For those in a hurry, the above snippets create 3 self hosted runners, running as systemd services on one of my NixOS based ThinkCentres. The systemd services allow the runners to “believe” they are separate machines, but they also get access to the host’s resources, as well as the nix cache server on the network.

Notice as well the use of approved-packages, where I pull in the github-runner package via the approved packages, thereby gating the package through an auditable process.

Nix is a beautiful thing, it really is.

So now I have these 3 runners, I need to start putting them to work.

Workflows, runner dependencies, and application dependencies

I’ve been using Github Workflows a lot over the last few years, due to it being in place at my most recent couple of roles. This has helped me form ideas for how to build workflows and actions with reduced friction. I’ve also started to think of Github Workflows as another platform at my disposal, not simply for build pipelines.

If you’ve used Github Workflows (or any of the declarative pipeline tech, I expect), you’ll appreciate that there is a friction between keeping the workflow so simple it’s “boring”, and yet also ensuring that the workflow is able to build you application / service / package etc.

There are a few ways of handling this, typically:

• You could simply run numerous setup steps in your pipeline. This might increase build times and mean multiple redundant downloads that could have just been done once.
• You could use docker images instead, and just docker exec the calls in each step. This has much the same problem as the pipeline approach, but you could build a custom image with everything installed, then just have the one image.
• You could also just install all of your dependencies onto your agents. That way, they already have everything ready. You are slightly restricted in terms of which language / tech you use, but you could always just accept that maintenance overhead and run with it.
• This is what you could do. I have Nix. I don’t have to do any of that.

The Nix Way: The environment is the dependency

Flakes define the dependencies for a development environment. Very often, the development environment is also the build environment. And my 3 self-hosted Github runners are all Nix-based. So just as I can run nix develop . to build the development environment and pull down the dependencies, so can my runners. Oh, did I mention I also have a nix cache on my network to reduce any flake build times? And also remote-builders to reduce the strain? Pretty sure I did.

Summary

So now I have 3 self-hosted Github Runners, fully Nix aware, and I have a shared workflow that automatically updates Nix flake based repos on a schedule. What’s next?

Well I know I’m going to need some help soon. I’m at least a one man band at the moment. I don’t want to be reviewing my own PRs and merging them in. I had Github Copilot at this point, and so that was an option. However, I know I’m going to want to push things further than the Copilot budget would allow. I could try the Anthropic / OpenAI route directly, but I was concerned about bills getting out of hand. That, and I was still trying to find the best fit for LLMs in my personal workflow.

What I needed was LLM sovereignty. The ability to experiment with the technology without fear of massive token usage, and large bills. I wanted to be able to try out ideas, knowing that it could just be a blind-alley or a waste of time. When it all costs money, that experimentation is generally hard to accept.

I needed a local LLM. But are they good enough? Are they fast enough? Could it even be cheap enough?

To find out, tune in next time. Same Bat place. Same Bat channel.