Nix is a beautiful thing, but there is a fine line between the joy of crafting the perfect configuration and the drudgery of maintaining it. As I pulled more of my machines into my home lab, that line started to blur. I was starting to feel the pain of rebuilding the same things over and over. Some of these builds can take more than 10 minutes, and so rebuilding them is clearly just a waste of time.

If this is going to scale, I needed somewhere to store my built artefacts. I needed a cache server.

Now I could just have added a cache server to an existing machine, but at this point my main machine was a Mac laptop, and my other machines were a beefy PC and an old but beefy Mac Pro. None of these were good candidates for a cache server that needed to be left on for most if not all of the day.

The best things in life are …cheap(ish)

I saw a bunch of videos on Youtube a while back saying that refurbished mini PCs are well worth looking at, so I did a quick search on Amazon and came across a bunch of Lenovo ThinkCentres.

You may well have come across this sort of thing if you’ve worked in one of those organisations that rely on VDIs. Thin client machines that allow you to dial into a desktop. Well those thin clients can be mighty beasts. And (relatively speaking) cheap as chips.

I picked one up and expanded its RAM to 32GB, and its drive to a 1TB NVMe. Once it arrived, I installed NixOS within about 30 mins by just creating a new profile for it in my nix-config, then rebuilding it.

I have to be honest, this little machine is a beauty. Initially I put KDE’s Plasma desktop on, and it was gorgeous to use. Thanks to the Nix configs, all my apps were present and correct - specifically Ghostty and my Neovim setup. I’ll go into the Neovim setup in a completely separate post.

The whole point of the server is to remote into it, and specifically to SSH into it, and so I removed the desktop in favour of just a terminal. A lot less faff.

We have the technology

Now onto the cache server. After a quick google and a chat with Dave, I found Harmonia. This once again confirmed my personal theory that there are no original problems, and someone else has usually solved it. In most cases, they’ve also packaged it and turned it into a OSS project.

Harmonia works by hosting the /nix/store/ of its host machine as a web server. Setting it up was straightforward enough. The docs are excellent, and the declarative nature of Nix means that adding a significant feature to the configs is genuinely not a lot of work at all.

As you can imagine, there are some certs to be created and also distributed to the machines. I’m quite a stickler for making sure my configs don’t require any ancillary or pre-requisite tasks. Mostly because I know I could well be coming back to this in 6 months time and I don’t want to have to remember to do something. Luckily I have a tool that helps me with this kind of cognitive load management: just.

I can’t remember where I came across it, but I’ve been using just for a few years now. It’s a cracking tool that bridges the gap between scripts and readme. I love the slightly cheeky name too, and how it encourages you to write a focused task name.

A just task is an ideal place to add a setup for a certificate on your local machine. In this particular case, the certificate needs to be created in a particular place so that my configs can pick up. You can see the task below. I parameterised it for reuse, even though I’ll almost certainly only be using it on my homelab. I do this a lot, but add default values too for convenience and to show me what kind of value is expected when I come back to in many months later. Like the Oracle says, “Know thyself”.

1
# generate public/private key pair for harmonia
2
[group("utilities")]
3
generate-cache-key service_name="harmonia" domain="machinology":
4
  #!/usr/bin/env bash
5
  sudo mkdir -p /var/lib/secrets/
6
  sudo nix-store --generate-binary-cache-key cache.{{domain}}.tld-1 /var/lib/secrets/{{service_name}}.secret /var/lib/secrets/{{service_name}}.pub

A brief note about the Nix store

As you read through Nix docs, blog posts, etc, you’ll frequently come across warnings that the “Nix store is world-readable”. I never thought much about it, as I’m the only user on all my machines so far. When it comes to caching on the server though, this becomes a more pressing issue. Especially as this exercise is all about seeing how easy / difficult / ready Nix is for the enterprise.

The point of the “world-readable” warning is that any secrets that might end up the store will all be readable by anyone that could access it. So if you were to simply add a secret directly in your configs, those configs will be copied into the store as raw, unencrypted files.

Nix currently doesn’t concern itself with things like this. Not because it doesn’t care about security, but because that problem is already solved by other technology. Nix is mostly concerned about reprodibility and auditability.

So the problem is that the configs etc need to be available in the store and unencrypted, but they also can’t have secrets in there. Sounds tricky. As usual, smarter people than I have already thought about this.

Keep it secret, keep it safe

Obviously I’m not just going to go storing my certificates in the config codebase. So what can I do? Well I have to credit some folks that I worked with in a recent role for this. They had a similar problem and they used SOPS to take care of it.

For anyone that doesn’t know about it, SOPS effectively handles the encryption and decryption of data that you tell it to manage. It does this in a really elegant way that doesn’t get in the way of the user. That said, you do have to be aware of how to use it too.

There is a Nix flavour of this available, and so that’s what I used. SOPS-Nix handles the nix store issue by storing the unencryped secrets in a file on your local system, then you setup an expression to read that file content into your config. It’s pretty elegant, while still feeling “Nixy”. Getting it up and running is pretty trivial (Nix aside at least), and the docs are great.

While doing this, I had a number of issues around the whole ‘chicken and egg’ for certs, where the cache server didn’t have the certificates that my consumer would be passing, and even just the basic connectivity. Also, since implementing this, I would also occasionally break it, and so my justfile now has a bunch of tasks that check the cache connectivity is working. I have these setup to run after every rebuild of the configs too. Just is lovely.

The full implementation is a bit of a rabbit-hole involving Age keys and decryption workflows, but I’ll be doing a dedicated deep-dive tutorial on this and the other interesting parts of this whole adventure soon.

Cache complete

Finally everything is setup. Now so long as I built everything on my ThinkCentre first, the packages would be available to the cache consumers. Yep, that’s right. Feels…“suboptimal”, no? If only there was a way to build from my other machines and have that appear in the cache…?

Again, a quick google and consultation with Dave, and I have a number of options. One option is crude, which is to simply copy across my local /nix/store to the cache server after a build. Pretty weak.

The other way is much better. I needed to be able to trigger a build locally on my laptop, but have the cache server perform the build (and therefore cache it too). I needed some kind of…remote builders. Fortunately, the Nix maintainers thought of this a long time ago.

Now before I continue, I have to say this part really boosted my already rock-solid belief in the wonderful folks that built Nix. To appreciate how and why this mechanism exists, it’s useful to think back on my first post in this series that covered the Nix packages.

Nix has the largest repository of packages available. Nix also makes declarative development environments very easy. Those development environments are also build environments. This is not a coincidence. Nix has the largest repository simply because building and packaging are first-class responsibilities of the system. The build process is so fundamental to how Nix works, the tools to distribute those builds (such as remote builders) are already part of the architecture. Pretty strong.

The remote-builders in NixOS are really very neat. You might expect that a remote builder would be like a build agent, and in a sense that’s correct. The difference though is that all Nix machines are builders anyway, as they build their own packages. So rather then being a specified “remote builder”, you instead delegate the build activity to another machine. You can configure various constraints too, so that the lightweigh builds are performed locally, and the heavier ones are delegated to the remotes.

By delegating the heavier builds to the new server, the build artefacts get stored in the server’s Nix store, which Harmonia then makes available to consumers on the network. It’s a beautiful thing.

The documentation for this is pretty good, and I was able to take a lot of what I learnt on the cache certificates to help with this too.

Finally

This was a lot of work really, and I had to learn quite a lot to implement it. I learnt a great deal about Nix along the way, and my respect for the folks that built it grew immensely. And my reward for this was that I no longer have to build things from scratch with every rebuild.

Well…about that…

I keep telling people that I accidentally built a home lab. It sounds silly, but you’ve just read through the first server acquisition. The home lab is about to grow again.

You see, my main development machine is a Macbook Pro laptop. It is Nix powered, however it is Nix-Darwin powered. It’s still a MacOS machine, but it uses Nix-Darwin to control the configuration of the machine.

Nix targets whatever architecture the host is running. Therefore, Nix-Darwin targets ARM chips (Apple Silicon). My ThinkCentre cache server targets x86_64 chips. Since the machine architectures don’t match, the binaries they produce don’t match either. This means they have different build artefacts, and therefore different build derivations (think Docker layers).

This means they can’t use the same cache.

What? All this work for nothing?

Well yes and no. You see, while I may be developing on a Mac, my target really is Linux. And so it’s important that I can build for Linux. I’m about to make a tiny leap, but it does make sense.

I have an old Macbook Pro that is x86_64 based. I can definitely use that, as it has long been my Linux distro testbed anyway. That said, it is a bit laggy. Not like the lovely ThinkCentre. That’s snappy! If only ThinkCentre’s were laptops…oh wait! I used to use a Lenovo laptop years ago. I wonder if they’re available refurbished too?

And so, long story short, I ordered a ThinkPad with 32GB RAM. This was last year, back when RAM was actually cheap! Which explains why the seller shoved an extra 8GB RAM in there too. Kind of them, but it’s almost certainly slowing the machine down in reality, as the dual channel memory won’t be working, but that’s not the main point here. No, the main point is my cache server now finally has a proper consumer!

Long Live ThinkPad!