Fast development on NixOS

We’ve been using NixOS for a few years, and it has proven to be a stable and reliable OS. More importantly, it’s declarative and reproducible, which is one of the main reasons we decided to migrate ~700 of our Ubuntu servers to NixOS.

However, it has a few downsides, and one of the biggest ones is the slow iteration of service configuration when you’re still in the development phase. For example: let’s say you want to create a new NixOS server where you’ll have your WooCommerce shop, your blog and some Python API service you need for managing WooCommerce. You decide to use HAProxy in front, to act as a reverse proxy.

Configuring this in NixOS is very easy because, for the most part, all the configuration is already set up, and in most cases, you only need to enable a service. In our case, if we want to install and start the HAProxy service, we put this in our .nix configuration:

services.haproxy.enable = true;

After rebuilding NixOS, we’ll have HAProxy running with the default settings. Great, but now we need to add our own custom HAProxy configuration:

services.haproxy.config = ''
global
    maxconn 5000
    log /dev/log local0
    ...
'';

After saving the configuration.nix file, you will quickly realize that the HAProxy configuration hasn’t changed. It’s still the same as before you made your changes to the services.haproxy.config even if you restart the service (i.e. systemctl restart haproxy). This is because of how NixOS works. You can’t simply change a file and expect changes to be applied. You need to rebuild NixOS (nixos-rebuild switch). You might think this is a bad thing, but I assure you it’s not. This step evaluates (among other things) all your .nix configurations and ensures you don’t have a typo or any other mistake. So, for example, if you write services.haproxy.enable = True; the build will fail, complaining that True should be true and that you need to fix this mistake.

But everything will still be up and running. No service will go down if you make a mistake here. This is very different from how, e.g. Ubuntu works. If you make a mistake in a configuration file, save it and restart a service, you will often bring down a service or even the whole server.

Because you have to do a rebuild every time you change, e.g. HAProxy configuration, the development process usually takes longer. The more complex the configuration, the longer it takes.

However, there is a way to (temporarily) speed up the iterations. If you check how HAProxy is executed (systemctl status haproxy), you’ll see something like this:

● haproxy.service - HAProxy
     Loaded: loaded (/nix/store/bvwv6xcvfnqm3ywp0nj1x6ffcfnnkyba-unit-haproxy.service/haproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-11-08 05:00:57 UTC; 3h 19min ago
     ...
     CGroup: /system.slice/haproxy.service
             ├─195706 /run/haproxy/haproxy -Ws -f /etc/haproxy.cfg -p /run/haproxy/haproxy.pid
...

You can see that it was executed with -f /etc/haproxy.cfg; if you inspect that file, you will see your HAProxy configuration. You cannot change it because it’s read-only, but you can replace it with another file. That other file can then be updated just like on Ubuntu.

So let’s create our own haproxy.cfg, and copy the content from /etc/haproxy.cfg.

$ mkdir /root/tmp && touch /root/tmp/haproxy.cfg
$ cat /etc/haproxy.cfg > /root/tmp/haproxy.cfg

Now we can run HAProxy with our config:

$ haproxy -Ws -f /root/tmp/haproxy.cfg -p /run/haproxy/haproxy.pid
haproxy: command not found

NixOS doesn’t have HAProxy installed? But we installed it with services.haproxy.enable = true;!?

This is one of the things you’ll need to wrap your head around. In NixOS, everything is compartmentalised. We’ve installed the HAProxy package, but not globally, just for one particular service (haproxy.service).

What we need to do is a) install the HAProxy package system-wide (i.e. add it to the environment.systemPackages list) or install it temporarily for one-time use with nix-shell. Let’s do this:

$ nix-shell -p haproxy
$ haproxy -Ws -f /root/tmp/haproxy.cfg -p /run/haproxy/haproxy.pid
...

And this is it. You can now stop the service (ctrl+c), update the config file and start it again. Once satisfied with your settings, add them to the .nix file and rebuild the system (nixos-rebuild switch). I also strongly recommend keeping .nix configuration files in a git repository. With this, you can set up a CI to trigger a rebuild whenever the .nix file changes, but this is a discussion for next time.