Project isolation beyond requirements.txt

—

by

Neyts Zupan
in

I’ve been a Python dev for almost 15 years now and I’ve recently completely overhauled how I keep my development tools and dependencies in check. I’m so happy with my new setup that I couldn’t wait to share it with fellow geeks!

Some background

I used to use Homebrew to install tools such as pyenv that I then used to install the exact Python version I needed that I then used to install the correct pipenv or poetry version that I needed that I then used to install all project Python dependencies. It’s a long chain but “if you know what you are doing” it works.

Most of the time. Until there was a new version of MacOS and I had to nuke my Homebrew environment and install everything again. Or when my system-installed black did not match the one that the project was using causing formatting failures in CI. Or when some other project implicitly depended on a non-Python library, such as chromedriver, to drive browser tests. I found myself frustrated more often than I would like. And often reaching for the “nuke from orbit” approach to fix broken system-level tooling. Surely there’s a better way? Maybe stop being lazy and finally, after a decade of duct-tape fixes, learn how to correctly maintain complex dependency trees?

To put it in other words: pinning down Python project dependencies is easy enough, with requirements.txt and similar files. The problem is pinning down tooling on a level above that. Executables that you need *before* you get to the level of installing Python libraries. And those that are written in other languages but your project still depends on them.

Then I found Nix. Wikipedia says Nix a cross-platform package manager that utilizes a purely functional deployment model where software is installed into unique directories generated through cryptographic hashes. Dependencies from each software installation are included within each hash, solving the problem of dependency hell. That’s a mouthful! This block of text kept me from trying Nix for years. It seemed way too demanding to delve into Nix when I had features to deliver. Such a mistake! I recently learned that it’s just the marketing part that is not great with Nix. The tool itself is fantastic! Let me convince you why with a simple example.

Current state of affairs

Let’s say you have a fresh MacBook in front of you and you need to work on a Python Web project. This usually means:

  1. Go through the README.md and hope there is documentation on what is needed to develop this project.
  2. Install Homebrew. Install pyenv. Install the Python version declared in README.md.
  3. Run the code and realize that the version declared in README.md is not correct — the authors forgot to update it. Classic. Find the correct version by looking into travis.yaml, runtime.txt or Dockerfile.
  4. Run browser tests and realize the README.md does not say anything about what is needed to run them. Try multiple versions of geckodriver and chromedriver to find one with which tests pass.
  5. Then the real trouble begins: the few bits of JavaScript code that the project ships with use jshint, and maybe parcel to bundle things together. Repeat all the above steps again, this time for the JavaScript ecosystem.
  6. Oh, … there’s also a Go executable in there. Oh well, the whole process one more time. Third time is the charm?

Just as everything is set up, an urgent support ticket comes in: a bugfix is needed on an old project. Open the project up, git pull origin master and run tests. Shit. Some of the tooling you just updated to work on the new project, breaks on this old project. Gawd, now figure out how to convince Homebrew to downgrade a certain package. It’s late afternoon already, maybe tomorrow you do some actual work.

There’s a better way

With Nix, to start working on a project you always run a single command: nix-shell. This assumes you have Nix installed globally, but you only do that once.

nix-shell reads the shell.nix file in your project and installs all the tooling that is needed to work on the project. Not just Python tooling. All tooling. It’s kinda like having requirements.txt & package.json merged into a single file. Everything is installed in a way that ensures:

  • you download & install a given tool only once,
  • projects are completely isolated and cannot pollute each other’s environments.

Let’s see how this looks in practice. Here’s an example shell.nix that provides Python 3.7:

let
  nixpkgs = builtins.fetchTarball {
    # https://github.com/NixOS/nixpkgs/tree/nixos-19.09 on 2020-04-15
    url = "https://github.com/nixos/nixpkgs/archive/f6c1d3b113ce9e7c908991df2e8eed02f03df1bc.tar.gz";
    sha256 = "08z6qbjmx64bcil3cnvflb7bv9ibdizxcr63yvhgcqzsja7m51n5";
  };
  pkgs = import nixpkgs {};
in

pkgs.mkShell {
  name = "dev-shell";
  buildInputs = [
    pkgs.python37Full
  ];
}

That’s it! Notice that nixpkgs is pinned to an exact commit in the nix packages repository so any time in the future you run nix-shell against this shell.nix file, you will get exactly the same Python installed. Talk about freezing your dependencies!

$ python --version  # system Python provided by MacOS
Python 2.7.16
$ nix-shell
...
[nix-shell:/tmp/test]$ python --version
Python 3.7.5  # Python provided by nix

Let’s say your project grows and you whip up a quick shell script that uses jq to fiddle some JSON files that your project needs. Instead of opening up README.md, typing Oh and you need to install jq to run these commands and then forgetting about it until you are banging your head the next time you are setting up your project, you simply add jq to the shell.nix file:

    ...
    buildInputs = [
      pkgs.python37Full
+     pkgs.jq
    ];
    ...

And re-run nix-shell.

$ jq --version  # jq is not available on MacOS out-of-the-box
zsh: command not found: jq
$ nix-shell
these paths will be fetched (0.33 MiB download, 1.24 MiB unpacked):
  /nix/store/6pgw7gk6b1kjmirgf6p7bwsjd4iq2xli-jq-1.6-dev
  /nix/store/9q5vskz61rs4d6vgnwl4k0s70bslkzp9-jq-1.6-bin
  /nix/store/kw2y88cw01nkq4bsplc755gdg8j9n3cr-onig-6.9.4
  /nix/store/nzmmv8ykpy8cr4fl64hx8k2cdmd9ddfb-jq-1.6-lib
copying path '/nix/store/kw2y88cw01nkq4bsplc755gdg8j9n3cr-onig-6.9.4' from 'https://cache.nixos.org'...
copying path '/nix/store/nzmmv8ykpy8cr4fl64hx8k2cdmd9ddfb-jq-1.6-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/9q5vskz61rs4d6vgnwl4k0s70bslkzp9-jq-1.6-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/6pgw7gk6b1kjmirgf6p7bwsjd4iq2xli-jq-1.6-dev' from 'https://cache.nixos.org'...

[nix-shell:/tmp/test]$ jq --version
jq-1.6

You also need Yarn bundle the frontend app? Easy! Add pkgs.yarn and re-run nix-shell. Need a database? Do the same with pkgs.postgresql_11 or pkgs.redis.

Automatic loading of project environments

But there’s more! You can use direnv to *automatically* load the nix-shell environment when you cd into the project folder. See this example where I have two projects using different Python versions and direnv to automatically load nix-shell:

$ python --version  # system Python provided by MacOS
Python 2.7.16

$ cd foo
direnv: loading /private/tmp/test/foo/.envrc
direnv: using nix
$ python --version  # inside project using Python 3.7
Python 3.7.5

$ cd ../bar
direnv: loading /private/tmp/test/bar/.envrc
direnv: using nix
$ python --version  # inside project using Python 3.8
Python 3.8.0

$ cd ..
direnv: unloading
$ test python --version  # back to system Python
Python 2.7.16

Where to go from here

Finally, remember that Nix is cross-platform? When you add shell.nix to git, all your colleagues, no matter what OS they use, will get the exact same tooling installed, greatly reducing the infamous “it works on my machine” scenario. Ditto for CI, staging and production environments.

Convinced to try out Nix yourself? Check these out:

  • The Introduction to NixOS talk given at a NYLUG meetup is what convinced me personally to give Nix a try. It covers all major features without using the jargon that Nix people like to use.
  • Search over all Nix packages to find what to add to shell.nix to install a specific tool.
  • A shell.nix of a “real world” Python Web app whose development (poetry), CI (CircleCI) and production (Heroku) environments are all built with Nix.

Neyts Zupan

Neyts is the Digital Overlord of Niteo, poking his nose into any and all technical things.