After three years, we finally got a proper IRL NixCon. Since the Nix community is not the biggest one, it was nice to talk about Nix for three days (two at the conference and one at the sprint). We heard many interesting talks and answered a few important questions. So let’s start with the most exciting things first.

If I had to give this conference a tagline, I would give it “What’s up with Flakes?”. This was probably the most asked question, and I think the organisers were aware of this beforehand, so Eelco gave a talk about them (I recommend you watch it). The short version is: Flakes are the de facto standard, but they have a major problem (primarily) with large packages.

The second biggest news was the new Nix Board. The board should help the community overcome obstacles and allocate resources for, e.g. meetups, conferences etc. Officially, they are not there to decide on any technical decisions (that is still in the hands of the community), but only time will tell if that will be the case.

Nix community grows rapidly, and several talks addressed this by pointing at the documentation and how to be more beginner friendly as a community. We all need to be aware that at this moment, at least half of Nix users are beginners, and with this growth rate, it will stay like this for a long time. And this is one complaint I have about the conference: it wasn’t “beginner friendly”. In 2019 in Brno, we had a talk about “Reading Nix expressions” where beginners could learn about Nix fundamentals. Because Nix is so complex, I think having solid fundamentals is very important, so I would say we need to have a talk like this at every Nix conference, especially if we know that ~50% of Nix users are beginners. I hope the organizers won’t make this mistake again.

Other than that, I enjoyed the conference, and I hope I won’t have to wait for three years to see the next one.

Links:
– Day 1 talks
– Day 2 talks

Talk Notes

Eelco Dolstra, Ron Efroni: Nix, State of the Union

NixCon, 11 organizers, over 30 volunteers, 200 attendees in person, 150 virtual via streaming.

Thank you sponsors!

Nix is old, started in 2003 as an academic project. Some people started using it for real work – scary!

Shoutout to 2013 Zidanca sprint, first in-person Nix event, thank you Plone people for letting us hijack your event.

NixOS community keeps growing, every release more contributors and more commits than the previous one.

All graphs go up! 50% YoY growth in github stars. Google Trends is up, cache traffic is up.

3k repos with default.nix, 3k with shell.nix, 2k with flake.nix (not really experimental feature, is it?)

Hackernews posts: Nix is still to hard to use, language is too hard, steep learning curve. They are right.

Don’t assume what the actual painpoints are, ask people.

Ran a survey at the beginning of the year. 2100 responses. Key takeaways:

Lowering barriers to Nix
Documentation !!!!!1111oneone
Better debugging and error messages
Contributing to Nix
Reproducibility

NixOS Foundation reboot, to better serve the community:

Expanded the board
More active support – empower teams
A public 3-month plan
Board panel / fireside chat later today
Board does not dictate anything, only empower the community

NixOS Vision

Nix everywhere
Universal build/conf management tool
Every project has a flake.nix
Principles are widely understood
Good UX
What else? Need input from the community!

NixOS Roadmap

We need roadmaps for the entire project and for subprojects
Need community input!
Starting points:
Empower teams in the NixOS project
Stabilize flakes
Stabilize the new CLI
Nix 3.0
Improve docs
Improve the installer

Q&A:

Has the board considered current supply chain security craze and if we can help the general devops community not reinvent the wheel?
Perfect example of good community input, we (the board) haven’t thought about it!
Fantastic idea, let’s ride the wave!
What’s up with Marketing Team, what’s the plan?
Tons of work needs to be done with awareness.
Marketing is beyond just owning the website. It is a a critical team.
More talks and discussions to follow.
Great example of a team that needs empowerment.

Live-streamed on https://www.youtube.com/watch?v=l70haNBm1wc&ab_channel=NixOS.

John Ericson: Learning the Nix Store layer

Been using Nix for many years.

Most of my RFCs touch the store layer. Point of commonality.

Gabriella Gonzales: Nix: Under the hood 2018
Valentin Gagarin: Nix architectures docs

What is the Nix Store:

Data model
Operations on that data model
Various implementations

Interface is not the bottom layer after all!

Nix Store Data Model

Build Inputs
Sources
Intermediate results
Build Outputs
Final results
Intermediate results

One of:

File (executable, contents)
Directory (entries)
Symbolic Link (target)

Store Path is a reference to a Store Object

Name
Digest
Store dir

Second task: Build plans

They are graphs
Nodes are tasks or data
Edges are dependencies
Sources are leaves
Final task is root

Derivations

outputs
inputs
builder
args
env
name

Inputs can be plain data, e.g. sources, but also outputs of other derivations. Derived Path.

Q: Does DerivedPath live on the filesystem? No. It’s a type in the C++ implementation. Better syntax coming soon.

Operations on Store objects: CRUD without U, because they are immutable.

Building is the interesting part. Before we build a task, we need to build all subtasks. All DerivedPaths need to be computed.
Finally, collect all outputs into proper store objects. Also, ensure outputs are acyclic.

An example of building a hello world example with a zlib input.

Greater Nix Ecosystem has more players interacting with the Nix Store concept: go-nix, TVix, Guix

A great example of how base concepts can be agreed upon as foundations for many different visions of how the end product should look.

Q: What are distinct features of Nix Store concept from similar systems and why is this a unique advantage
Bazel has a bit of this separation of layers. CMake has some layering as well. Not as sophisticated though. And lack sandboxing. Layering is flexibility, sandboxing allows us to stay sane when scaling starts.

Live-streamed on https://www.youtube.com/watch?v=l70haNBm1wc&ab_channel=NixOS. Talk description on https://talks.nixcon.org/nixcon-2022/talk/YPUVQK/.

Emi Nicole: Nix for particle accelerators, and the adventure in cross-compilation

CEA, fundamental physics research.

NixIS image for an embedded board to be used in a particle accelerator. How we did it!

What is cross-compilation?

We have different workload types. Raspberry PI vs MacBook Pro. Both well supported by Nix. We want to compile on the MacBook for the Raspberry PI. For dev UX reasons, faster compilation times, sometimes availability issues — need to compile for Windows, don’t have a Windows machine handy.

Build Platform vs Host Platform vs Target Platform

Jorg Thalheim, An introduction into Cross-Compiling with Nixpkgs
John Ericson “LLVM Toolchain in Nixpkgs”

An example of compiling htop on x86_64 to run on RaspberryPI.

buildInputs vs nativeBuildInputs vs depsBuildBuild

pkgs.pkgsCross.raspberryPi.hello -> compile hello on your platform for running on RaspberryPI

lib.systems.examples -> list pre-configured systems

You can configure additional systems yourself.

nixpkgs.crossSystem = lib.system.examples.raspberryPI -> compile the NixOS image for target system

My context: acquisition boards getting data about electromagnetic fields

IFC1410 from IOxOS
powerpc64 (Big Endian)
No storage
Acquisition software is already packaged using Nix.

We took /profiles/minimal.nix and 22.05 stable channel.

A lot of packages. A LOT. Huh, why is it compiling Rust compiler?

nix build ‘–keep-going’ to skip failures:

nettle
kexec-tools
linux kernel
rustc

“ABI” is the theme in all error dumps.

Some assembly code error for the kernel.

ABI -> Application Binary Interface

Calling conventions
How data is stored in memory

Many standards for ABI. Of course 🙂

gcc has different default ELF versions for different platform variations. Some projects assume different ELF versions for different platform variations. Boom.

nettle, kexec-tools and rustc make the wrong assumptoin that ELFv1 is going to be used.

Luckily some other distros (Void Linux, FreeBSD) already have patches to support ELFv2.

But we realy don’t need rustc.

nix path-info --derivation --recursive .|rg rustc and nix why-depends --derivation . <path to derivation> to find what depends on rustc. Seems to be polkit, so I disable it. There is more stuff that dependes on rustc, disable them too.

Now rustc is no longer compiled, yay!

OK, everything fixed, linux kernel is remaining. Fixing this took a long time.

The culprit was binutils, kinda? It includes assembler and sometimes gcc pases the wrong machine to the assembler.
It was fixed upstream.

Now everything is working!

Keep all overlays in hacks.nix file to allow others to understand what is going on.

Things that were hard:

How to add a custom kernel (missing docs)
How to add a custom kernel configurations and DTBs (missing docs)
The U-Boot netboot
Removing some unneeded file-systems/grub

Also looked at Yocto and Buildroot. Nix could learn from them:

grow the community to support more niche systems
improve docs to get started

Q&A:

NixOS defaults to BigEndian with ELFv2 for powerpc64. Why?
Some projects really require this.
How stable is cross-compilation over time? Any regressions?
So far so good, 22.05 has seen little issues over time.

Live-streamed on https://www.youtube.com/watch?v=l70haNBm1wc&ab_channel=NixOS. Talk description on https://talks.nixcon.org/nixcon-2022/talk/8HARVG/.

Jade Lovelace: Debugging closure sizes graphically

Working in Mercury on internal tools for risk management.

What is closure size? Size of the thing, all dependencies and all their dependencies.

Why is it a problem? Every time you use your app, you ship around tons of useless bytes. Containers don’t work great with large images. Slower pulls, slower deploys, etc.

Docker layers solve this, to a point.

nix path-info for figuring out closure sizes.

You can generate nice visual graphs of your dependency graph. But graphviz does not work great with large graphs.

nix-closure-graph --lg -> much better rendering

nix why-depends is great for figuring out why something is included in the dependency graph.

GHS puts dependency paths into executables for fun. Bug reported.

fly.io is cool.

Use nix to build a Docker image for my JS app. 500+ MB? That’s too much? Why is Python in the deps graph??

NodeJS has a legacy build system that depends on Python. It was inherited from Chromium, but is not longer used.
Turns out, nixpkgs has nodejs-slim without this dependency!

Second question: why is our app included twice in the deps graph?

dontPatchShebangs = true;
extraBuildInputs = [ removeReferencesTo];
disallowedReferences = [ "nodejs-16_x"];

Closure size down 40%! Yay!

But hey! OpenSSL depends on Perl?! Open bug in nixpkgs to potentially delete this dependency and save another 40MB.

Live-streamed on https://www.youtube.com/watch?v=l70haNBm1wc&ab_channel=NixOS. Talk description on https://talks.nixcon.org/nixcon-2022/talk/KFR3ZF/.

NixOS Foundation Board fireside chat

Eelco why did you start the foundation?

Needed to raise donations. No OpenCollective at that time, I was just out of Uni.

Some work is being neglected, lost of volunteering that is not seen. How will foundation improve this? We don’t yet have funds to pay people for critical work like the Rust/Haskell foundation does. We will first improve sponsorships and donations. We need to get the Foundation to be a charitable entity. We are already doing some things, like Summer of Nix, NGI Zero grants. We are still in the process of migrating to this new board, transferring the accounting, improving transparency. We are working on a roadmap, will need feedback. Like tech debt, we have “board debt”. We need to clear it.

What is the interface between the Board and the community? Does the board influence decisions?

The Board is in service of the project/community. We are volunteering time for survey, to help teams, etc. We don’t make decisions. We can’t even if we wanted. We build structure and help the community to reach decisions.

Why these people in the board? Why not someone else?

We wanted something that is representative of the various big stakeholders, so big companies and also long time community members. We have “maximum one person per company” rule.

Will board members be elected in the future?

Not for now, we will start talking about this in the future.

How will you address the funding?

ATM we don’t spend enough money. Only then we can start raising. Come up with sponsorhip packages for companies. Needs more transparency into how we spend money. Foundation donated 10k to NixCon. And we collected all sponsorhips through the foundation. We will keep doing this. Empowering teams means funding them too. Sponsorhips of local meetups, small events, not just the NixCon

How will you make sure teams are supplied with all needed resources to be productive?

We have organic teams. Keep people doing what they want, keep organic feel, but provide some structure for critical parts.

RFC process and how can Foundation help with it?

RFC process needs some improvements. Some RFCs just won’t go through, and that’s OK. RFCs need shepherds. People feel lack of authority, Foundation needs to encourage people to step up. Get things unstuck by talking to people. Board does not make technical decisions. At some point we have “core team” that can do that.

What are some of the critical issues the Board is working on?

AWS S3 buckets. We don’t own them. We need to be able to resolve issues fast. To have access.

People are unhappy with flakes RFC, make issues more visible, make communication better.

Docs are often forgotten, left behind, need better structure.

Who do you escalate to when an RFC is stuck? Noone knows.

Live-streamed on https://www.youtube.com/watch?v=l70haNBm1wc&ab_channel=NixOS.

Markus Kowalewski: Nix in a scientific environment: bringing together Nix’s reproducibility with computational chemistry

Stockholm University, supporting people working on computational chemistry, keep computers busty by doing theory and running simulations.

In practice:

Differnetial equations
Linear algebra, matrices
(Big) Fourier transforms
Parallel computing
Jobs take 50.000 CPU hours and more

Traditional approach

software packages grown over decades, non-standard build systems
optimizations for specific machines, platforms
“Environment Modules” are still common
Software is built manually, entangled with OS (breaks any time OS is updated)
Run jobs on cluster with a work load manager, Slurm, PBS, Torque

Let’s use Nix to do all of this!

Reproducability of scientific simulations is hard. Potentially even need the same hardware. Nix makes this much easier.

nixpkgs already has packages for:

biology
chemistry
math
physics/astronomy
machine learning, data science

NixOS-QChem overlay

incubator for new packages
quantum chemistry packages
optimizations for modern CPUs
support proprietary/non-free packages
CUDA support
tests
upstreaming patches back to nixpkgs
published in a scientific journal

If you change CPU optimization flags, you get a massive rebuild. Not ideal. We only set those on overlayed packages.

Slurm workers using nix-shell. Much easier control over scripts.

Vector extensions is not pure because of dynamic CPU optimization features detection. Need to recompile OpenBLAS for your specific hardware.

Tests could become greedy, blowing up all CPU cores. OMP_NUM_THREADS to fix this.

Yes, Fortran is still widely spread 🙂

Lots of old software from the 80s or even earlier, with broken Cmake scripts, interactive installers, home dir installations, updating .bashrc, missing dependencies.

Have fun patching!

In NixOS-QChem, nixpkgs is always pinned. Flakes are supported too. Pinning is essential for practical use. Used in production in Stockholm and Jena (Germany). 15 nodes, 500 cores, AMD and Intel. Contributions welcome!

Future:

benchmarking
vendored based compilers
motivate computing centers to provide Nix

Live-streamed on https://www.youtube.com/watch?v=l70haNBm1wc&ab_channel=NixOS. Talk description on https://talks.nixcon.org/nixcon-2022/talk/MYHSKT/.

Domen Kozar: Why we’ve built yet another Nix deployment tool

Last two years I focused on Documentation (https://nix.dev) and Infrastructure (binary caches, deployments).

There are 10+ deployment tools for/with Nix. Why build another one?

Developer Experience.

What would be the minimal tool that is the simplest to use and covers most use cases.

Intuitive interface. Support for known dev workflows. Documentation. Simple setup.

How do you manage your NixOS laptop?

Ideally:

vim config.nix
git commit && git push
CI/CD does the rest

Design Decisions:

Pull architecture instead of pushing stuff to servers. Closer to Nix mentality. Easier to scale.
Binary deployments. No reason to build/evaluate on target machine.
CI/CD mentality. Zero-effort deploys.
Cloud scale. Thousands of NixOS servers should be easy.
Firewall agnostic. No need to open ports on end devices for stuff to be pushed to it.
Slow/intermittent connection friendly. Laptops when travelling, IoT devices should have good support.

Terraform is great for provisioning, we should use it instead of reinventing it.

You need to run Cachix Deploy agent on end device.

--bootstrap flag to start from scratch. Useful for nix-darwin setups.

rollbackScript for sanity checks.

Three tutorials already written: normal profile, nixos profile, nix-darwin. Coming soon: home manager, AWS, terraform, Hetzner bare-metal.

Diffing deployments. Partnership with Serokell. A new tool to show diff between two deployments.

OceanSprint.org! Come hack on Nix(OS)! Also, looking for more sponsors to cover catering and similar expenses!

Cachix Deploy is public beta! 10 agents for free! Give it a try https://docs.cachix.org/deploy/.

Live-streamed on https://www.youtube.com/watch?v=l70haNBm1wc&ab_channel=NixOS. Talk description on https://talks.nixcon.org/nixcon-2022/talk/JJJT7S/.

NixCon Paris 2022