Writing a Nix Flake Library

This post is one that I wish had existed when I set out to create my first Nix project. While the Nix ecosystem has a plethora of documentation, I've often found it to lack an overview of the big picture for one reason or another. This lack of clarity probably contributes to the learning curve associated with Nix. While I can learn all about syntax and creating derivations, how all that fits into a larger project or configuration can be a rugged mountain to summit.

I've been slowly working on improving my development environment with Nix. My latest addition is a flake library which generates temporary configuration files for local development tools. This project was my first attempt at a Nix project that was larger than just writing a simple derivation, and as noted above, it had a bit of a learning curve. In this post, I want to share the general architecture I ended up with in hopes that someone else may find it helpful when tackling this type of project.

What is a Flake Library?

You'll likely come back with very little if you search around for this term. Indeed, I'll admit that I more or less made it up. However, I didn't do so without basis, as there are several projects out there that I would classify as flake libraries. The most popular is flake-utils. Other contenders would be the various 2Nix libraries (i.e., Poetry2Nix). The general idea behind each of these is that you can add them as an input to your flake and then utilize their functions to accomplish some task.

In this way, these projects are much like libraries you'd find in other programming languages. They abstract standard functionality into a reusable form that can be composed into something more significant. Thus, I settled on referring to them as flake libraries.

The Components of a Flake Library

The components of a flake library are perhaps the most difficult to quantity since, as far as I am aware, there exists no definitive standard which details them. I have more or less naturally discovered these as I have perused the projects of other flake authors. They are summarized as:

  • An external API
  • A combination of internal functions and modules
  • A suite of tests

Those familiar with developing third-party libraries will not be surprised after reading these. They serve as the foundation of practically any significant software development project. However, even being familiar with these components can lend very little to understanding how to accomplish it with Nix. It's not that Nix isn't capable enough for the task, but instead that it does little to hold your hand along the way. This difficulty was the biggest hurdle that I had to overcome, and I went through dozens of refactors trying to create the best representation of these components. To help divulge all of this, we will take a short stroll through my project to see practical examples.

External API

Like other language libraries, a flake library is intended to be consumed by a third party. In our case, this would be another flake developer interested in using the interface we provide. The consumer would add our flake as an input in their flake and then proceed to interact with it in the way we've defined.

Output Schema

The Nix wiki has a short section that documents the current schema of a flake. When flakes were first introduced, they were a revolutionary concept because before this Nix had very little ability to organize something into a single entity. Like many other languages offered, composition was challenging due to the lax nature of Nix and the inability of the community to settle on a single definitive way to organize a project. Flakes serve as the building blocks for a Nix project – they abstract a single idea that can then be composed into a final product.

What's interesting about the schema is that it's open-ended. Meaning, that we can add additional sections to the output and receive little more than a warning when we run nix flake check. Should we do this? This question was the first significant one that I came across, and from what I gathered from reading the source of many projects, it's not entirely frowned upon. Indeed, perhaps the most common output I saw across projects was lib. It's so popular that the author of Cachix created an issue about it some time ago (which, unfortunately, has had little traction).

With all of this in mind, I have settled on the idea that it's acceptable to create custom outputs to a degree. I say this with hesitation because flakes are not stable and subject to change, and in the future, the open-ended nature of the output schema may be squashed. So, when creating your flake API, bear all of this in mind and perhaps try to stick to outputs that many other projects are using, as they are likely to find a permanent home when the schema is finalized.

The external API that I landed on for Nixago uses two outputs: lib and plugins. The latter, of course, is pretty unorthodox, but it made sense in the grand scheme of what I was trying to accomplish. Nixago is built on the idea of generating temporary configuration files for use in a development environment. It offers plugins that can be developed to create configurations specific to a tool to support this. For example, it currently covers pre-commit, prettier, and others. Since this list can grow to a much larger number based on contributions, it made sense for me to aggregate them under a standard output which I chose to call plugins. The result is something like this:

{
  # Add input
  inputs = {
    # ...
    nixago.url = "github:jmgilman/nixago";
    # ...
  };
  
  # ...
  # Somewhere in outputs build the configuration
  preCommitConfig = {
    nixpkgs-fmt = {
      entry = "${pkgs.nixpkgs-fmt}/bin/nixpkgs-fmt";
        language = "system";
        files = "\\.nix";
    };
  };

  preCommit = nixago.plugins.pre-commit.mkLocalConfig preCommitConfig;
  # ...
}

Note line 19: this is the external API of my flake in use. The consumer picks which plugin they want to use and then calls one of the mk... functions to create the desired configuration.

Modules

These functions serve as only the first half of my external API. The use of a Nix module defines the other half. I found this to be a staple across many Nix projects that I came across while developing this flake. Perhaps the most confusing bit is that nothing in the documentation points to using modules as a form of structure for an external API. Indeed, they seem relegated to mostly system configuration within NixOS. Yet, many projects I came across seemed to use them as one would use classes or structs in a typical programming language.

Whether it is right to use modules in this way is likely up for debate. However, given the overwhelming usage of them in this way across various projects, I think it's a mostly settled one. In Nixago, I defined a single module that serves as the contract for what the various mk... functions return. One of the biggest problems I ran into is, unlike other languages, that there's no concept of private or public attributes. For example, some of the attributes are used only internally (i.e., shellHookExtra), but they are still accessible externally. I was okay with accepting this and decided to clarify what attributes the consumer should be concerned with in the documentation.

The primary benefit of using modules is creating a natural definition for your API outside of what flake outputs are available. They abstract away any internal details from the consumer and provide a clean interface for accessing the information required. In my case, the configFile attribute provides the derivation which builds the configuration file, and the shellHook attribute provides the hook necessary for installing the configuration in the local environment. In this way, an external consumer only needs to be worried about which function to call and where they will put the shell hook – the rest of the internal details are abstracted away.

A secondary benefit to modules is that they are self-documenting. The type and description fields, together, describe to an end-user how each attribute is used. It's possible to point a user to a module definition and have them walk away with a decent idea of how they can use it.

Together, the flake outputs and module form the concrete external representation of my flake. I believe this to be a solid base for growing a project and is a combination that I will continue to use in my future Nix projects.

Internal Functions

The second component of a flake library is the internal structure which does the necessary work being abstracted away by the external API. Once again, when one sits down to start thinking about how to structure this, Nix does very little in guiding you in one particular direction. Indeed, this area received the most refactoring in my attempts at solving it. The remainder of this section will detail the approach I took. Note that it's very opinionated. However, I believe it to be a solid foundation to work from.

Structure

Below is the structure that I ended up with:

  • lib/*
  • modules/*
  • plugins/*
  • tests/

Most of these are self-explanatory, but the lib folder serves as the primary location of most of the internal work. The other alternatives I came across were nix or a folder named after the project. In some cases, extensive projects use both (i.e., mach-nix). For me, both alternatives were too ambiguous. The usage  lib speaks pretty broadly across the software ecosystem as the central place for internal logic (i.e., Rust uses lib.rs as the entry-point for library crates).

The modules folder is more ubiquitous across Nix projects. As noted earlier, the name is self-explanatory, the location where module definitions are contained. Seeing as I only have a single module, I could have included this under lib. However, I felt it was better to maintain consistency with what other projects were doing.

The plugins folder is, of course, unique to my project. I opted not to include this under lib because this forms a higher layer of structure above the internal functions. The contents of the plugins folder are directly accessible from the flake output and serve as the primary entry point into the external API. The flow from internal to external is something like this: lib -> modules -> plugins. The laxness of Nix comes in handy here as I was able to define a structure that worked best for my project.

One last note to make is that I opted to expose lib as a flake output. The reasoning behind this is that I appreciate developers who provide an onramp to the internals of a library for creating something that the external API might not explicitly support. It's provided as-is, with little documentation, but the source code is sufficient to speak to what it contains.

Passing around objects

One of the most challenging obstacles I had to overcome when developing the internals of the flake was figuring out how to pass around internal objects. This primarily meant a copy of nixpkgs and the internal lib set that was used throughout my code. Since almost all of the utility functions in Nix are contained within nixpkgs.lib, it's necessary to ensure that most functions have access to it. However, with a flake, you can't just import <nixpkgs> due to the constraint of maintaining purity. Unfortunately, this nature leaves the burden on the developer to figure out how to pass around a copy of nixpkgs without unnecessary duplication.

The approach I settled on came from another project, which, unfortunately, I don't recall the name of at the time of this writing. The principle it follows isn't complicated. Here is an example of one of my internal functions under lib:

{ pkgs, lib }:
all:
with pkgs.lib;
let
  plugins = import ../plugins { inherit pkgs lib; };
  makeAll = name: data: (
    let
      # If the input is not in the `plugin.function` format, assume we want
      # `plugin.default`
      path =
        let s = splitString "." name;
        in
        if (builtins.length (s) > 1) then s else [ name "default" ];

      make = getAttrFromPath path plugins;
    in
    make data
  );

  result = mapAttrsToList makeAll all;
in
{
  configs = catAttrs "configFile" result;
  shellHook = concatStringsSep "\n" (pkgs.lib.catAttrs "shellHook" result);
}

Pay particular attention to line 1: here, we ingest an attribute set as a function argument which is expected to have the pkgs and lib attributes. In this case, pkgs is a copy of nixpkgs and lib is a copy of the internal lib attribute set from my flake. At the outset, this appears to be circular importing, which should result in an infinite recursion (the above all.nix is exposed in the lib attribute set). However, thanks to Nix's lazy evaluation, we can get away with it.

Putting it together, all.nix actually returns a function with two arguments: the attribute set and all. The all argument is the actual argument with any natural consequence as it controls what the function returns. To fully understand what's happening here, we have to take a peek into the main flake.nix at the project root:

# Load lib functions
lib = (import ./lib { inherit pkgs lib; });

As noted earlier, the internal lib set is exposed as a flake output. However, note how it's being exposed: all functions contained within lib/* are imported with the first argument already provided. If you examine the files in the lib folder, they all take this attribute set as their first argument. The result is that other areas of the code need not be worried about figuring out where pkgs and lib are coming from. I can call directly into the mkAll function shown above without worrying about it:

# ...
lib.mkAll configurations
# ...

You'll see this pattern repeated all over the internals of the flake. It allows the flake.nix to be the primary holder of nixpkgs while still allowing other functions to call each other without having to worry about sourcing it themselves.

Tests

The final component of a flake library is tests. I opted to call this component out in particular because I believe it's often overlooked. I won't reiterate the importance of writing tests, but suffice to say that it's an excellent service to your downstream contributors to ensure that the external API you are providing works the way it's expected.

Once again, though, we find ourselves stuck in the ambiguous mire of developing Nix projects. Unfortunately, no single framework or entity exists for testing Nix projects. Worse, I could not find any standardized methodology beyond the brief support given in the flake schema. As we will see, the flake schema tells you where to put tests, but it says nothing about how to write tests.

Testing Flakes

What exactly should a Nix test accomplish? As I mentioned before, I think the primary thing is verifying that the external API behaves correctly. Thus, it's crucial to establish what that behavior is before you can write tests for it.

How to write tests

In the case of Nixago, the API provides a derivation that, given a specific input, should produce a configuration file with a particular output. Thus, testing this involves calling the plugins with predefined information and comparing the result to a predefined outcome. Since this is essentially true for all plugins, I decided to abstract this into a single test runner function:

{ pkgs, plugins }:
path: expected: config:
let
  parts = pkgs.lib.splitString "." path;
  plugin = builtins.elemAt parts 0;
  make = pkgs.lib.getAttrFromPath parts plugins;
  output = make config;

  result = pkgs.runCommand "test.${plugin}"
    { }
    ''
      cmp "${expected}" "${output.configFile}"
      touch $out
    '';
in
result

The internals are not too important, but essentially it takes a relative path to the plugin function to test (i.e., pre-commit.mkConfig), the path to a file that contains the expected output, and then the raw configuration data to use in the test. The critical thing to note here is what's returned: a derivation. The pkgs.runCommand function is a small wrapper that runs a set of bash commands in an isolated environment. Like any derivation, it expects $out to be populated at the end. We employ a bit of trickery in this area: we're not interested in producing an output here. All we're interested in doing is failing the test if the generated output does not match the expected outcome. Luckily, any exit status other than 0 from the derivation will show itself as a failure to Nix. With this in mind, we do a cmp of the generated and expected output and then run a touch $out so that Nix doesn't complain the derivation produced nothing.

From what I've found, this sleight of hand is not uncommon. It more or less lets you perform any test you desire so long as it can be done in Bash (which, albeit not my favorite language, is still very versatile!).

Structure

There remains the question of how to structure the tests folder. Again, it's ambiguous at best, and I think the format largely depends on the number of tests you have and how they are operating on your flake. In my case, I have a natural structure around a plugin ecosystem, so my tests folder contains subfolders that contain tests for each plugin. For example, here is the test for the pre-commit plugin:

{ runTest }:
runTest "pre-commit.mkConfig" ./expected.yml {
  repos = [
    {
      repo = "https://github.com/my/repo";
      rev = "1.0";
      hooks = [
        {
          id = "my-hook";
        }
      ];
    }
  ];
}

The runTest argument is simply a copy of the function I showed previously. All tests follow this exact format, which allows me to define a default.nix in the tests directory that looks like this:

{ pkgs, runTest }:
{
  conform = pkgs.callPackage ./conform { inherit runTest; };
  just = pkgs.callPackage ./just { inherit runTest; };
  lefthook = pkgs.callPackage ./lefthook { inherit runTest; };
  pre-commit = pkgs.callPackage ./pre-commit { inherit runTest; };
  prettier = pkgs.callPackage ./prettier { inherit runTest; };
}

Adding additional tests is as simple as creating a new subfolder with the associated tests and then appending it to default.nix. They are then all tied together in flake.nix:

runTest = import ./tests/common.nix { inherit pkgs plugins; };

# ....

# Local tests
checks = import ./tests { inherit pkgs runTest; }

The checks output is a part of the flake schema and is what gets evaluated when one calls nix flake check from the CLI. It takes an attribute set of name/derivation pairs (which, in our case, we created in the default.nix file shown previously).

The result is that testing the flake becomes trivial: run nix flake check from the CLI, and all included tests will be executed. This usage is, of course, by design. When writing tests, you should make every effort to follow the provided schema of the flake by having them executed with nix flake check.

Conclusion

A lot of ground was covered, but hopefully, at the end of this, you will have a better understanding of how a flake library can be built using Nix. This is, of course, not the only way, but I do believe it represents some of the core concepts I found across multiple projects.

I would encourage you to review the project in its entirety. It's small enough to serve as a great entry point into writing fake libraries. As a bonus, you can take a look at the CI pipeline which demonstrates how to run automated checks in a Nix environment. A lot of this is also covered by nix.dev.

Flakes have a very hopeful future. While they've not tackled all of the problems Nix currently has, they at least provide a building block for composing larger projects together more concretely. As they continue to evolve, my hope is the community will begin settling on idiomatic ways for writing flake libraries that make creating them a lot less painful.

Show Comments