Feature Request: Bring nix builds to buildkit

[blaggacao]

I'd like to propose to bring the capability of doing nix builds into buildkit.

nix builds propose a convincingly saner alternative of building container images, see this talk: https://www.youtube.com/watch?v=pfIDYQ36X0k

I'd be interested in preliminary feedback if / how that could be done.

[AkihiroSuda]

You can build Dockerfile with FROM nixos/nix. Does it work for you?

[blaggacao]

Ah, sorry! I wasn't clear.

What I'm interested in doing is have buildkit build something like using the nix builder

{
  redis = buildImage {
    name = "redis";
    tag = "latest";

    # for example's sake, we can layer redis on top of bash or debian
    # fromImage = bash;
    # fromImage = debian;

    contents = pkgs.redis;
    runAsRoot = ''
      mkdir -p /data
    '';

    config = {
      Cmd = [ "/bin/redis-server" ];
      WorkingDir = "/data";
      Volumes = {
        "/data" = {};
      };
    };
  };
}

The buildImage directive builds a single layered redis container with all runtime dependencies, and nothing more. It's basically a "single" binary container (+ runtime dependencies) int the root filesystem.

It leverages nix's ecosystem knowledge of how to build and run redis and puts that knowledge into a container.

It eliminates the need for builder containers or apt-get cleanups or - even - for alpine itself: it efficiently produces single layered single binary containers without the need to know how to actually package them (thanks to the huge nix ecosystem fundus).

I would imagine:

something loosely like:

Nixfile

# syntax=nix/nixfile:master

{ pkgs, buildImage }: {
# `buildkit` - the magic attribute for buildkit to consume, since there can be various outputs, but buildkit presumes one `*file` - one artifact.
buildkit = buildImage {
    name = "redis";
    tag = "latest";

    contents = pkgs.redis;
    runAsRoot = ''
      mkdir -p /data
    '';

    config = {
      Cmd = [ "/bin/redis-server" ];
      WorkingDir = "/data";
      Volumes = {
        "/data" = {};
      };
    };
  };
}

[tonistiigi]

Yes you should be possible to do this as a frontend. The efficiency depends on how much work is put into it. The simplified version is similar to the Dockerfile approach @AkihiroSuda suggested, just the user does not need to write a Dockerfile.

You might also want to take a look at https://github.com/talos-systems/bldr/ https://github.com/sipsma/bincastle for a take on natively defining build dependencies/rules in a similar fashion.

[tonistiigi]

the magic attribute for buildkit to consume, since there can be various outputs, but buildkit presumes one *file - one artifact.

Actually, we have --target for that where you can specify the name.

[blaggacao]

I was having a closer look at the current nix dockerfile tooling: it uses a VM for the construction of the file system. I'd imagine this to cause trouble when running form nixos/nix. - What's sort of the best argument in-depth argument against using a VM - if there is?

[nlewo]

@blaggacao it only uses a VM when you need to run commands as root user. But in practice, you rarely need this. This means we generally don't rely on a VM to build Docker images.
I didn't read the whole thread but building Docker images with Nix from a container has an important drawback: you can't (easily) use the Nix store (/nix/store) where everything is cached.

[blaggacao]

Docker images with Nix from a container has an important drawback: you can't (easily) use the Nix store (/nix/store) where everything is cached.

A show stopper. @tazjin has build some interesting caching strategies into nixery (which also caches the completed image layers (noct obly the nix store results) based on the requested packages. Maybe in his codebase are clues how to map a nix derivation (after going through the layered image "popularity" algorithm to content adressable container layers.

Code entrypoint for layer caching: https://github.com/google/nixery/blob/ba1d80323175b61c4f6348827daa8254e3aa13a5/config/pkgsource.go#L35-L41

[tonistiigi]

You can probably used cache mounts if you want to keep things in nix store over multiple builds. A more complicated version likely doesn't even need that and could connect straight to instruction caching. Eg. examples I posted above don't use cache mounts afaik.

[sipsma]

Eg. examples I posted above don't use cache mounts afaik.

Bincastle's current approach doesn't involve cache mounts and should hopefully be formalized upstream once buildkit has support for merge-op (right now it's sort of a hacked version of merge-op).

It's been a long time since I've looked at how nix works underneath the hood so I don't have any concrete ideas. That being said, I quickly skimmed this blog post linked to in the nixery repo and, on the surface, it seems like a lot of the issues involved here might be positively impacted by the existence of an efficient merge-op.

Just to give an example, in bincastle the equivalents of packages are all single layer images arranged in a DAG that get topologically sorted and efficiently merged together via merge-op make a final rootfs. I'd be curious to hear if anyone with more knowledge of nix internals thinks a similar approach would work with nix and allow the nix-store to be easily used. Not sure what the issues preventing use of the nix-store actually are.

[blaggacao]

Or maybe it's enough/better to implement nixc, a runner that scaffolds containers upon reception of a manifest from something like ipfs, see https://notes.burke.libbey.me/nix-squashfs/ - with some crfs features. - just throwing in loose ideas.

[AkihiroSuda]

My experimental Nix frontend for BuildKit: https://github.com/AkihiroSuda/buildkit-nix

cd examples/nginx
export DOCKER_BUILDKIT=1
docker build -t nginx-nix -f default.nix .