[Discussion] Disko Use with PXE Deployments

[andar1an]

Preface

I am currently investigating (and enjoying) using nixos as my metal host OS, where I will deploy distributed hosts using netboot.

I truly appreciate any thoughts and discussion on this topic. Something I am passionate about and am working towards for my own infrastructure is a baremetal orchestration experience akin to Canonical's Metal as a Service, Equinix's Tinkerbell, or something like Sidero Metal but using nix.

I am trying to understand how disko may fit into my baremetal orchestration plan, and would appreciate any discussion pertaining to use. I am only about 1 month into my nix journey, so please excuse any ignorance in questions. If there is a better place to discuss this, please let me know.

Disko

I am currently working on generating an ISO with nixos-generator to deploy via some pxe server like pixecore or netboot, and would like help to understand how I may use disko in this process.

Questions

1. If using pxe is there a way to have no user input to have disko format drives during pxe boot or would a subsequent step like using nixos-anywhere be required? (I am trying to avoid the ssh requirement of nixos-anywhere and just deploy a built image)

2. Is it possible to not run formatting if drives are already formatted (e.g. just deploying a new build of image)? I currently have circumstances where filesystems are not ephemeral, and also where they are.

Potential Solutions

This morning I am wondering about using some script with disko and nixos-anywhere to install nixos minimally to format drives when needed, and then deploy a generated iso with pxe where fileSystems are then defined without disko, but mapped to what disko formatted in previous step.

Is this logical? Is there a better solution?

[andar1an]

This seems to be relevant to what I am working on: https://github.com/nix-community/disko/blob/master/docs/reference.md#generating-disk-images-with-secrets-included-using-disko

[Mic92]

If you pxe image already contain all services and you just need those disks for data, than you can also use fileSystems.<name>.autoFormat
to integrate those disks into the running system.

[Mic92]

My pxe setup for installation currently looks like this: nix-community/nixos-anywhere#50
The basic idea is to have a nixos installer being booted with pxe boot and connect to the same using SSH to finish the installation with nixos-anywhere.

[andar1an]

Thank you @Mic92! I will look into both of those options and do some tinkering.

One of my fears is if those disks already contain data, I will overwrite/wipe with a format.

For example, on some of my machines I will be hosting Minio and Gitea. It won't be as big a concern for me when I have multiple nodes clustered, but right now I just have the 1 Host for testing. I will hopefully have more in January if lead time works out.

[andar1an]

I also think I may be thinking about this wrong. On investigation this morning, I have bumped into a rebuild command like nixos-rebuild switch --use-remote-sudo --target-host example.com --build-host localhost -I nixos-config=./configuration.nix in a forum post.

This leads me to believe I can remotely rebuild and switch an os once initial installation is handled which could make things way less complex.

I have been playing with the thought of creating logic to just rebuild flakes when updates are needed. Still new in this thinking though.

[andar1an]

Seems like it is fairly straight forward to rebuild a remote host and switch!: https://nixos.wiki/wiki/Nixos-rebuild, though ssh is necessary can maybe use wireguard too. I will add this to tinker list, and see if I can use local configuration to rebuild the remote host; would make it easy to orchestrate updates from a single server. Ugh, I just keep getting more excited by Nix lol.

[Mic92]

Thank you @Mic92! I will look into both of those options and do some tinkering.

One of my fears is if those disks already contain data, I will overwrite/wipe with a format.

For example, on some of my machines I will be hosting Minio and Gitea. It won't be as big a concern for me when I have multiple nodes clustered, but right now I just have the 1 Host for testing. I will hopefully have more in January if lead time works out.

Than start the installation only manually or have a check if a system already has a partition table.

[znd4]

This seems to be relevant to what I am working on: https://github.com/nix-community/disko/blob/master/docs/reference.md#generating-disk-images-with-secrets-included-using-disko

unfortunately this link is dead, here's the new location