Location of configuration

[dmvianna]

I have deployed a VM to GCP. It works fine.

Now I want to change its configuration without redeploying. I just want to use sudo nixos-rebuild switch. But /etc/nixos/configuration.nix just has

{ ... }:
{
  imports = [
    <nixpkgs/nixos/modules/virtualisation/google-compute-image.nix>
  ];
}

Where can I find a copy of the configuration I used within the VM? Is it anywhere there? Can I just place a new configuration anywhere and do the switch?

[liamdiprose]

The file is being imported from the nixpkgs repository. This is directory where the google-compute-image.nix file lives:
https://github.com/NixOS/nixpkgs/tree/master/nixos/modules/virtualisation

You can leave the import statement there, and add (or overwrite) the options you want to change.

[dmvianna]

So that means there's no declarative configuration within the VM? And what I add to /etc/nixos/configuration.nix will be added to the current configuration, rather than clobbering it?

[phonkd]

I have the same question.

[Mic92]

You have to either manually update /etc/nixos/configuration.nix to be in sync with what you have used creating the image, or you just ignore this file completely and use sudo nixos-rebuild switch --flake .#myvm --target-host root@ip on your local machine update your VM.

[ODAncona]

@Mic92 This solution imply that you have an external machine that will rebuild the configuration from distance.
Just let say I have 32 VMs and I just want to change a minor parameter such as the hostname. This would completely overwhelm the network ? Is it not possible to save the configuration directly ?

I was able to copy the configuration to my vm with

  # ~~~ Flake Copy ~~~
  system.activationScripts.saveFlakeConfig = {
    deps = [];
    text = ''
      rm -rf /etc/nixos/current-systemconfig
      mkdir -p /etc/nixos/current-systemconfig
      cp -rf ${self}/* /etc/nixos/current-systemconfig/
      cd /etc/nixos/current-systemconfig
      chown -R ${username}:users /etc/nixos/current-systemconfig
      chmod -R u=rwX,g=rX,o=rX /etc/nixos/current-systemconfig
    '';
  };

However, I do have a problem with filesystem

       error: The option `fileSystems."/".device' has conflicting definition values:
       - In `/nix/store/a59nywcf635014w55rhzddgajjd2rz2r-source/scheduler_benchmark/nix/vm-config.nix': "/dev/vda3"
       - In `/nix/store/zfr9q0312byaiphnmdknqj792aci3654-source/formats/qcow.nix': "/dev/disk/by-label/nixos"
       Use `lib.mkForce value` or `lib.mkDefault value` to change the priority on any of these definitions.

I believe it looks like I'm doing something wrong here. I feel that I shouldn't override the configuration in nixos-generator instead to have a nice solution to update the configuration of my vms at runtime ? What is the clean solution to modify the configuration in the generated VM ?

Because as the nix/store is mounted in read-only I couldn't just change my parameters and restart apps that relied on hostname e.g.

[odancona@nixos:~]$ sudo hostnamectl set-hostname compute-node-1
Could not set pretty hostname: Changing system settings via systemd is not supported on NixOS.

What is your take on this ? How can we configure VMs runtime with nixos-rebuild switch using a nixos-generator generated image ?

---

*edits:

I was able to execute sudo nixos-rebuild switch --flake /etc/nixos/current-systemconfig#k8s-worker directly on the VM

We only need to override the configuration of nixos-generators

  fileSystems."/" = lib.mkForce {
    device = "/dev/vda3";
    fsType = "ext4";
  };

I still believe that something is wrong but it worked for me.

[ODAncona]

For clarity here is my full nix config:

Directory tree

nix
├── flake.nix
├── modules
│   └── kubernetes
│       ├── master.nix
│       └── worker.nix
└── vm-config.nix

nix/flake.nix:

{
  description = "NixOS Cluster VMs";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/24.11";
    nixos-generators = {
      url = "github:nix-community/nixos-generators";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, nixos-generators }:
  let
    system = "x86_64-linux";
    definitions = {
      k8s-master = [ ./vm-config.nix ./modules/kubernetes/master.nix ];
      k8s-worker = [ ./vm-config.nix ./modules/kubernetes/worker.nix ];
      # slurm-master = [ ./vm-config.nix ./modules/slurm/master.nix ];
      # slurm-worker = [ ./vm-config.nix ./modules/slurm/worker.nix ];
    };
  in
  {
    packages.x86_64-linux =
      nixpkgs.lib.mapAttrs
        (name: modules:
          nixos-generators.nixosGenerate {
            inherit system modules;
            format = "qcow";
            specialArgs = { self = self; };
          }
        )
        definitions;

    nixosConfigurations =
      nixpkgs.lib.mapAttrs
        (name: modules:
          nixpkgs.lib.nixosSystem {
            inherit system modules;
            specialArgs = { self = self; };
          }
        )
        definitions;
  };
}

nix/modules/kubernetes/master.nix:

{ config, lib, pkgs, ... }:

let
  apiserverHost = "k8s-master";
  apiserverFQDN = "${apiserverHost}.local";
in
{

  # ~~~ Packages ~~~
  environment.systemPackages = with pkgs; [
    kompose
    kubernetes
  ];

  # ~~~ Networking ~~~
  networking.useDHCP = true;
  networking.hostName = apiserverHost;
  services.kubernetes.masterAddress = apiserverHost;
  services.kubernetes.apiserverAddress = "https://${apiserverFQDN}:6443";
  networking.firewall.enable = true;
  networking.firewall.allowedTCPPorts = [
    22      # SSH
    6443    # Kubernetes API Server
    2379    # etcd server client API
    2380    # etcd peer communication
    8888    # kube-scheduler
    10250   # Kubelet API
    10251   # kube-scheduler
    10252   # kube-controller-manager
  ];
  networking.firewall.allowedTCPPortRanges = [
    { from = 30000; to = 32767; }  # NodePort services
  ];

  # ~~~ System Configuration ~~~
  services.kubernetes = {
    roles = ["master"];
    easyCerts = true;
    addons.dns.enable = true;
    kubelet.extraOpts = "--fail-swap-on=false";
    apiserver = {
      securePort = 6443;
      extraSANs = [ apiserverFQDN ];
    };
    # Overload the flannel.nix to setup certificates
    pki.certs.flannelClient = config.services.kubernetes.lib.mkCert {
      name = "flannel-client";
      CN = "flannel-client";
      hosts = [ apiserverHost apiserverFQDN ];
      action = "systemctl restart flannel.service";
    };
    pki.cfsslAPIExtraSANs = [ apiserverHost apiserverFQDN ];
  };

  # Kubectl setup
  environment.sessionVariables = {
    KUBECONFIG = "/etc/kubernetes/cluster-admin.kubeconfig";
  };

}

nix/modules/kubernetes/worker.nix:

{ config, lib, pkgs, ... }:

let 
  kubeMasterFQDN = "k8s-master.local";
in

{

  # ~~~ Packages ~~~
  environment.systemPackages = with pkgs; [
    kubernetes
  ];
  
  # ~~~ Networking ~~~
  #networking.hostName = hostname;
  networking.firewall.enable = true;
  networking.firewall.allowedTCPPorts = [
    22      # SSH
    10250   # Kubelet API
  ];
  networking.firewall.allowedTCPPortRanges = [
    { from = 30000; to = 32767; }  # NodePort services
  ];

  # ~~~ System Configuration ~~~
  services.kubernetes = {
    roles = ["node"];
    masterAddress = kubeMasterFQDN;
    apiserverAddress = "https://${kubeMasterFQDN}:6443";
    easyCerts = true;
    addons.dns.enable = true;
  };
}

nix/vm-config.nix:

{ config, pkgs, lib, self, ... }:

let
  username = "odancona";
  password = "$y$j9T$u8873E9Qlnqv13YKSP2NB/$pggfnnIDxsnoUVr9uXXYRqAdajpKDn5la2UShIw36z.";
  rhodeyPublicKey = "";
  x1CarbonPublicKey = "";
in
{
  # ~~~ System Configuration ~~~
  boot.kernelPackages = pkgs.linuxPackages_6_1;
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
  services.openssh.enable = true;
  swapDevices = [];
  system.stateVersion = "24.05";
  virtualisation.diskSize = 16384; # 16GB
  environment.systemPackages = with pkgs; [
    vim curl tree btop bat fastfetch git jq openssl
  ];
  nix.settings.experimental-features = [ "nix-command" "flakes" ];
  fileSystems."/" = lib.mkForce {
    device = "/dev/vda3";
    fsType = "ext4";
  };

  # ~~~ Networking Configuration ~~~
  networking.useNetworkd = true;
  systemd.network.enable = true;
  services.avahi = {
    enable = true;
    nssmdns4 = true;
    publish = {
      enable = true;
      addresses = true;
      domain = true;
      workstation = true;
    };
  };

  # ~~~ User Configuration ~~~
  time.timeZone = "UTC";
  users.users.${username} = {
    isNormalUser = true;
    initialHashedPassword = password;
    extraGroups = [ "wheel" ];
    openssh.authorizedKeys.keys = [ rhodeyPublicKey x1CarbonPublicKey ];
  };
  security.sudo.wheelNeedsPassword = false;

  # ~~~ Flake Copy ~~~
  system.activationScripts.saveFlakeConfig = {
    deps = [];
    text = ''
      rm -rf /etc/nixos/current-systemconfig
      mkdir -p /etc/nixos/current-systemconfig
      cp -rf ${self}/* /etc/nixos/current-systemconfig/
      cd /etc/nixos/current-systemconfig
      chown -R ${username}:users /etc/nixos/current-systemconfig
      chmod -R u=rwX,g=rX,o=rX /etc/nixos/current-systemconfig
    '';
  };

}

You can build the images using nix build .#k8s-worker or nix build .#k8s-master.

After provisioning, on the VM you can successfully rebuild the image:

On the worker:
sudo nixos-rebuild switch --flake /etc/nixos/current-systemconfig#k8s-worker

On the master:
sudo nixos-rebuild switch --flake /etc/nixos/current-systemconfig#k8s-master

---

Question

I still have the same question:

I need to build a worker image for n kubernetes nodes and the only setting that change is the hostname. Therefore, I don't want to build n images for my n workers. I would prefer to have the option to keep the configuration.nix in the generated VM. What would be the canonical way to do this ?

It feels wrong to copy the configuration manually even though it works ...