Skip to content

host: opt-in stop for orphaned microvm@<name>.service on removal from microvm.vms - #509

Open
brandonros wants to merge 4 commits into
microvm-nix:mainfrom
brandonros:fix/stop-orphaned-declarative-vms
Open

host: opt-in stop for orphaned microvm@<name>.service on removal from microvm.vms#509
brandonros wants to merge 4 commits into
microvm-nix:mainfrom
brandonros:fix/stop-orphaned-declarative-vms

Conversation

@brandonros

@brandonros brandonros commented Apr 22, 2026

Copy link
Copy Markdown

Summary

Closes the reconciliation gap where nixos-rebuild switch does not stop a microvm@<name>.service after <name> is removed from microvm.vms. Addresses the stop-on-removal goal from #302 (@astro: ""I'd accept PRs that stop declarative VMs on removal""), scoped to stopping only — state under /var/lib/microvms/<name>/ is preserved per the same comment.

Tracked as #508.

Opt-in by default. New behavior is gated behind microvm.host.stopOrphans which defaults to false — existing users see no change. Users who want NixOS-style reconciliation for their declarative VMs set microvm.host.stopOrphans = true;.

Why restartIfChanged / the prior #302 discussion doesn't cover this

The fix briefly discussed in #302 (flipping restartIfChanged to match oci-containers) never shipped, and would not have fixed this bug. X-RestartIfChanged controls restart on config change; stop-on-removal is a separate path in switch-to-configuration-ng (collect_unit_changes, src/main.rs:1078-1089) that checks whether the service's base unit file disappeared.

For microvm@<name>.service, the base unit file is the template microvm@.service, which the host module unconditionally defines at nixos-modules/host/default.nix:240. So new_base_unit_file always exists and the stop-on-removal branch never fires — regardless of restartIfChanged. The .wants symlinks under microvms.target.wants/ are not examined either (no references in the Rust source).

microvm.vms.<name>.restartIfChanged already exists as a per-VM option (#104, #110); that half of the #302 agreement effectively landed. The stop-on-removal half is the remaining gap.

What this PR does

Adds system.activationScripts.microvm-stop-orphans to nixos-modules/host/default.nix. When enabled, on each activation it:

  1. Lists install-microvm-*.service files in /run/current-system/etc/systemd/system/ — these are the VMs that were declarative in the previous generation.
  2. For each, checks if its <name> is still in attrNames config.microvm.vms of the new generation.
  3. If not, and microvm@<name>.service is currently active, calls systemctl stop microvm@<name>.service.

Using install-microvm-<name>.service as the declarative-VM marker is important: imperative VMs managed via the microvm command have no install-microvm-<name>.service unit, so they are never touched by this reconciliation.

What this PR explicitly does NOT do

  • Does not change default behavior — opt-in via microvm.host.stopOrphans = true;.
  • Does not remove /var/lib/microvms/<name>/ or any VM state, matching the concern in #302. State cleanup remains manual, as documented in doc/src/microvm-command.md.
  • Does not affect imperative MicroVMs, since they lack the install-microvm-<name>.service marker.
  • Does not touch Restart=always, restartIfChanged, or any other lifecycle semantics.

Files changed

  • nixos-modules/host/default.nix — new activation script (16 lines, inert when the option is false).
  • nixos-modules/host/options.nix — new microvm.host.stopOrphans option (default false).
  • doc/src/declarative.md — new ""Reconciliation on host rebuild"" section, framed as opt-in.
  • doc/src/microvm-command.md — split ""Removing MicroVMs"" into Imperative / Declarative / State cleanup subsections; declarative path mentions the opt-in.

Test plan

I don't have a NixOS test host to iterate on, and local evaluation of the full flake checks exhausts my machine's memory, so this relies on CI. Happy to add a nixosTest in a follow-up — the natural shape is to declare two VMs with microvm.host.stopOrphans = true;, switch to a specialisation that drops one, and assert microvm@<dropped>.service is inactive while /var/lib/microvms/<dropped>/ remains.

  • CI evaluation passes
  • Manual verification on a NixOS host with microvm.host.stopOrphans = true; set
  • nixosTest added in follow-up (if desired)

🤖 Generated with Claude Code

… microvm.vms

When a VM is removed from microvm.vms.<name> and nixos-rebuild switch
is run, the corresponding microvm@<name>.service instance keeps
running. switch-to-configuration detects service removal by looking
for the base unit file's disappearance, but the base file for
microvm@<name>.service is the template microvm@.service, which the
host module unconditionally defines — so the removal branch never
fires. The .wants symlink under microvms.target.wants is not
inspected either.

Add system.activationScripts.microvm-stop-orphans, gated behind a new
microvm.host.stopOrphans option (default false — opt-in, no behavior
change for existing users). When enabled, it iterates
install-microvm-*.service files in the previous generation
(/run/current-system/etc/systemd/system) and stops any
microvm@<name>.service whose <name> is no longer declared. Using the
install-microvm-<name>.service marker scopes the reconciliation to
previously-declarative VMs only; imperative VMs started via the
microvm command are never touched.

State under /var/lib/microvms/<name>/ is preserved, per the position
stated in microvm-nix#302 — VMs may be only temporarily deactivated.

Refs: microvm-nix#508, microvm-nix#302

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@brandonros
brandonros force-pushed the fix/stop-orphaned-declarative-vms branch from e34c7f3 to fa52032 Compare April 22, 2026 13:48
@brandonros brandonros changed the title host: stop orphaned microvm@<name>.service on removal from microvm.vms host: opt-in stop for orphaned microvm@<name>.service on removal from microvm.vms Apr 22, 2026
Comment thread nixos-modules/host/default.nix Outdated
Comment thread nixos-modules/host/default.nix Outdated
Comment thread nixos-modules/host/default.nix Outdated
Comment thread doc/src/microvm-command.md Outdated
Comment thread doc/src/microvm-command.md Outdated
Comment thread doc/src/microvm-command.md Outdated
Comment thread doc/src/declarative.md Outdated
Comment thread doc/src/declarative.md Outdated
Comment thread doc/src/declarative.md Outdated
Comment thread doc/src/declarative.md Outdated
- Use lib.concatMapAttrsStringSep instead of concatStringsSep + attrNames.
- Use lib.getExe' config.systemd.package "systemctl" instead of
  hardcoding ${pkgs.systemd}/bin/systemctl, so the configured
  systemd package is used. Note: getExe' (not getExe), since
  systemd's meta.mainProgram is unset and getExe would resolve to
  the `systemd` binary, not `systemctl`.
- Reformat new doc sections to one sentence per line per project
  style, plus small wording tightens.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@brandonros

Copy link
Copy Markdown
Author

Thanks for the review @SuperSandro2000! All ten suggestions applied in 2f61b71 with one minor correction:

  • systemd → systemctl lookup: used lib.getExe' config.systemd.package "systemctl" rather than lib.getExe config.systemd.package. getExe falls back to ${systemd}/bin/systemd (not systemctl) because the systemd package has no meta.mainProgram set — locally I confirmed this with a deprecation warning. getExe' lets us specify the binary explicitly, which matches what's actually wanted here.
  • concatMapAttrsStringSep: used (name: _: name) as the mapper function so it correctly picks out the attribute names.
  • All doc suggestions applied verbatim; the rest of the new sections reformatted to one sentence per line per the style rule.

Comment thread nixos-modules/host/default.nix Outdated
Comment thread nixos-modules/host/default.nix Outdated
Per review from @astro: switch the orphan-detection source from
filesystem introspection of the previous generation to systemd's
in-memory state via `systemctl list-unit-files`.

During activation scripts, /etc already reflects the new generation
(etc activation runs first), but systemd has not yet been told to
daemon-reload — so `systemctl list-unit-files
'install-microvm-*.service'` returns the previous generation's
unit files. This removes the need to reach into
/run/current-system/etc and simplifies the script.

Also: drop the `lib.getExe' config.systemd.package "systemctl"` wrapper
in favor of bare `systemctl`, which is in the activation script
`$PATH` as @astro notes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@brandonros

Copy link
Copy Markdown
Author

Thanks @astro — both addressed in a2de3b2. The activation script now uses systemctl list-unit-files 'install-microvm-*.service' to find previously-declarative VMs (systemd's in-memory state is still pre-daemon-reload during activation scripts, so it reflects the old generation without needing to touch /run/current-system), and uses bare systemctl throughout. Net diff vs prior commit: -14/+11 lines — the new version is meaningfully simpler. Ready for another look whenever you have a moment.

Comment thread nixos-modules/host/default.nix Outdated

system.activationScripts.microvm-stop-orphans = lib.optionalString config.microvm.host.stopOrphans ''
declared=" ${lib.concatMapAttrsStringSep " " (name: _: name) config.microvm.vms} "
for unit in $(systemctl list-unit-files --no-legend 'install-microvm-*.service' 2>/dev/null | awk '{print $1}'); do

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with LLM-generated code and PRs, such as this one, is that they don't work:

...
/nix/store/a0kzb6wmdkp6h9f6z6cs367iw7f44f6f-nixos-system-jupiter-26.05.19700101.dirty/activate: line 130: awk: command not found
...

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's see if a6c3690 fixes it

awk is not on the activation script's PATH, causing "awk: command not
found" during system activation. Parse the unit name with bash builtins
via `read` from a process substitution instead.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants