[Enrique Saurez (esaurez)]

Summary

Pins OpenVMM to nanvix/openvmm#116 (67b641d8), which adds bind-once read-only virtio-blk image slots to the microVM profile as part of ABI 2, and adopts them in NVX.

nanvix/openvmm#116 is based on OpenVMM main f4566bab, the revision dev already pins (it includes the merged #124 and #125), so this promotion adds exactly #116's four commits:

• f5c8dab0 virtio_blk: add bind-once image slots
• 0a3dc07e microvm: add bind-once read-only image slots
• 4eb09487 microvm: fold image slots into ABI 2
• 67b641d8 microvm: harden host-control framing, logging, and cleanup

No other fork changes are included. Dependency state: nanvix/openvmm#116 is open.

• ABI: image slots extend microVM ABI 2 in place. Every machine and snapshot keeps ABI 2, so SOURCE-MANIFEST.json, aci_edge_sandboxes, and the benchmark history dimensions are unchanged. A machine that does not declare slots keeps its device topology. Image slot 0 reuses the MMIO window and status word of the second virtio-fs share, and slot 1 reuses its IRQ 13, so a slot-declaring launch accepts at most one --mount.
• CLI and harness: --image-slot-boot-count for managed sandboxes, --restore-image-slots, query-image-slots, and bind-image-slot. Slot launches may omit the distro layer; other launches keep the existing layer requirement. A managed configuration that declares slots uses configuration format 6, which earlier releases reject.
• Host control: authenticated protocol-v1 client, using the existing 32-byte capability and the owner-restricted local endpoint. Only a slot-declaring sandbox records a host-control endpoint; slot operations on another sandbox fail at once, and stop and deprovision remove the socket.
• Guest:
  • kernel shared-status mapping for slots 1 to 3; slot 0 uses the second share's word;
  • nvx-time reads the image-slot target from restore packet v4 (flag bit 4, header byte 7), validates it, and passes it to nvx-snapshot. It never finishes a restore with a target by itself;
  • restore-time prefix activation before acknowledgement;
  • capacity/config-change discovery, partition rescan, and stable read-only /dev/nvx-imageN links. Placeholders stay unbound, and nvx-image-slot fails at once for an inactive slot or a window that holds a live share. Managed sandboxes probe slots only when the launch declares them. The Azure Linux initramfs also installs nvx-image-slot.
• CI: cross-backend image-slot correctness, with every console monitored for the time ABI:
  • boot with one active slot, then bind and read;
  • idempotent bind, and rejection of a different identity;
  • restore to four active slots, then bind and read slots 1, 2, and 3, so every slot's IRQ and shared-status word delivers a configuration change.
• Benchmarks: benchmark --suite image-slot-boot compares launches without slots and with one or four active slots.
• Docs: machine and device ABI, restore packet, snapshot and restore, limits, remaining work, run/usage/setup guides, implementation status, and benchmark methodology.

Rebase onto dev

This is one commit on dev 3468901f; the previous head was 59219c57 on b1b5c5bc.

• Pin: dev moved the pin to Harden source provenance integrity #124's head and then to main f4566bab once Harden source provenance integrity #124 and ci: materialize Windows release provenance from run head #125 merged. guest: add Ubuntu userland support #116 was rebased onto that main, and this commit pins its head, so no dev change is dropped.
• ABI: guest: add Ubuntu userland support #116 now folds image slots into ABI 2. This commit therefore drops every ABI-3 change: the manifest version, the aci_edge_sandboxes 2-or-3 acceptance, the per-suite benchmark ABI, and the ABI-3 documentation. The benchmark series are no-slots, slots-b1, and slots-b4.
• Capability pipe: dev now hands OpenVMM its control capability in a prepared pipe, because OpenVMM reads it without blocking as soon as it starts (Managed start can fail when OpenVMM reads its control capability before NVX writes it #440). The image-slot scenario and the boot-cost benchmark wrote it after the spawn, so they now use capability_pipe() as well.
• Conflicts: documentation (limits, restore-packet flags, setup lint lists), test imports, and sandbox_lifecycle.start's failed-start cleanup, which keeps dev's outcome reporting and also removes the host-control socket.

Review fixes:

• deprovision refused a stale host-control.sock left by a killed slot sandbox, and stop did not remove it.
• The Docker-built Azure Linux initramfs lacked nvx-image-slot, which the default suite's image-slot scenario needs.
• A slotless managed launch recorded a host-control endpoint that never existed, so slot operations on it retried until their timeout.
• bind-image-slot accepted identities up to 4096 bytes; OpenVMM accepts 1 to 1024.
• The image-slot scenario now binds and reads slots 1 and 2 as well as slot 3 after the restore, and its different-identity bind must fail with already_bound.
• bind-image-slot no longer prechecks the media path, which broke idempotent recovery after the path was removed. It passes the absolute path, and OpenVMM validates the media only when the slot is empty.
• The host-control client has socket-backed tests for its requests, error codes, oversized frames, malformed JSON, mismatched responses, and invalid success payloads.
• Documentation: the goals and machine-ABI pages no longer imply a separate slot ABI, the --layer row matches the CLI, the run guide notes that earlier releases reject format 6, the filesystem architecture describes the scratch-only root and the single live share of a slot launch, and the time ABI separates an image-slot-only restore from a restore with nothing to activate.

Validation

Linux (Pedro, Azure Linux 3.0, Python 3.12), on c793a0e0 (9393894d changes only documentation): the nine unittest modules of the CI CLI validation passed 807 tests (34 skipped); test_adversarial.py, test_hosts.py, and nvx.py verify passed.

Linux/MSHV, with OpenVMM 67b641d8: the image-slot runs used 874fd540; the other runs used 3b885041, which differs only in the scenario's stricter rebind check and the bind-image-slot client. The head 9393894d differs from 874fd540 only in dev's documentation and baseline-data commits, the host-control client tests, and documentation. Host: AMD EPYC 9V74, Linux 6.6.148-200.mshv-b1f02da.azl3.

• build-openvmm --backend mshv (musl, Rust 1.95.0): OpenVMM SHA-256 b2c02064…. build-guest for Alpine and Azure Linux: initramfs d5773270… and 0565e5fd….

• This host's hypervisor cannot present the built-in amd.genoa.v1 profile that --cpu-profile auto selects (it lacks tsa_l1_no_support and tsa_sq_no_support). A test-only wrapper therefore passed --cpu-profile host to cold microVM boots. Restores, which keep the snapshot's profile, and the cpu-profile-fallback scenario ran unchanged.

• test-microvm --scenario image-slots passed on Alpine and on Azure Linux: boot with B = 1, bind, idempotent rebind, rejection of a different identity, and a guest read; then capture an empty-slot snapshot, restore with --restore-image-slots 4 (NVX-IMAGE-SLOTS-ACTIVE-OK: count=4 before acknowledgement), and bind and read slots 1, 2, and 3. A deliberately wrong expectation for slot 2 failed the scenario.

• test-microvm --scenario managed-lifecycle and the default suite (39 runs, 362 s, including cpu-profile-fallback and the L3/L4 egress protocol and port-range acceptance) passed.

• benchmark --suite image-slot-boot --processors 1 --warmups 3 --runs 21, cold boot to ALPINE-MICROVM-BOOT-OK:

  Configuration	p50	p95	Δ p50 vs no slots
  no-slots	234.4 ms	236.4 ms	—
  slots-b1	237.6 ms	240.7 ms	+3.3 ms
  slots-b4	240.5 ms	244.7 ms	+6.1 ms

  Configurations ran in consecutive blocks, not interleaved.

Local (Windows), on the same change before the last dev rebase: ruff check, ruff format --check, pyright (Linux and Windows), shellcheck, shfmt, and nvx.py verify are clean. The nine unittest modules ran 838 tests; the 74 failures and errors are exactly those of dev on the same machine. New tests fail without their fixes and pass with them (host-control endpoint, deprovision of a stale socket, and both capability-pipe launchers). On c793a0e0, ruff, pyright, and nvx.py verify are clean (and nvx.py verify on 9393894d), and the test_control_session.py and test_microvm_tests.py failures on this machine are exactly those of dev. The new host-control tests fail when the client's request-ID, response-size, or request-size check is removed.

OpenVMM 67b641d8: Windows nextest passed 583 tests; Linux cargo test passed 621. Clippy, fmt, and rustdoc are clean (details in nanvix/openvmm#116).

CI on 9393894d: the KVM, MSHV, and WHP microVM jobs (including the image-slot scenario), platform jobs, OpenVMM unit and vmm-tests, NVX CLI on Linux and Windows, and the performance regression gate passed. The lint job could not pull its pinned shellcheck image from Docker Hub (toomanyrequests) on three attempts; its checks pass locally (ruff, compileall, shellcheck, pyright for Linux and Windows, ruff format, shfmt, the PowerShell parse, and the Specula tests on Linux).

Not covered here: no boot populates every IRQ line at once.

[Copilot]

🟡 Changes recommended

Standard benchmarks are mislabeled as ABI 3, and stale host-control sockets can prevent sandbox deprovisioning.

2 open findings

• Keep e2e result metadata at ABI 2 · New
• Clean up stale host control sockets during deprovision · New

What changed in this PR

Adopts OpenVMM’s bind-once image-slot ABI and integrates it across NVX guest tooling, managed sandboxes, testing, benchmarks, and documentation.

Changes:

• Pins OpenVMM image-slot support and adds ABI-3 guest activation and host control.
• Extends sandbox, restore, CI, and benchmark workflows.
• Documents the new ABI, constraints, and operational model.

File	Description
openvmm	Pins the image-slot OpenVMM revision.
SOURCE-MANIFEST.json	Advertises microVM ABI 3.
scripts/​nvx.py	Adds image-slot CLI operations.
scripts/​nvx_tools/​sandbox.py	Configures sandbox image slots.
scripts/​nvx_tools/​sandbox_lifecycle.py	Persists slots and exposes host control.
scripts/​nvx_tools/​host_control.py	Implements the host-control client.
scripts/​nvx_tools/​control_session.py	Shares local-stream transport logic.
scripts/​nvx_tools/​microvm_tests.py	Adds cross-backend slot scenarios.
scripts/​nvx_tools/​benchmark.py	Adds image-slot boot benchmarking.
scripts/​nvx_tools/​build.py	Installs the guest slot helper.
scripts/​nvx_tools/​build_constants.py	Updates the supported ABI version.
scripts/​test_nvx_tools.py	Tests CLI, lifecycle, and benchmarks.
scripts/​test_microvm_tests.py	Tests restore activation and helper behavior.
scripts/​test_guest_time.py	Tests restore-packet slot targets.
kernel/​patches/​0003-microvm-virtio-mmio-shared-status.patch	Maps shared status for four slots.
guest/​common/​nvx-image-slot	Discovers stable read-only slot devices.
guest/​common/​nvx-time.c	Parses restore image-slot targets.
guest/​common/​nvx-snapshot	Activates slots during restore repair.
guest/​common/​nvx-init-agent	Supports slot-aware scratch-only workloads.
guest/​common/​nvx-virtio-restore-probe	Adds image-slot restore probing.
.github/​actions/​run-benchmark/​action.yml	Runs image-slot benchmarks.
.github/​actions/​check-quality/​action.yml	Lints the new shell helper.
doc/​usage.md	Documents CLI options.
doc/​run.md	Documents runtime workflows.
doc/​setup.md	Updates validation commands.
doc/​build.md	Updates manifest ABI documentation.
doc/​benchmarks.md	Documents benchmark methodology.
doc/​design/​time-abi.md	Defines the restore-packet field.
doc/​design/​snapshot-and-restore.md	Defines restore-time activation.
doc/​design/​machine-and-device-abi.md	Defines ABI-3 slot topology.
doc/​design/​current-limits.md	Records image-slot limits.
doc/​design/​goals.md	Updates ABI goals.
doc/​design/​integrated-implementation-status.md	Updates implementation status.
doc/​design/​remaining-production-work.md	Distinguishes slots from general volumes.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Benchmark ABI metadata, Azure Linux helper packaging, and stale host-control socket cleanup are currently incorrect.

4 open findings

• Install nvx-image-slot in the Azure Linux Docker stage · New
• Keep e2e result metadata at ABI 2
• Separate package ABI from ordinary launch ABI · New
• Clean up stale host control sockets during deprovision

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Host-control endpoint persistence and socket cleanup currently break normal ABI-2 queries and Linux slot-sandbox deprovisioning.

4 open findings

• Install nvx-image-slot in the Azure Linux Docker stage
• Avoid recording host-control endpoint when listener is not created · New
• Separate package ABI from ordinary launch ABI
• Clean up stale host control sockets during deprovision

1 resolved since last review

• Keep e2e result metadata at ABI 2

🧠 Review effort: Balanced

[Pedro Henrique Penna (ppenna)]

IRQ budget: this spends the last dedicated lines

I checked whether the image slots overlap with, or run us out of, dedicated IRQ lines when other devices, notably virtio-fs, are in use. They never overlap, but this design consumes essentially every remaining dedicated IRQ line on the microVM.

The MP table routes only ISA IRQs 0, 1 and 3–15, and every virtio line is edge-triggered, so devices can't safely share a line. After the PIT (0) and the RTC (8), 13 lines remain for virtio devices. This PR changes these:

ISA IRQ	ABI 2	ABI 3 (this PR)
1	free	image slot 0
13	second virtio-fs share	image slot 1
14	free	image slot 2
15	free	image slot 3
5 or 10	free (whichever virtio-net doesn't use on that backend)	still free

• 12 of 13 lines used. A fully populated ABI-3 machine (virtio-net, first share, console, control console, four sandbox disks and four image slots) uses 12 of the 13 lines. The spare is IRQ 5 on KVM/MSHV and IRQ 10 on WHP, so no line is left that is free on every backend.
• Reserved even when idle. All four slot IRQs stay reserved with --image-slot-boot-count 1, because a restore may activate up to four slots.
• Already costing a feature. Slot 0 takes the second share's MMIO window and status word, and slot 1 takes its IRQ 13, so a VM with image slots gets only one virtio-fs share.
• No room to grow. Any future fixed device (vsock, RNG, a second NIC, more slots, or a second share next to the slots) has no line left. Getting more lines, for example by routing IOAPIC pins 16–23 through the MP table (the IOAPIC already has 24 pins), would be yet another machine-ABI change.

Machine ABI

This also bumps the microVM machine ABI to 3. New guest-visible addresses, IRQs and command-line tokens require a new ABI version, so any design that adds fixed transports will. We'd rather not bump the machine ABI for this, so it's worth exploring whether image delivery can fit in ABI 2 without new guest-visible transports.

What holds up

• Overlap is rejected at every layer. OpenVMM refuses image slots plus a second share in both discovery and machine validation, which the worker runs on every launch, restores included. An ABI-2 snapshot can't activate slots, and NVX rejects more than one --mount with slots.
• Nothing in the guest competes for the new lines. IDE/ATA, parallel port, serial, ACPI and PCI are off. The PS/2 driver is built in but never claims IRQ 1, because unclaimed ports read 0xFF and it finds no controller.

Test gaps

• No CI boot populates every line. The image-slots scenario boots only the console, the dormant first-share slot and the slots. Managed sandboxes with slots are covered only by unit tests.
• Two of the new lines never fire. The scenario binds and reads only slots 0 and 3, so slots 1 and 2 (IRQs 13 and 14) never raise an interrupt.

Merge conflict

dev has moved the OpenVMM pin to 3e1f5bd4 (WHP CPUID change). This PR's 5ffe5c22 is based on 762bc1c7 and lacks that commit, so the promotion needs rebasing onto 3e1f5bd4. Resolving the gitlink in this PR's favor would drop that change.

[Copilot]

🟡 Changes recommended

The goals document still assigns slot-declaring machines ABI 3, contradicting the implemented ABI-2 contract.

1 open finding

• Correct slot-declaring snapshot documentation to ABI 2 · New

4 resolved since last review

• Install nvx-image-slot in the Azure Linux Docker stage
• Avoid recording host-control endpoint when listener is not created
• Separate package ABI from ordinary launch ABI
• Clean up stale host control sockets during deprovision

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

It changes the guest-visible ABI and kernel/device lifecycle while its upstream dependency remains open and cross-backend CI is still pending.

0 open findings

2 resolved since last review

• Clarify targetless versus image-slot-only restore behavior
• Document image-slot exception to the two-share limit

🧠 Review effort: Balanced

[Pedro Henrique Penna (ppenna)]

Heads-up for your next rebase onto dev: #576 (merged) dropped the runtime and custom layers, and guest/common/nvx-init-agent now sets lowerdirs=$layers/distro unconditionally.

That breaks the scratch-only image-slot boot here without a textual conflict. The agent merges cleanly, but if [ -n "$lowerdirs" ] is then always true. With no nvx_layer= token, mount_layer never creates $layers/distro, so a zero-layer boot fails with failed to assemble the container overlay instead of taking the mount --bind "$scratch/upper" branch.

When rebasing, set lowerdirs only when the distro layer is mounted:

lowerdirs=
if [ "$distro" = true ]; then
    lowerdirs=$layers/distro
fi

The same zero-layer case shows up as a textual conflict in scripts/nvx_tools/sandbox.py: SandboxLaunch.__post_init__ now requires exactly one distro layer, so the resolution needs to keep allowing no layers when image_slot_boot_count is set.

[Enrique Saurez (esaurez)]

Closing: the image-backed runtime plan in #535 uses role-keyed blocks and leaves multiple image slots out of scope. The branch stays, so existing pins remain fetchable.