Repository navigation
nvx-rs: final acceptance and measurements for the image-backed runtime #624
Description
Parent: #535. Phase: A6, after A1-guest, A1-host, A3, A4 and A5. Tests stay owned by their phases; A6 collects the final evidence, runs the measurement campaign and re-pins bundled. Delivers: the cold part of #545 and the acceptance part of #541.
Goal. Prove that NVX's adapter and the in-repo test adapter run the shared host and guest code on KVM, MSHV and WHP, and agree budgets from measurements that don't mix new configurations into before-and-after comparisons.
Today (dev ff95973):
- CI runs microVM correctness and benchmark jobs on self-hosted Linux/KVM, Linux/MSHV and Windows/WHP runners, which are virtual machines (
ci.yml:960-1056). Performance jobs use the onebenchmark-labelled runner per pool (ci.yml:1151-1274). benchmark.pymeasures OpenVMM launch to a console marker, teardown, and the VMM process's peak RSS, not the full runtime contract. The stored results keep p50 only.- The gate compares a candidate's p50 with the median of the last ten base-branch p50s, and fails only when the increase is above both 50% and 10 ms (
ci.yml:24-27). A metric with too little history is labelled warmup. - Public lifecycle tests cover EROFS over ext4 scratch.
Scope:
- Conformance:
- A versioned coverage manifest maps every retained A1, A3, A4 and A5 requirement, and the retained cold requirements of nvx-rs: provide a standalone image-backed sandbox cold-start path #541 and nvx-rs: prove shared host and guest mechanics through two independent adapters #545, to its tests. Missing vectors go back to the phase that owns them.
- All 24 positive cold cells pass: both adapters, both formats, both uppers, all three backends. Negative vectors run where they apply.
- Consumer-contract scenarios C0–C8 pass, including A5's wire coverage for both roots.
- Both adapters run the same pinned NVX code through public APIs, each with its own matched host and guest artifacts.
- The CLI's Stop bypasses the queue and never escalates; other library callers may request forced termination. Recovery means an orchestrator restart within the same host boot.
- Comparison inputs:
- The starting commit is
devjust before the first A1 PR merges, recorded here when that happens. The result commit and the measurement-driver commits are recorded too. - Both commits are measured at the end, with matched artifacts and separate state directories.
- The common reference case is identical for both: the same EROFS bytes, fresh scratch, workload, identity, limits, mappings, network policy and cache regime. A starting point is today's 256 MiB, non-root fixture with no maps and no network. VMM, kernel and toolchain differences are recorded.
- Each commit gets its own measurement driver; the result keeps no legacy code for it.
- GPT, tmpfs, the test adapter, the admission queue and credit streaming are new: they get budgets of their own and are marked "not supported" for the starting commit, never compared with it.
- The starting commit is
- Seven measurements, with stated boundaries and a host-monotonic event log:
- Cold start: Start invocation to validated Ready, p50 and p95, with VMM spawn and Start completion recorded separately, and mapping time after Ready reported so that moving work past Ready can't look like a gain.
- First exec: the first submission after Start to the first workload marker, and Ready to that marker, p50 and p95, in a fresh VM with no earlier warmup exec.
- Exec control: later exec start, cancel through settlement, and Stop through verified exit and cleanup, p50 and p95, with an active exec and with waiters queued. Outcomes are not pooled.
- Streaming: delivered stdin, stdout and stderr throughput for fixed windows, payloads and sink rates, plus bounded memory and control progress under slow or blocked consumers.
- Memory: host process roles sampled together (the aggregate is the peak of simultaneous samples), and guest tmpfs use under a controlled write workload, with cgroup accounting and swap policy recorded.
- Sizes: each stripped guest ELF, their total, and the compressed and uncompressed initramfs, with build settings. The test adapter's artifacts are reported separately.
- Artifact reads: logical content bytes read from large artifacts, by artifact, caller and phase. The shared runtime reads zero content from staged artifacts; metadata access and the checked handoff are allowed. Staging, CLI preflight and VMM or guest disk I/O are reported in separate lanes. A positive control proves the counter works, and incomplete observation counts as unknown, never zero.
- Hosts, statistics and budgets:
- The three CI benchmark runners plus one dedicated WHP host with exclusive measurement windows. Both commits run on the same host, and each host's series stays separate.
- After A5, a calibration run sets per-case budgets, agreed in this issue; a separate confirmation run then checks them.
- Starting protocol: 5 discarded warmups and at least 200 measured runs per latency case, commit and host, over at least three interleaved sessions. Confidence intervals are reported, and a rule fixed in advance extends the run or declares it inconclusive.
- The gate learns case identity (adapter, format, upper, geometry, workload, cache regime), p95, sample counts and per-case budgets. Missing rows, warmup history and inconclusive runs don't pass.
- Automation and evidence: build checks in Check nvx-rs, conformance in Run NVX microVM tests, and measurements in Run NVX platform, reused on the dedicated WHP host. Raw results, coverage manifests and budget decisions are published durably, for passing runs too.
bundledre-pin:nvx-rs/artifacts.jsonpoints at the release built from the final merge, and acceptance also runs with that bundled release.
Not in scope: bare-metal KVM or MSHV measurements; restore and platform clones (meta B); image slots; #547's virtio-fs topology change; opened-object receipts; old runtime formats and NVXC v1.
Acceptance:
- Every required cell and applicable vector passes with exact source and artifact provenance. Ignored or skipped tests, and runs against an older bundle, are not passes.
- Both adapters demonstrably run the shared code, including a real cold launch, exec and teardown.
- Measurements use the stated boundaries and comparable inputs; unsupported starting-commit cases are explicit.
- Confirmation results meet budgets agreed before acceptance, with adequate precision.
- The zero-read checks pass with their positive control, and the CLI's verification stays visible and correct.
- CI's coverage and performance gates use the new case-aware results, and the evidence is published durably.
bundledserves the final matched release.
Carries over: the cold conformance and measurement obligations of #545, and the cold acceptance of #541 retained in this issue. Restore obligations stay in meta B.
Meta A: #535. Phases: A1-guest #619, A1-host #620, A3 #621, A4 #622, A5 #623, A6 #624.
Activity
esaurez commentedon Oct 20, 2026
A6 baseline: the first A1 PR to merge was #627, at 2026-10-20 12:28 UTC. The
devcommit just before that merge, 99f4c1c, is the starting commit for the measurements and the like-for-like comparisons.