Repository navigation
Shared image-backed sandbox runtime #535
Description
Objective. nvx-rs provides composable host and guest libraries. They are used by NVX's own adapter (the nvx-rs CLI and the NVX guest agent, over NVXC, NVX's workload-control protocol) and by a small in-repo adapter with its own transport. Adapters own product configuration, image resolution, authorization, application framing and admission. The libraries own runtime records, launch recovery and the execution mechanisms.
NVXS is the outer authenticated record session between host and guest. NVXC and the test adapter's application framing both run over it. MAPS delivers ordered host mapping entries.
A cold sandbox has one read-only distro, which is either an EROFS filesystem or a GPT disk with ordered read-only lowers. Its writable upper is either a bounded tmpfs or ext4 scratch. The sandbox supports:
- host-enforced mappings and network policy;
- authenticated readiness;
- recovery after an orchestrator restart within the same host boot;
- bounded guest logs;
- stop and teardown.
NVX's adapter runs one active exec at a time, with bounded waiting and a priority Stop. It streams stdin, stdout and stderr with credits and bounded memory, and has no total output cap. Shared provision, start and exec paths never hash large content; the CLI keeps explicit distro verification in its run and provision preflight.
Cold-path order:
- Authorize the attachments and install the VMM policy.
- Boot: time step, root assembly and guest settings.
- Authenticated Ready.
- Ordered MAPS.
- Start completes.
- Exec.
Zero mappings do no mapping work. A failed mapping leaves no usable Start.
API acceptance. The independent adapter builds with default features off and uses documented public APIs only. It supplies its own init, status collector and application framing, and runs concurrent executions without changing NVX's admission policy. A CI public-API check guards the adapter-facing surface, and a PR that changes the adapter contract carries a migration note and conformance cases.
Consumer-contract suite. Both adapters run the shared scenarios. Concurrent-exec, root-identity and pooled frame-credit cases use the test adapter. Most faults run on one representative configuration; both adapters run the full format, upper and backend matrix.
| ID | Scenario | Added in |
|---|---|---|
| C0 | Independent build: its own lockfile, default features off; protocol-only, host-only and guest-only builds; a static musl guest; the declared MSRV | A1 |
| C1 | NVXS with independent framing: partial records, record lengths, bootstrap identity, ERROR payloads, sequence errors, disconnect and reset, bounded buffers, application messages that span records | A1, A5 |
| C2 | Concurrency and backpressure: 16 executions over 32 streams with reserved control capacity; a blocked sink stalls no other execution; pooled frame credit; stdin beyond a window; EOF, cancel, reset, timeout and dropped handles | A1, A5 |
| C3 | Root and isolation: the four format and upper combinations; root and non-root profiles; per-execution namespaces; unrelated child status preserved; no inherited port I/O | A3, A4 |
| C4 | Mappings and enforcement: zero and ordered maps after Ready; a failed map blocks Start; descriptor containment; flags set before attachment; #552; adapter-chosen protected roots; network deny, allow, proxy and loopback | A3, A4 |
| C5 | Staged artifacts: a manifest present with content reads forbidden; a stale seal fails without hashing; replacement between check and open; the Linux inherited descriptor; the Windows guard lifetime | A4 |
| C6 | Crash recovery and admission: a crash at each launch step; the log claim; reattach to a non-child; a wrong identity is never signaled; the cross-process queue | A1-host, A5 |
| C7 | Runtime identity and live upgrade: two orchestrator builds on the new store; the same runtime identity reattaches; a changed one stops only the verified VM; a wrong guest build fails at Ready | A1-host, A4, A5 |
| C8 | Single-container non-regression: the seven metrics; zero maps and no network do no optional work | A6 |
Out of scope:
- platform clones (meta B);
- multiple image slots and device hotplug;
- TAP networking;
- exec concurrency in NVX's adapter, TTY and arbitrary signals;
- capability discovery (a matched bundle implies its features);
- the Python-parity layering inside nvx-rs.
Assumptions, verified at ff95973:
| # | Assumption | Evidence |
|---|---|---|
| A-1 | The three phases of #356 have merged: #357, #358 (#561) and #359 (#594–#598). #356 itself is still open, and #601 keeps the direct-launch test debt. | issue states |
| A-2 | Source-built guests ship nine Rust binaries, each of which exports only main(). Shell still owns init, root assembly and the restore steps. |
nvx-rs/src/guest/mod.rs:59-72 |
| A-3 | The layered root has one EROFS distro and requires ext4 scratch. The /run tmpfs is not an overlay upper. GPT is not supported. |
guest/common/nvx-init-agent:438-479 |
| A-4 | Exec is synchronous and coupled to NVXC: stdin is null, and stdout and stderr together are capped at 1 MiB. Reaping takes any child. Whole-cgroup settlement happens only in direct mode; layered mode kills the process group. Mapping targets are absolute paths. | managed_agent/{exec,launch,cgroup,mappings}.rs |
| A-5 | The library plans EROFS, scratch and shares. The layered lifecycle is CLI-private, and host::instance uses the blockless profile. |
host/sandbox/mod.rs:1-18; host/launch.rs:1-6; host/cli/sandbox/lifecycle.rs |
| A-6 | NVXC and NVXS are both v1. The CLI sends FEATURES, but its layered requirement is only EXEC_ENVIRONMENT. |
protocol/features.rs:53-63 |
| A-7 | run and provision hash the whole distro when a manifest sits beside it; start does not. The hash binds the package list that the systemd check reads. |
host/sandbox/options.rs:355-417,463-480 |
| A-8 | OpenVMM d0bdbed8 runs the managed layered root, and already accepts a cold RAM-upper layout with one read-only distro. It rejects capture and restore of that layout. The VMM still accepts the runtime and custom roles (#556). |
OpenVMM openvmm_defs/src/microvm.rs:1179-1233; cli_args/microvm.rs:784-790 |
| A-9 | #591 is the proposed target: Ready means the root is assembled and control is up, MAPS follow Ready, Start completes after the mappings, and exec uses NVXC v2. The phase contracts below define workload identity and the cold-path attachment scope. | #591 |
| A-10 | The bundled feature stages the Oct 4 release v0.1.0-dev.9ceb89fcc31f, whose guest still has the C agent. |
nvx-rs/artifacts.json:3 |
| A-11 | The CLI buffers exec output in memory under the 1 MiB cap. | host/cli/session.rs:407-430 |
Phases. A1 comes first. A3–A5 can then overlap through agreed interfaces, and tests ship with each change.
| Phase | Scope | Carries over |
|---|---|---|
| A1-guest (#619) | Stack G1–G6. The guest role of a shared NVXS session that doesn't depend on NVXC. Seven API families, callable without NVX's PID 1: a per-execution exec driver (events, cancellation, backpressure, caller-reaped status), neutral spawn, one caller-owned reaper, cgroup handles, mappings with explicit roots, typed time and reseed with per-transaction ports, and the cold-boot time step. The guest half of the test adapter, with its CI wiring and change detection. | #537, #539, #358 (library entry points) |
| A1-host (#620) | Stack H1–H5. The host role of the shared NVXS session. The store, in one new versioned format, with a launch journal, a log claim and the runtime identity. The supervisor: event waits, a graceful stop with caller-selected forced termination, and reattach within the same boot. The CLI and host::instance delegate, and the CLI's stop is unchanged. The host half of the test adapter, and the public-API check. |
#538 (supervisor) |
| A3 (#621) | Guest cold runtime, as a small stack: the Rust PID 1 for the layered root and bootstrap; an EROFS or GPT distro over an ext4 scratch or tmpfs upper; the overlay default metacopy=on,xino=on with an untrusted-lowers opt-in; descriptor-rooted mappings; identity set by the adapter; a private UTS namespace and hostname, and the machine ID mode chosen by the adapter; no account creation in the layered root; zero and ordered maps, reserved targets, and fatal partial application. |
#539, #541 (guest), #566 |
| A4 (#622) | Host cold integration. PR 1 is the verification split and seal. Then: role-keyed blocks with an explicit format token for both formats and both uppers; the checked file handed to the VMM with no VMM change; one planner for mappings and network, keeping today's identity recheck; #552 refused in plan; reattach pinned to the runtime identity; OpenVMM rejecting command lines over 2,048 bytes (#570); the CLI's own protected-path check; and the test adapter running for real. |
#538, #541 (host), #552, #570, the cold block part of #548 |
| A5 (#623) | Exec v2. First, the rest of #581's single structural EXEC and MAPS decoder (#603 merged, #616 open). Then one framing-free credit and stdio engine shared with other adapters; NVXC v2 replacing v1 and FEATURES for both roots, with build identity validated at Ready; the bounded admission queue across CLI processes (8 waiters, 60 s default wait) with a Stop that bypasses it and never escalates; and CLI streaming. #513 is fixed outside A5, by open PR #585. | #537, #581 |
| A6 (#624) | Final acceptance and measurements. The conformance matrix runs on the CI runners (KVM, MSHV and WHP) with both adapters: all four format and upper combinations, lifecycle failures, map and network enforcement, and bounded memory. Seven metrics are measured for the starting commit and the result, on the CI runners plus one dedicated WHP host. The starting commit is dev just before the first A1 PR merges. Like-for-like cases are compared separately, and budgets are agreed before acceptance. A6 also re-pins bundled to the final matched release. |
#545 (cold), #541 (acceptance) |
Meta B, platform clones (restore from a tenant-free template): a separate meta issue, to be filed.
Activity
[-]Track a reusable trusted-sandbox runtime and standalone NVX profile[/-][+]Track a reusable image-backed sandbox runtime and standalone NVX implementation[/+]ppenna commentedon Oct 15, 2026
#529 replaced
doc/design/with the architecture guide and dropped the proposed designs, so this issue's link to NVX's filesystem/agent design now returns 404 ondev. The proposal it refers to is still available at the lastdevcommit before that change: Replaceable configuration region (Proposed).The architecture guide describes only the current design. For the configuration and control paths that the region would replace, see 2.1.2 Configuration Ownership and 4.4 Host-Guest Control Interface.
doc: resolve review findings in the platform-clone guide changes
doc: name external orchestrators and external guest agents
[-]Track a reusable image-backed sandbox runtime and standalone NVX implementation[/-][+]Shared image-backed sandbox runtime[/+]