vfs: preserve lazy artifact integrity across hosts

[brandonpayton]

Contract

Lazy VFS metadata is authoritative system state. A successful HTTP response must not replace a declared executable with unrelated, truncated, partially stored, or otherwise wrong-sized bytes, and Node/browser hosts must expose the same post-materialization VFS state.

Root cause

The browser runner requested a lazy ABI 18 executable, but Vite returned its 200 HTML fallback for the missing asset. MemoryFileSystem trusted HTTP success and replaced the stub without comparing the decoded body to the declared size, so exec failed later as ENOEXEC. Adjacent v1 integrity gaps also allowed malformed section lengths/inode mappings, unbounded declarations and ZIP inflation, short VFS writes to be reported as success, and concurrent resolutions of one inode to race. Node executed lazy bytes out of band without materializing the VFS, diverging from the browser.

The first full merge-gate run exposed a second state contradiction. The staged shell image correctly carried /usr/bin/dash as an empty lazy stub, with /bin/dash as a symlink. Browser live setup then opened /bin/dash with O_TRUNC and wrote concrete bytes, but MemoryFileSystem retained the inode's lazy metadata. Serializing that impossible combination caused the new strict importer to reject the image. A successful O_TRUNC now detaches the resolved inode from its lazy backing. Replacing one member of a lazy archive also removes only that member from the pending group, so materializing a sibling cannot overwrite the local replacement. Truncation to zero has the same safe transition; nonzero truncation is unchanged because it requires preserving the existing prefix.

Change

• Treat lazy sizes as exact decoded logical bytes, capped at 1 GiB, and bound HTTP, local-file, metadata, and ZIP decode paths before allocation.
• Strictly validate v1 lazy metadata sections, entry counts, canonical entry paths, URLs, inodes, file types, empty stubs, collisions, truncation, and trailing bytes.
• Require full VFS writes; roll file/archive stubs back on short write or ENOSPC and preserve retry metadata.
• Verify every decoded archive member and coalesce concurrent materialization per inode/archive group.
• Materialize successful Node lazy execs into the same VFS state used by browser exec.
• Detach pending lazy metadata when local code intentionally replaces an inode through O_TRUNC or truncation to zero.
• Make mkrootfs require/cap the exact logical size and make the benchmark derive it from fetched bytes rather than transfer headers.
• Add real browser kernel/process-worker coverage for guest-visible EIO followed by a successful retry.

ABI and artifacts

No shared ABI shape or syscall behavior changed. ABI_VERSION remains 18 and abi/snapshot.json is unchanged. This tightens validation and state transitions for v1 images rather than changing the v1 format. No package binaries or VFS images were rebuilt. The exact staged ABI 18 shell image was inspected before browser boot: /usr/bin/dash was a valid empty stub with a 459,958-byte declaration, and all 77 lazy entries imported successfully.

Validation

• ./scripts/dev-shell.sh npm --prefix host run build
• focused host/VFS/Node integration before the follow-up: 94 passed
• final lazy VFS/archive/image regression suite: 92 passed
• full default host Vitest run: 115 files passed, 11 skipped; 892 tests passed, 29 skipped, 2 expected failures
• host declaration build: npm --prefix host run typecheck
• Playwright real kernel/process-worker integrity gate: 6 passed across Chromium, Firefox, and WebKit
• exact prepared CI workspace: all 91 browser asset imports resolved
• exact previously failing Chromium shell and modeset tests: 2 passed
• mkrootfs suite: 134 passed
• ./scripts/dev-shell.sh bash scripts/check-abi-version.sh
• exact ABI 18 shell-image compatibility probe
• git diff --check
• independent devil's-advocate review of c69948e9c: approved with no remaining blockers

The exact browser evidence above intentionally uses Kandelo's staged ABI 18 release packages because this PR tests platform VFS assembly, not Homebrew bottle pouring. Homebrew Node and Chromium bottle acceptance remains a separate distribution gate. The full repository/conformance matrix and a manual full browser-demo walkthrough were not run locally; CI remains the broader evidence gate. The prior Linux Vitest failure at teardown-reclaim.test.ts:71 is the fixed-delay readiness race addressed independently by #889, not a lazy VFS failure.

[github-actions]

Phase B-1 matrix build status — pr-897-staging

ABI v18. 6 built, 0 failed, 6 total.

Package	Arch	Status	Sha
kandelo-sdk	wasm32	built	03c5fe40
rootfs	wasm32	built	0bbbbc02
shell	wasm32	built	db390c62
lamp	wasm32	built	07ea08f6
node-vfs	wasm32	built	b2ed8f02
wordpress	wasm32	built	77da1ebb

_{Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.}

[brandonpayton]

CI follow-up at exact head 8df8489bc: the browser smoke workflow now resolves only the three ABI-declared artifacts used by the real-kernel integrity test (kernel, dash, and hello) through xtask build-deps --fetch-only. The first rerun proved kernel resolution and then exposed the remaining hidden dependency on locally generated examples/*.wasm; the test now uses published ABI 18 Dash to lazily exec published ABI 18 GNU Hello instead. No source fallback or broad registry fetch is permitted.

The larger Hello response also proved that browser fetch progress chunk counts are transport-specific. The assertion now preserves the exact lifecycle order while coalescing consecutive progress notifications.

Local evidence through scripts/dev-shell.sh: all three packages resolved from binaries-abi-v18; the real-kernel test passed 6/6 across Chromium, Firefox, and WebKit on an isolated Vite port; git diff --check passed; worktree clean. CI is rerunning on the pushed head.

[brandonpayton]

Follow-up root-cause fix is now at exact head c69948e9cc49b6b4173ab9a6e07108159558e7af.

The staged shell artifact was valid before browser assembly: /usr/bin/dash was an empty 459,958-byte lazy declaration. stageShellUtilities wrote through /bin/dash -> /usr/bin/dash with O_TRUNC, while MemoryFileSystem incorrectly retained the lazy metadata; the worker then correctly rejected the concrete-file-plus-lazy-record contradiction. The shared VFS now detaches that resolved inode (and an individually replaced archive member) after a successful full truncation.

Exact-head evidence: focused lazy VFS/archive/image suite 92/92; full default host Vitest 892 passed with 29 skipped and 2 expected failures; host declarations built; the exact prepared CI workspace resolved all 91 browser imports; the previously failing Chromium shell and modeset tests both passed. Independent devil review APPROVED after narrowing ftruncate detachment to length zero so nonzero prefix-preservation is not falsely claimed.

The earlier Linux teardown-reclaim.test.ts:71 failure is the independent fixed-delay readiness race addressed by #889. This browser validation uses Kandelo ABI 18 release packages by design; Homebrew bottle pour acceptance remains a separate gate.

[brandonpayton]

Fresh-head CI at c69948e9cc49b6b4173ab9a6e07108159558e7af completed the VFS/browser-relevant gates successfully: browser suite, COI smoke, shell/rootfs/node-vfs/LAMP/WordPress/SDK matrices, fork-instrument, kernel, and POSIX all passed. The Vitest job ran 894 passing tests and failed only host/test/teardown-reclaim.test.ts:71 (exits.has(pid)), the same fixed-delay readiness race independently repeated on #887 and #885 and addressed by green PR #889. This is not a lazy-VFS integrity failure; further reruns before #889 lands would exercise the same known race.