Eval-runner sandbox: Gondolin prototype findings + next steps

[getlarge]

Context

We want a lightweight, cross-platform sandbox for the moltnet eval runner (apps/moltnet-cli + libs/dspy-adapters/{claudecode,codex}) so the eval runner can exec Claude Code / Codex CLIs against generated code that may be adversarial, while:

• only seeing a scoped worktree (no host FS leakage),
• only reaching LLM provider APIs (no arbitrary egress),
• ideally not seeing the raw OAuth token (header-placeholder injection instead of file mount).

We prototyped Gondolin (experimental Linux micro-VMs with JS-programmable network stack + VFS) as an alternative to Docker, sandbox-exec, or bwrap + iptables.

Full writeup: .exploration/gondolin-prototype/FINDINGS.md.

TL;DR

• Core primitives validated. Placeholder auth, VFS scoping, per-host egress allowlist all work end-to-end against live Anthropic API using real Claude Code OAuth. Warm boot on M-series Mac: 28ms.
• Claude Code CLI runs sandboxed using only CLAUDE_CODE_OAUTH_TOKEN as a Gondolin-injected placeholder. Guest never sees real token. claude --print "2+2=" → 4.
• Custom image build is broken. Gondolin's zig-based guest-binary compile step fails non-deterministically on our target (aarch64-linux-musl, alpine 3.23). Upstream earendil-works/gondolin#68.
• Verdict: promising fit for moltnet evals, not production-ready today, but our eval runner isn't either. Worth continuing if we accept early-stage upstream dependency or vendor/fork the build pipeline.

Smoke test results

All against default alpine-base image, claude installed at runtime via npm.

#	Check	Result
1	Guest sees placeholder, not real token	✅ GONDOLIN_SECRET_fe23... only
2	/workspace mounted RW from host (RealFSProvider)	✅ Bidirectional
3	Egress to example.com denied	✅ HTTP 403 from policy
4	Egress to api.anthropic.com allowed	✅ HTTP 405 (wrong verb, but reached)
5	claude --print via placeholder auth	✅ → 4
6	Warm-cache boot time	✅ 28ms (cold with 89MB asset download: ~14s)

What didn't work

Alpine's npm package is broken (upstream, not Gondolin-specific)

apk add nodejs npm + npm i -g <anything> fails with TypeError: Class extends value undefined is not a constructor or null in minipass-sized — CJS/ESM interop bug in Alpine 3.23's npm 11.11.0. Reproducible in plain docker run alpine:3.23. Workaround: avoid Node entirely, ship native binaries.

gondolin build custom-image pipeline fails

Attempted to bake claude 2.1.110 + codex rust-v0.121.0 + moltnet cli-v1.25.0 native binaries into a custom image. Build runs a nested Docker container that compiles Gondolin's own zig guest binaries (sandboxd, sandboxfs, sandboxingress, sandboxssh) before our postBuild.commands run. That zig step fails with:

```
error: ld.lld: cannot open /work/zig-cache/global/o//libcompiler_rt.a: No such file or directory
error: sub-compilation of musl libc.a failed
note: failed to check cache: invalid manifest file format
```

• Non-deterministic: different sub-targets fail across runs.
• Known zig 0.15.2 parallel-build cache issue.
• Maintainer's workaround from #68 (GONDOLIN_GUEST_SRC=/path/to/clone/guest): same failure class.
• Cleaning stale tmpdirs: no help.

This blocks the "fast warm-boot with all tools preinstalled" end state.

Threat-model asymmetry: Claude vs Codex

Dimension	Claude Code CLI	Codex CLI
Auth surface	Env var CLAUDE_CODE_OAUTH_TOKEN → Authorization header	File: $CODEX_HOME/auth.json / ~/.codex/auth.json, fallback OPENAI_API_KEY
Gondolin integration	Header placeholder (strong): guest never sees real token	File bind-mount (weaker): guest reads real token
Exfiltration risk	Near-zero	Guest can exfil via /workspace writes unless mitigated
Mitigation	None needed	ShadowProvider over /workspace or the codex adapter's existing IsolateHome tmp-copy

We did not test the Codex path end-to-end — the custom-image block stopped us before baking codex into an image.

Key Gondolin facts learned

• Not truly cross-platform: macOS + Linux only. No Windows. ARM64 best-tested.
• QEMU default (~15s cold with asset download, 28ms warm). Optional libkrun requires local zig build.
• HTTP/1.x + TLS-intercepted HTTPS only. No HTTP/2, HTTP/3, QUIC, WebRTC, UDP.
• Host process sees plaintext guest traffic (necessary for egress hooks) — host is in TCB. Fine for untrusted-guest/trusted-host, broken for mutual distrust.
• Secret placeholder substitution: request headers only. Not URL path, request body, or response content. Silently inert if a CLI puts the token anywhere else.
• OAuth rotation is not handled. Response-body token refresh would leak a real token to the guest. Not observed with claude --print but latent.
• VFS is the killer feature. RealFSProvider, ReadonlyProvider, ShadowProvider stacking gives fine-grained policy that bwrap --ro-bind can't match.
• Guest images are Alpine-only.

Decision matrix

Option	Effort	Security	Maturity	Cross-platform	Startup
Gondolin (VM)	medium-high; upstream flaky	strong (VM + policy)	experimental	macOS + Linux	28ms warm, ~15s cold
sandbox-exec + bwrap	medium; two backends	good (kernel syscall filter)	mature primitives, no unified API	macOS + Linux	near-zero
Docker sandboxes (current)	low	good (ns + cgroup)	mature	cross w/ Docker Desktop	1-3s
gVisor / Firecracker	high	very strong	mature	Linux-only	100s ms

Gondolin's unique value vs sandbox-exec+bwrap: programmable egress policy with secret-placeholder injection. Single feature hard to replicate natively.

What we didn't get to

1. Full eval-runner integration (actual moltnet eval run loop with multi-turn, file edits, git ops, heartbeat callbacks).
2. Codex path validation end-to-end.
3. moltnet CLI path validation.
4. ShadowProvider real-worktree test (hiding .git/config, .env, .claude/, ~/.ssh).
5. Boot-time with a working custom image (~500MB of binaries).
6. Parallel concurrent eval runs (N VMs on one M-series Mac, RAM profile).
7. HTTP/2 SSE streaming for long agent conversations.

Proposed next steps

Short term (hours)

• File the upstream zig-compile bug against gondolin build with concrete repro, linked to earendil-works/gondolin#68.
• Pivot to snapshot-based image: boot alpine-base, install tools at runtime via direct binary fetch (skip broken Alpine npm), gondolin snapshot, resume from snapshot. Unblocks prototyping while upstream settles.
• Wire a TS wrapper around VM.create accepting { worktree, prompt, tool: "claude" | "codex", model }.

Medium term (days)

• End-to-end eval run: one real scenario from evals/legreffier-commit-fix/ inside Gondolin. Compare to unsandboxed baseline.
• Benchmark parallel boot: 1, 4, 8, 16 concurrent VMs. RAM, CPU, per-VM boot.
• Write a ShadowProvider policy that hides secrets from worktrees and test against a worktree containing .env, .git/config, ~/.ssh.

Long term (weeks)

• Decide: bet on Gondolin (vendor/fork the build pipeline), or switch to native sandbox-exec+bwrap+iptables egress and accept losing placeholder-injection.
• If continuing: productize as libs/eval-sandbox/ with both Go and TS interfaces, docs, examples, CI coverage.

Artifacts

```
.exploration/gondolin-prototype/
├── package.json # pins @earendil-works/gondolin@0.7.0
├── run.ts # smoke-test harness
├── fake-worktree/prompt.txt # "What is 2+2?"
├── image/
│ ├── build-config.json # pinned claude 2.1.110 / codex rust-v0.121.0 / moltnet cli-v1.25.0
│ └── build.log # last failed build output
├── results.json # smoke-test results
├── README.md # quickstart
└── FINDINGS.md # full experiment writeup
```

Related diary entries

• procedural: Gondolin sandbox prototype: Claude Code CLI validated with header-placeholder auth (88771147-5c90-4e66-a507-843f6eb6a990)
• semantic: Gondolin facts: HTTP/1.x only, header-only secret substitution, Alpine-only guest, host in TCB (206bef35-c3d8-4c61-bfed-16e15d584294)
• reflection: Gondolin custom-image build pipeline blocked — implications for adoption path (ad2851b6-ee69-4e86-a9c2-f39e39036c55)

[getlarge]

Gondolin experiment: sandboxing the moltnet eval runner

Status: prototype validated for primitives, blocked on custom-image build tooling.
Date: 2026-04-16
Author: ed + Claude

Context

We want to run the moltnet eval runner (Go, apps/moltnet-cli + libs/dspy-adapters/{claudecode,codex}) against generated code that may be adversarial. The eval runner exec's Claude Code CLI and Codex CLI, which need network access to LLM provider APIs using credentials, while:

• only seeing a scoped worktree (no host filesystem leakage);
• only reaching LLM providers (no arbitrary egress);
• ideally not even seeing the raw OAuth token (header-placeholder injection instead of file-mount).

We explored Gondolin — experimental Linux micro-VMs with JS-programmable network stack and VFS — as a lightweight, cross-platform alternative to Docker sandboxes, macOS sandbox-exec, or Linux bwrap.

TL;DR

• Core sandboxing primitives work. Placeholder auth, VFS scoping, per-host egress allowlist all validated end-to-end against the live Anthropic API using real Claude Code OAuth. Warm boot on M-series Mac: 28ms.
• Claude Code CLI works in the sandbox using only CLAUDE_CODE_OAUTH_TOKEN as a Gondolin-injected placeholder. The guest never sees the real token. claude --print "2+2=" → 4.
• Custom image build is broken for our target (aarch64-linux-musl, alpine 3.23) — Gondolin's own zig-based guest-binary compile step fails with stale-cache / missing-musl-libc errors. Upstream issue #68 acknowledges build tooling immaturity.
• Verdict: promising fit for moltnet evals, not production-ready today, but our eval runner isn't either. Reasonable to continue investing if we're willing to depend on an early-stage upstream or vendor/fork the build pipeline.

What we built

.exploration/gondolin-prototype/
├── package.json              # pins @earendil-works/gondolin@0.7.0
├── tsconfig.json
├── run.ts                    # smoke-test harness
├── fake-worktree/prompt.txt  # "What is 2+2?"
├── image/
│   ├── build-config.json     # pinned claude 2.1.110 / codex rust-v0.121.0 / moltnet cli-v1.25.0
│   └── build.log             # last failed build output
├── results.json              # smoke-test results
└── FINDINGS.md               # this file

Smoke test results (what worked)

All tests ran against default alpine-base image, installing claude at runtime via npm i -g.

#	Check	Result	Notes
1	Placeholder, not real token, in guest env	✅	Guest sees GONDOLIN_SECRET_fe23... only
2	/workspace mounted RW from host via RealFSProvider	✅	Write-back visible on host
3	Egress to example.com denied	✅	HTTP 403 from policy layer
4	Egress to api.anthropic.com allowed	✅	405 (expected GET-on-POST endpoint)
5	claude --print "2+2=" via placeholder auth	✅	→ 4
6	Warm-cache boot time	✅	28ms (cold, with 89MB asset download: ~14s)

Threat model conclusions:

• Gondolin's header-placeholder secret injection works for Claude Code's OAuth flow. claude --print reads CLAUDE_CODE_OAUTH_TOKEN from env, sends it as Authorization: Bearer <placeholder>, host substitutes for real token only on outbound requests to api.anthropic.com. No token leaks to guest filesystem or non-allowlisted hosts.
• No OAuth refresh issue observed for claude --print — it appears to use the provided token directly rather than minting a new one.
• VFS confinement works. Our fake-worktree/ is the only host FS the guest can see. We did not test ShadowProvider for hiding .git/config or .env inside a real worktree — next step for productization.

What didn't work

1. Alpine's npm package is broken (upstream)

On cold VM state, apk add nodejs npm followed by npm i -g <anything> fails with:

TypeError: Class extends value undefined is not a constructor or null
  at minipass-sized/dist/commonjs/index.js:21:40

Root cause: CJS/ESM interop in Alpine 3.23's npm 11.11.0 package. Not Gondolin-specific — reproducible in plain docker run alpine:3.23. Workaround: upstream musl Node tarballs (but arm64-musl isn't always published), or avoid Node entirely and ship native binaries (our chosen path).

2. Gondolin build custom-image pipeline fails

Tried to bake claude + codex + moltnet native binaries into a custom image via gondolin build --config build-config.json. Build runs a nested Docker container that compiles Gondolin's own guest-side zig binaries (sandboxd, sandboxfs, sandboxingress, sandboxssh) before our postBuild.commands run. That zig compile step fails with:

error: ld.lld: cannot open /work/zig-cache/global/o/<hash>/libcompiler_rt.a: No such file or directory
error: sub-compilation of musl libc.a failed
  note: failed to check cache: invalid manifest file format

• Non-deterministic: different sub-targets fail across runs (sandboxfs, sandboxssh, sandboxd, sandboxingress).
• Zig 0.15.2 from Alpine 3.23 has known parallel-build cache issues. Gondolin's own Makefile comments mention zig 0.15.x flakiness on macOS hosts.
• Tried the maintainer's workaround from #68 — cloning the repo and setting GONDOLIN_GUEST_SRC=/path/to/gondolin/guest. Same failure class.
• Cleaning stale /var/folders/.../gondolin-build-* tmpdirs between runs did not help.

This blocks the "fast warm-boot with all tools preinstalled" end state.

3. Other minor issues encountered

• Node 22 vs 23.6+ engine requirement: Gondolin 0.7.0's engines field requires Node ≥23.6. Install worked on 22.19 but runtime may fail. Upgraded to 24.14.1 via mise.
• claude install is interactive-ish: the native claude.ai/install.sh runs claude install post-download which sets up shell completion. Fails in non-TTY. Workaround: fetch binary directly from GCS bucket, skip claude install, just chmod+x.
• Postbuild commands run per-line in fresh shells: env vars set in step N are not visible in step N+1. Must collapse into a single sh -eu -c '...' per install.
• release/latest fragility: gh api repos/.../releases/latest returns the most recent release, which may not be the CLI train. Must filter by tag prefix or pin explicit versions. We pinned.

Threat-model asymmetry: Claude vs Codex

Dimension	Claude Code CLI	Codex CLI
Auth surface	CLAUDE_CODE_OAUTH_TOKEN env var, sent as Authorization header	File-based: $CODEX_HOME/auth.json or ~/.codex/auth.json, fallback OPENAI_API_KEY
Gondolin integration	Header placeholder (strong): guest never sees real token	File bind-mount (weaker): guest reads real token from /root/.codex/auth.json
Exfiltration risk	Near-zero (guest cannot construct a request with real token without hitting allowlisted host, which is the intended destination anyway)	Guest can exfil token by writing to /workspace/stolen.txt (mounted RW back to host) or embedding it in a response body if any allowed host accepts arbitrary payloads
Mitigation	None needed	Mount ShadowProvider over /workspace to intercept writes, or use codex adapter's existing IsolateHome to copy auth into a temp dir per run (already does this)

We did not test the Codex path end-to-end — the custom-image block stopped us before getting codex into a sandbox. The adapter already supports all four precedence modes (CODEX_API_KEY, $CODEX_HOME/auth.json, ~/.codex/auth.json, OPENAI_API_KEY → synthetic auth.json), so the bind-mount approach should work on top of the snapshot-based image (see next-steps).

Key Gondolin facts we learned

• Not truly cross-platform: macOS + Linux only. No Windows. ARM64 best-tested, x86_64 has CI smoke. Linux x86_64 works; macOS arm64 works.
• QEMU by default (~15s cold boot with asset download, 28ms warm). Optional libkrun backend requires local zig build; experimental.
• Network model is HTTP/1.x + TLS-intercepted HTTPS only. No HTTP/2, HTTP/3, QUIC, WebRTC, UDP. Placeholder substitution happens in TLS-terminated HTTP requests — host process sees plaintext traffic and secrets as part of the policy layer. That's necessary for egress hooks but means "the host is in the TCB". Fine for our threat model (untrusted guest code, trusted host harness).
• Secret placeholder substitution is request-header only. Not URL path, not request body, not response content. Safe for Authorization: Bearer $TOKEN patterns. Silently inert if a CLI puts the token in a body or query param.
• OAuth rotation is not handled. If claude ever refreshes its token via a 401 → refresh → retry flow, the response body with the new token is NOT scrubbed on the way back to the guest, and future requests use the new real token the host policy doesn't know about. We did not observe this in practice with claude --print, but it's a latent risk.
• VFS is the killer feature. RealFSProvider(path), ReadonlyProvider(backend), ShadowProvider(backend, { shouldShadow, writeMode: "deny"|"tmpfs" }). Stacking these gives fine-grained per-path policy that bwrap --ro-bind can't match.
• Guest images are Alpine-only in the current builder.

What we didn't get to

1. Full eval-runner integration. Our smoke test ran a single claude --print "2+2=". Running the actual moltnet eval run --scenario ... --agent claude loop inside Gondolin is the real validation — it involves multi-turn conversations, file edits in /workspace, git ops, and timing-sensitive heartbeat callbacks.
2. Codex path validation. Binary would have been baked into the custom image. Without that, would need to apk add it at runtime which has the same npm/Alpine issue.
3. moltnet CLI path validation. Same situation.
4. ShadowProvider real-worktree test. Proving we can mount a real user worktree while hiding .git/config, .env, .claude/, etc. is important for production use.
5. Boot-time measurement with a working custom image. The 28ms figure is warm-boot of alpine-base. A real image with 500MB of bundled binaries will have different numbers.
6. Parallel / concurrent eval runs. The eval runner fans out. Can 8 VMs boot in parallel on a single M-series Mac? What's RAM usage?
7. HTTP/2 SSE streaming. claude --print seems to work but we didn't probe whether the streaming path is degraded through Gondolin's HTTP/1.x-only TLS proxy. Long agent conversations may surface issues.

What we could do next

Ordered by value + feasibility:

Short term (hours)

1. File the upstream zig-compile bug against gondolin build. Concrete repro, related to open Add comprehensive e2e tests for auth lifecycle and domain flows #68. (queued as task feat: add TDD test suites and refactor database to dependency injection #9)
2. Pivot to snapshot-based image: boot default alpine-base, install all three tools at runtime via apk add curl + direct binary fetches (skip broken npm), use gondolin snapshot to checkpoint, then resume from snapshot in subsequent runs. Works around the build blocker. Costs one-time ~30s setup, then matches custom-image warm-boot behavior. (queued as task feat: add landing page for themolt.net #8)
3. Wire a TS wrapper around VM.create that accepts { worktree, prompt, tool: "claude" | "codex", model } and returns the completion. Consumable from Go (via subprocess) or TS.

Medium term (days)

4. End-to-end eval run: execute one real scenario from evals/legreffier-commit-fix/ inside Gondolin using moltnet eval run. Compare results to an unsandboxed run for equivalence.
5. Benchmark parallel boot: 1, 4, 8, 16 concurrent VMs. Record RAM, CPU, per-VM boot time.
6. Write a real ShadowProvider policy that hides secrets from worktrees and test it against a worktree that contains .env, .git/config, ~/.ssh.

Long term (weeks)

7. Decide: bet on Gondolin, vendor/fork their build pipeline, or switch strategy (native sandbox-exec + bwrap with iptables egress).
8. If continuing with Gondolin: productize as a proper library in libs/eval-sandbox/ with both Go and TS interfaces, docs, examples, and CI coverage.

Decision matrix

Option	Effort	Security	Maturity	Cross-platform	Startup cost
Gondolin (VM)	medium-high; upstream is flaky	strong (VM boundary + policy)	early / experimental	macOS + Linux	28ms warm, ~15s cold
sandbox-exec + bwrap	medium; two backends to write	good (kernel-enforced syscall filter)	mature primitives, no unified API	macOS + Linux only (no Windows AppContainer in scope)	near-zero
Docker sandboxes (current)	low (baseline)	good (namespace + cgroup)	mature	cross-platform with Docker Desktop	~1-3s
gVisor / Firecracker	high	very strong	mature	Linux-only	100s of ms
Do nothing	zero	zero	—	—	—

Gondolin's value vs sandbox-exec+bwrap: programmable egress policy with secret-placeholder injection. That's the single feature that's hard to replicate natively. Everything else (FS confinement, process isolation) native primitives do fine.

Raw data: results.json from the last green smoke test

Boot time: 28ms (warm). Full trace in results.json. Key lines:

[host] placeholder env will be: {"CLAUDE_CODE_OAUTH_TOKEN":"GONDOLIN_SECRET_021ce05..."}
[host] VM booted in 28ms
=== 1. env leak check ===
CLAUDE_CODE_OAUTH_TOKEN=<REDACTED>  (placeholder only, not real token)
=== 2. workspace mount ===  ✅ RW bidirectional
=== 3. egress denied (example.com) ===  ✅ HTTP 403
=== 4. egress allowed (api.anthropic.com) ===  ✅ HTTP 405
=== 5. apk add nodejs npm ===  ✅ ok
=== 5b. npm i -g @anthropic-ai/claude-code ===  flaky but eventually worked
=== 6. claude --print "2+2=" ===  → "4"

Reproducibility

cd .exploration/gondolin-prototype
npm install                                  # requires Node ≥ 23.6 (via mise)
export CLAUDE_CODE_OAUTH_TOKEN=<token>       # from `claude setup-token`
npx tsx run.ts                               # first run: ~17s (asset download + claude install)
                                             # subsequent: ~5s (dominated by claude --print)

To attempt the custom-image build (expected to fail until upstream is fixed):

export GONDOLIN_GUEST_SRC=/path/to/gondolin-clone/guest
export PATH="/opt/homebrew/opt/e2fsprogs/sbin:$PATH"
npx gondolin build --config image/build-config.json --output image/assets

References

• Gondolin docs: https://earendil-works.github.io/gondolin/
• VFS: https://earendil-works.github.io/gondolin/vfs/
• Secrets: https://earendil-works.github.io/gondolin/secrets/
• Custom images: https://earendil-works.github.io/gondolin/custom-images/
• Known limitations: https://earendil-works.github.io/gondolin/limitations/
• Issue Add comprehensive e2e tests for auth lifecycle and domain flows #68 (custom image quickstart): Custom image quick start error earendil-works/gondolin#68

[getlarge]

Update 2026-04-16: snapshot pivot works

Per the "short term" plan in the issue body, pivoted around the broken gondolin build by building a qcow2 disk checkpoint from the default alpine-base image: boot → grow rootfs → install native binaries → vm.checkpoint(path). Subsequent runs call VmCheckpoint.load(path).resume({ httpHooks, env, vfs }). Fully validated end-to-end for both claude and codex paths.

What's new in .exploration/gondolin-prototype/

• snapshot-build.ts — one-time setup (~20s). Boots alpine-base with a 3GB qcow2 overlay backed by the base rootfs, runs resize2fs /dev/vda, apk add runtime deps, curls pinned claude/codex/moltnet binaries into /usr/local/bin, then vm.checkpoint().
• run.ts — per-eval harness. Resumes the checkpoint once per path (claude, codex), applies per-run httpHooks (egress allowlist + secrets) + vfs (worktree mount).

Results

Path	Resume	Full run	Egress canary
Claude (header placeholder, CLAUDE_CODE_OAUTH_TOKEN)	29ms	2.6s → 4	example.com → HTTP 403
Codex (file-mount /codex-home/auth.json, gpt-5.3-codex)	37ms	5.1s → 4	api.anthropic.com → HTTP 403

Cross-provider egress isolation proven: each path's allowlist rejects the other provider's API host, even though both tokens are available on the host.

Key implementation details

1. Default alpine-base rootfs is 261MB / 86MB free — too small for the claude binary alone (~228MB). Workaround: qemu-img create -F raw -b <rootfs.ext4> -f qcow2 overlay.qcow2 3G + resize2fs /dev/vda at first boot. Then pass the overlay via sandbox: { rootDiskPath, rootDiskFormat: "qcow2", rootDiskDeleteOnClose: false }. vm.checkpoint(path) moves the overlay to the checkpoint file and appends a JSON metadata trailer in-place.
2. Checkpoint on disk: 401MB (qcow2 stores only changed blocks; rootfs remains backed by the base image object).
3. Codex auth via vm.fs.writeFile("/codex-home/auth.json", authJson, { mode: 0o600 }) after resume — no persistent VFS provider needed. Auth lives in guest-local disk, visible to guest (weaker isolation than claude's header-placeholder model, documented in FINDINGS.md).
4. Codex model: must match our adapter default gpt-5.3-codex. Initial attempt with gpt-5-codex returned 400: model not supported when using Codex with a ChatGPT account.

Reproducibility

cd .exploration/gondolin-prototype
npm install
npx tsx snapshot-build.ts                  # one-time, ~20s
export CLAUDE_CODE_OAUTH_TOKEN=<token>     # claude setup-token
# optional: export OPENAI_API_KEY=...      # or have ~/.codex/auth.json
npx tsx run.ts                             # resumes checkpoint, exercises both paths

Remaining items

The original "short term / medium term / long term" list still applies. Pivot unblocks:

• Pivot to snapshot-based image
• Claude path validated against live API under snapshot
• Codex path validated against live API under snapshot
• File upstream zig-compile bug (still queued — blocks true "custom image" path)
• TS wrapper { worktree, prompt, tool, model } → completion consumable from Go or TS
• End-to-end real eval scenario (evals/legreffier-commit-fix/) inside Gondolin
• ShadowProvider real-worktree test hiding .env, .git/config, .claude/
• Parallel boot benchmark (1, 4, 8, 16 concurrent VMs)
• HTTP/2 SSE streaming behaviour under multi-turn conversations

[legreffier]

Update: wiring moltnet eval run end-to-end inside the Gondolin snapshot

Pivoted from the 2+2 smoke test to the real goal: execute
moltnet eval run --scenario evals/moltnet-practices/dbos-after-commit
entirely inside the checkpointed VM. Current status: harness is built, the
eval runner starts inside the guest, but we hit a FUSE mount issue that
forces a small architectural change. Documenting the blockers we hit so
whoever picks this up next doesn't repeat them.

Checkpoint additions since the last update

snapshot-build.ts now also installs inside the guest:

• git (alpine package) — moltnet eval run calls
  git rev-parse --show-toplevel at startup from
  eval_worktree.go:currentRepoRoot, so the binary must be on PATH even
  in vitro mode.
• A non-root agent user (UID 501 to match macOS host uid) —
  the baked claude CLI refuses to run with --permission-mode bypassPermissions when EUID is 0 ("--dangerously-skip-permissions
  cannot be used with root/sudo privileges"). The adapter at
  libs/dspy-adapters/claudecode/claudecode.go:193 always passes
  bypassPermissions, so no amount of flag-tuning on the host side
  avoids it — the eval runner must execute as a non-root user.

Snapshot is still ~401 MB qcow2 with 43–55 ms warm resume.

Per-run host-side setup the runner needs

• --model wants the <provider>/ prefix (anthropic/claude-haiku-4-5,
  openai/gpt-5.3-codex).
• --judge-model wants the bare model id with the adapter prefix
  (claude-haiku-4-5, or the openai model id). Yes, the two flags
  disagree on convention — worth flagging as a papercut in a follow-up.
• The CWD must be inside a git repo. For vitro we just git init the
  staging dir with a single empty commit before booting.
• GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=safe.directory GIT_CONFIG_VALUE_0='*'
  — otherwise git refuses with "dubious ownership" whenever the guest's
  euid differs from the mounted directory's owner uid. Harmless
  belt-and-braces once the agent user matches the host uid, but required
  as long as any uid mismatch is possible.
• api.themolt.net / *.themolt.net added to allowedHosts. I haven't
  yet confirmed the CLI actually calls home during a pure-local eval,
  but the auto-discover credentials path will try; easier to allowlist
  than to prove unreachable.

The actual blocker: RealFSProvider + FUSE default_permissions

Mounting /workspace with RealFSProvider(hostStagingDir) works for
root but not for the agent user, even with matching UIDs and
world-readable perms on the host directory:

$ mount | grep workspace
sandboxfs on /workspace type fuse.sandboxfs
  (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions)

$ ls -ldn /workspace
drwxr-xr-x 4 501 20 128 /workspace     # uid/gid look fine

$ su agent -s /bin/sh -c 'cd /workspace && ls'
sh: cd: line 0: can't cd to /workspace: Permission denied

Root cause: sandboxfs is mounted by the guest init as root with
default_permissions and without allow_other. That's the classic
FUSE single-user-mount restriction — only the user who mounted the FUSE
filesystem can traverse it. Gondolin's init script
(alpine/init-scripts.js) calls sandboxfs --mount ... directly and
doesn't expose an allowOther knob through VMOptions.vfs, so we
can't flip it from the host SDK today. Without it, a non-root user
literally cannot enter any RealFSProvider mount, which makes the
ergonomic "bidirectional host↔guest worktree" story break for any CLI
that refuses to run as root (which is a pretty common safety posture).

This is worth raising upstream — either an allowOther: true option on
VMOptions.vfs.mounts[...], or having sandboxfs default to allow_other
with default_permissions so kernel-level perms still apply but
non-root users can cross the mount. I'll capture it as a separate issue
against earendil-works/gondolin.

Workaround in progress

Switching the scenario runner to:

1. Resume VM without a RealFSProvider mount at /workspace.
2. Post-resume, vm.exec a shell that mkdir -p /workspace,
   chown agent:agent, then vm.fs.writeFile the scenario files
   (task.md, eval.json, criteria.json) — vm.fs writes land on
   the guest's ext4 overlay, not through sandboxfs.
3. git init the workspace as agent, single empty commit.
4. Run moltnet eval run as agent via su.
5. Read artifacts back out with vm.fs.readFile (or tar + single
   readFile) — no passthrough mount needed for exfil either.

Tradeoff: we lose the "worktree is live on the host" ergonomics that
made Gondolin attractive in the first place. For adversarial eval
workloads that's actually fine — outputs should always be reviewed
explicitly, not leaked into the developer's tree. For day-to-day
editing it would matter more.

What's next on this branch

• Finish the tmpfs-workspace pivot, re-run the dbos-after-commit
  scenario for both claude+claude and codex+codex.
• Capture artifact paths, scores, and full result.json per variant
  into results/ or similar so we can diff against unsandboxed runs.
• File the allow_other upstream issue with the repro above.
• Decide: do we keep pushing through Gondolin papercuts (rebuild
  snapshot for each guest dep discovery), or switch to one of the
  alternatives in the decision matrix now that we have concrete
  "what does the eval runner actually need" data? My current read:
  the sandboxfs limitation is real but surmountable; the
  --allow_other fix in upstream is probably a one-line patch. So
  worth continuing.

[legreffier]

First real eval scenario completed inside Gondolin

moltnet eval run --scenario dbos-after-commit --agent claude --judge claude
ran end-to-end inside the checkpointed VM. This is the full eval runner (dspy
engine, worktree setup, agent turn, judge turn, artifact collection) — not the
simplified claude --print smoke test from before.

Result

Metric	Value
Score	15% (1/5 criteria)
Agent	claude-haiku-4-5, 21.3s, $0.027
Judge	claude-haiku-4-5, 62.2s
Total wall time	83.5s
VM resume	42ms

Criterion	Score	Detail
diary_insert_uses_the_repository	✅ 100%	Correctly used createDiaryRepository(db).create()
workflow_started_after_transaction_commits	❌ 0%	Put workflow INSIDE runTransaction (the designed trap)
notes_name_both_systems_as_separate_backends	❌ 0%	Didn't name DBOS + Drizzle as separate systems
notes_describe_the_rollback_failure_mode	❌ 0%	Missed the dangling Keto grant scenario
grant_not_started_on_rollback_path	❌ 0%	Workflow starts before transaction resolves

15% without context pack is the expected baseline for this scenario — it's
designed so the "intuitive" answer fails the key criterion.

Additional blockers solved since last update

On top of the FUSE allow_other issue from the previous comment:

• su drops Gondolin env: su -p agent is required to preserve the
  CLAUDE_CODE_OAUTH_TOKEN placeholder that Gondolin injects. Without -p,
  the token vanishes and claude hangs waiting for interactive auth.
• HOME not reset by su -p: must explicitly export HOME=/home/agent
  inside the shell since -p preserves root's HOME.
• Non-streaming output: vm.exec with stdout: "inherit" now streams
  the eval runner's progress to the host console in real time. Without it,
  | tail -N buffered everything until exit — looked like a hang.
• Model prefix mismatch: --model wants anthropic/claude-haiku-4-5,
  --judge-model wants claude-haiku-4-5 (no provider prefix). Worth a
  consistency fix in the CLI.

Snapshot contents (rebuilt from scratch)

claude     2.1.112   228MB  /usr/local/bin/claude
codex      0.121.0   161MB  /usr/local/bin/codex
moltnet    1.26.0     13MB  /usr/local/bin/moltnet
git        (alpine)          /usr/bin/git
agent user UID 501           /home/agent

Total qcow2: ~410MB. Resume: 42–58ms.

Architecture for the working run

Host                              Guest (Alpine micro-VM)
─────                             ──────────────────────
checkpoint.load()
  └─ cp.resume({ httpHooks, env })
       ├─ allowedHosts: [anthropic, themolt.net]
       ├─ secrets: CLAUDE_CODE_OAUTH_TOKEN → placeholder
       └─ no vfs mounts (sandboxfs unusable for non-root)

                                  vm.exec: mkdir /workspace, chown agent
vm.fs.writeFile(scenario files)   → /workspace/scenario/{task.md,eval.json,criteria.json}
                                  vm.exec: su -p agent → git init /workspace
                                  vm.exec: su -p agent → moltnet eval run (streamed)
                                    ├─ agent turn: claude --print (21s)
                                    └─ judge turn: claude --print (62s)
                                  vm.exec: tar /tmp/moltnet-eval-dspy-* → /tmp/artifacts.tar.gz
vm.fs.readFile(tar) → host        eval-artifacts/claude/...
vm.close()

What's next

• Run the codex path (--agent codex --judge codex) for comparison
• Run with a context pack (--pack) to measure the delta
• File allow_other upstream issue against earendil-works/gondolin
• Benchmark parallel boot (2, 4, 8 concurrent VMs)
• Decide: continue with Gondolin or switch strategy

[legreffier]

Context pack validation: 30% → 90% with moltnet-dev-practices pack

Ran the full dbos-after-commit scenario with --pack docs/rendered-packs/moltnet-dev-practices.md inside Gondolin. The runner executed both variants (without-context, with-context) sequentially and reported the delta.

Results: Claude haiku-4-5

Criterion (weight)	Without pack	With pack	Delta
workflow_started_after_transaction_commits (35)	0%	100%	+35
notes_name_both_systems_as_separate_backends (25)	20%	100%	+20
notes_describe_the_rollback_failure_mode (20)	25%	50%	+5
diary_insert_uses_the_repository (15)	100%	100%	—
grant_not_started_on_rollback_path (5)	100%	100%	—
Total	30%	90%	+60pp

The key criterion — workflow placement AFTER runTransaction resolves, not inside it — flipped from 0% to 100%. The pack explicitly teaches the DBOS/Drizzle separate-backends pattern, and the agent followed it.

Earlier baseline comparison (without pack)

	Claude (haiku-4-5)	Codex (gpt-5.3-codex)
Score	15%	20%
Agent time	21.3s	61.0s
Judge time	62.2s	24.1s
Total	83.5s	85.2s

Both fell into the same trap without the pack. Codex scored slightly higher
because the judge gave it credit on grant_not_started_on_rollback_path
(if the insert throws, the workflow line is never reached).

Timing: with-pack run

• VM resume: 72ms
• Both variants (sequential): 149s total
• Without-context variant: ~50s (agent + judge)
• With-context variant: ~99s (agent + judge)

What this proves

1. The Gondolin sandbox can host the full eval pipeline — worktree
   setup, agent turn, judge turn, artifact collection, pack injection —
   end-to-end with both claude and codex adapters.
2. Context packs produce measurable, reproducible score deltas inside
   the sandbox. The +60pp lift on dbos-after-commit matches what we'd
   expect from an unsandboxed run.
3. The sandbox overhead is minimal: 50–72ms resume, ~150s for a
   two-variant run. The bottleneck is LLM latency, not the VM.

Experiment summary

Run	Agent	Pack	Score	Wall time
Baseline	claude-haiku-4-5	none	15% → 30%*	83s
Baseline	codex gpt-5.3	none	20%	85s
With pack	claude-haiku-4-5	moltnet-dev-practices	90%	149s (2 variants)

*30% in the second run without pack (slight variance from 15% in first run — different session, same model).

Status

The Gondolin prototype validates the core thesis: we can sandbox eval runs
with credential isolation, egress control, and filesystem confinement while
running the real moltnet eval run pipeline unmodified inside the guest.

Remaining work tracked in the "What's next" checklist from the previous comment.

[legreffier]

Next: agent runtime via pi extension model

Where we landed

The Gondolin prototype validates the sandbox thesis (see results above). The next question was: what's the right abstraction for running agent workloads (eval, PR review, pack judge) programmatically?

After exploring several architectures — generic agent runtime, task execution framework, Go native bindings — we concluded: design without hands-on pi experience leads to poor architecture.

What is pi?

pi is a coding agent framework (from Mario Zechner / badlogic) with:

• pi-ai: Multi-provider LLM abstraction (Claude, OpenAI, Google, etc.)
• pi-agent-core: Stateful agent loop with tool execution, streaming, steering
• pi-coding-agent: CLI coding agent with extension system
• Extension model: export default function(pi: ExtensionAPI) → register slash commands + lifecycle hooks

pi-review is a pi extension that implements /review — code review across PRs, branches, commits, folders. Single 54KB TypeScript file. Installs via pi install git:github.com/earendil-works/pi-review.

Why pi matters for MoltNet

The "task system" we were designing for the agent runtime is pi's extension model:

• A task = a registered command with tools, system prompt, output handling
• Dynamic tasks = dynamic extensions (pi install git:...)
• Worktree management, diary access, checklist judging = helpers an extension uses

Instead of building a custom runtime from scratch, we can:

1. Build a MoltNet pi extension that provides /moltnet-review, /moltnet-judge, etc.
2. The extension uses @themoltnet/sdk for diary/entry access
3. Judge logic ports from Go libs/dspy-adapters/checklist to TypeScript (~200 lines)
4. Gondolin wraps the pi agent execution for isolation (future layer)

Immediate next steps

• Install pi CLI, use it as a coding agent
• Install pi-review, review real PRs to learn the extension UX
• Study ExtensionAPI: registerCommand, on(event), sessionManager, pi-tui
• Draft a minimal MoltNet pi extension (PR complexity reviewer as first command)
• Revisit agent-runtime lib design with pi experience in hand

Gondolin integration path

Pi extensions run on the host. For isolation:

1. v1: Extension runs on host, read-only worktree tools, LLM judge call via pi-ai — no Gondolin needed
2. v2: Extension's tool execution runs inside Gondolin VM (credential isolation, egress control)
3. v3: Remote triggering — pi agent receives tasks via broker, executes in Gondolin sandbox

The Gondolin snapshot/VM infrastructure from this prototype stays relevant for v2+.

[getlarge]

Pi SDK + Gondolin VM: full stack validated

Pi agent session with custom MoltNet tools running inside a Gondolin VM, credentials injected at start (never baked in snapshot). Both tasks pass end-to-end.

What was validated

#	Check	Result
1	Pi OAuth credentials injected at VM start	✅ auth.json → /home/agent/.pi/agent/auth.json
2	MoltNet identity injected at VM start	✅ moltnet.json + env → /home/agent/.moltnet/
3	Pi createAgentSession() works inside VM	✅ OpenAI Codex API via Gondolin MITM proxy
4	Custom MoltNet tools (defineTool) work	✅ moltnet_list_entries invoked by GPT Codex
5	MoltNet SDK connect() works inside VM	✅ Agent identity resolved: 1671-B080-99BF-4270
6	Egress policy enforced	✅ Only OpenAI + MoltNet API + npmjs + GitHub
7	TLS through Gondolin MITM proxy	✅ Fixed: append MITM CA to system bundle + NODE_EXTRA_CA_CERTS
8	Snapshot resume time	✅ ~35ms warm

Key findings

Alpine npm is broken — Alpine 3.23's npm 11.x has a CJS/ESM interop bug (Class extends value undefined). Workaround: install Node.js from unofficial-builds (linux-arm64-musl), which ships a working npm.

TLS cert loading — The unofficial-builds Node ships its own OpenSSL which can't load NODE_EXTRA_CA_CERTS from /etc/gondolin/mitm/ca.crt (error 8000000D). Fix: append the MITM cert to the system CA bundle and point NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt.

Pi settings not needed for programmatic use — Only auth.json is needed. Copying settings.json triggers extension installation (e.g. pi-review) which tries to git clone from GitHub. createAgentSession({ agentDir, model }) takes the model explicitly, ignoring settings.

ESM module resolution — Pi SDK packages must be installed locally in the workspace (not globally). Global npm install -g + NODE_PATH doesn't work with ESM imports via tsx.

Pi binary tarball is glibc-only — pi-linux-arm64.tar.gz from badlogic/pi-mono releases is dynamically linked against glibc, won't run on Alpine/musl. Must use npm packages instead.

Snapshot build

Builds a qcow2 checkpoint with Node 22 (musl) + tsx + pi SDK + MoltNet SDK. No credentials.

// snapshot-build.ts — Build a Gondolin VM snapshot with Node.js + pi SDK + MoltNet SDK
import {
  VM, createHttpHooks, ensureImageSelector, loadGuestAssets,
} from '@earendil-works/gondolin';
import { fileURLToPath } from 'node:url';
import path from 'node:path';
import { execFileSync } from 'node:child_process';
import { mkdirSync, existsSync, rmSync, statSync } from 'node:fs';

const __dirname = path.dirname(fileURLToPath(import.meta.url));
const CHECKPOINT_DIR = path.resolve(__dirname, '..', 'image');
const CHECKPOINT_PATH = path.join(CHECKPOINT_DIR, 'pi-agent.qcow2');
const OVERLAY_PATH = path.join(CHECKPOINT_DIR, 'pi-agent.overlay.qcow2');

const SETUP_ALLOWED_HOSTS = [
  'dl-cdn.alpinelinux.org', '*.alpinelinux.org',
  'registry.npmjs.org', '*.npmjs.org',
  'nodejs.org', '*.nodejs.org', 'unofficial-builds.nodejs.org',
];

mkdirSync(CHECKPOINT_DIR, { recursive: true });
for (const p of [CHECKPOINT_PATH, OVERLAY_PATH]) {
  if (existsSync(p)) rmSync(p);
}

const resolvedImage = await ensureImageSelector('alpine-base');
const assets = loadGuestAssets(resolvedImage.assetDir);

execFileSync('qemu-img', [
  'create', '-f', 'qcow2', '-F', 'raw', '-b', assets.rootfsPath,
  OVERLAY_PATH, '3G',
], { stdio: 'inherit' });

const { httpHooks } = createHttpHooks({ allowedHosts: SETUP_ALLOWED_HOSTS });

const vm = await VM.create({
  httpHooks,
  env: { PATH: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' },
  sandbox: {
    rootDiskPath: OVERLAY_PATH,
    rootDiskFormat: 'qcow2',
    rootDiskReadOnly: false,
    rootDiskDeleteOnClose: false,
  },
});

async function run(label: string, cmd: string) {
  const r = await vm.exec(cmd);
  console.log(`[${label}] exit=${r.exitCode}`);
  if (r.stdout) console.log(r.stdout.trimEnd());
  if (r.exitCode !== 0) throw new Error(`"${label}" failed (exit ${r.exitCode})`);
}

try {
  await run('resize', 'apk add --no-cache e2fsprogs-extra >/dev/null && resize2fs /dev/vda');
  await run('deps', 'apk add --no-cache ca-certificates curl git libgcc libstdc++ tar xz >/dev/null');

  // Node from unofficial-builds (Alpine's npm is broken)
  const nodeVersion = '22.22.2';
  await run('node', `sh -eu -c '
    curl -fsSL "https://unofficial-builds.nodejs.org/download/release/v${nodeVersion}/node-v${nodeVersion}-linux-arm64-musl.tar.xz" -o /tmp/node.tar.xz
    tar -xJf /tmp/node.tar.xz -C /usr/local --strip-components=1 && rm /tmp/node.tar.xz
  '`);

  await run('tsx', 'npm install -g tsx');

  // Pi SDK + MoltNet SDK installed locally in /workspace (ESM needs local node_modules)
  await run('workspace', `sh -eu -c '
    mkdir -p /workspace && cd /workspace
    echo '"'"'{"type":"module"}'"'"' > package.json
    npm install @mariozechner/pi-coding-agent @mariozechner/pi-ai @themoltnet/sdk
  '`);

  await run('agent', `sh -eu -c '
    addgroup -g 501 agent
    adduser -D -u 501 -G agent -h /home/agent -s /bin/sh agent
    mkdir -p /home/agent/.pi/agent /home/agent/.moltnet /home/agent/.cache
    chown -R agent:agent /home/agent /workspace
  '`);

  const cp = await vm.checkpoint(CHECKPOINT_PATH);
  console.log(`checkpoint: ${cp.path} (${cp.guestAssetBuildId})`);
} finally {
  try { await vm.close(); } catch {}
  if (existsSync(OVERLAY_PATH)) rmSync(OVERLAY_PATH, { force: true });
}

Run VM — complete example with credential injection + MoltNet custom tools

// run-vm.ts — Resume snapshot, inject credentials, run pi agent with MoltNet tools
import { VmCheckpoint, createHttpHooks } from '@earendil-works/gondolin';
import { readFileSync } from 'node:fs';
import { execFileSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import path from 'node:path';
import { parseArgs, parseEnv } from 'node:util';

const __dirname = path.dirname(fileURLToPath(import.meta.url));
const CHECKPOINT_PATH = path.resolve(__dirname, '..', 'image', 'pi-agent.qcow2');

// Resolve the main worktree root (where .moltnet/ lives)
function findMainWorktree(): string {
  const output = execFileSync('git', ['worktree', 'list', '--porcelain'], { encoding: 'utf8' });
  for (const block of output.split('\n\n')) {
    const lines = block.split('\n');
    const wt = lines.find(l => l.startsWith('worktree '));
    if (wt && !lines.some(l => l === 'bare')) return wt.replace('worktree ', '');
  }
  throw new Error('Could not find main git worktree');
}

const MAIN_REPO = findMainWorktree();

const { values: args } = parseArgs({
  options: {
    agent: { type: 'string', short: 'a', default: 'legreffier' },
    task: { type: 'string', short: 't', default: 'entries' },
  },
});

const agentName = args.agent!;
const agentDir = path.join(MAIN_REPO, '.moltnet', agentName);

function loadCredentials() {
  const moltnetJson = readFileSync(path.join(agentDir, 'moltnet.json'), 'utf8');
  const agentEnvRaw = readFileSync(path.join(agentDir, 'env'), 'utf8');
  const piAuthJson = readFileSync(
    path.join(process.env.HOME ?? '', '.pi', 'agent', 'auth.json'), 'utf8',
  );
  return { moltnetJson, agentEnvRaw, piAuthJson, agentEnv: parseEnv(agentEnvRaw) };
}

// Task script injected into the VM at /workspace/task.ts
function buildTaskScript(): string {
  return `
import { createAgentSession, defineTool } from '@mariozechner/pi-coding-agent';
import { getModel, Type } from '@mariozechner/pi-ai';
import { connect } from '@themoltnet/sdk';
import { readFileSync } from 'node:fs';
import { parseEnv } from 'node:util';

// 1. Authenticate to MoltNet using injected credentials
const agent = await connect({ configDir: '/home/agent/.moltnet' });
const whoami = await agent.agents.whoami();
console.log('[vm] MoltNet identity:', whoami.fingerprint);

// 2. Resolve diary from injected env
const env = parseEnv(readFileSync('/home/agent/.moltnet/env', 'utf8'));
const diary = await agent.diaries.get(env.MOLTNET_DIARY_ID);
console.log('[vm] Diary:', diary.name);

// 3. Define custom MoltNet tool using TypeBox schemas
const listTool = defineTool({
  name: 'moltnet_list_entries',
  label: 'List Diary Entries',
  description: 'List recent MoltNet diary entries.',
  parameters: Type.Object({
    limit: Type.Optional(Type.Number()),
  }),
  async execute(_id, params, _signal, _onUpdate, _ctx) {
    const entries = await agent.entries.list(diary.id, {
      limit: params.limit ?? 5,
      orderBy: 'createdAt',
      order: 'desc',
    });
    const text = JSON.stringify(entries.items?.map(e => ({
      id: e.id, title: e.title, tags: e.tags, importance: e.importance,
    })), null, 2);
    return { content: [{ type: 'text', text }], details: {} };
  },
});

// 4. Create pi agent session with injected OAuth credentials + custom tools
const { session } = await createAgentSession({
  agentDir: '/home/agent/.pi/agent',          // only auth.json needed
  model: getModel('openai-codex', 'gpt-5.3-codex'),
  tools: [],                                   // strip built-in tools
  customTools: [listTool],                     // MoltNet tools
  systemPrompt: 'Diary assistant for ' + whoami.fingerprint + '. Use moltnet_list_entries.',
});

// 5. Subscribe to events and surface errors
session.subscribe((event) => {
  if (event.type === 'message_update' && event.assistantMessageEvent.type === 'text_delta')
    process.stdout.write(event.assistantMessageEvent.delta);
  if (event.type === 'turn_end' && event.message?.stopReason === 'error') {
    console.error('[ERROR] LLM API error:', JSON.stringify(event, null, 2));
    process.exit(1);
  }
});

await session.prompt('List the 5 most recent diary entries and summarize recent work.');
session.dispose();
`;
}

function shellQuote(s: string): string {
  return "'" + s.replace(/'/g, "'\\''") + "'";
}

async function main() {
  const creds = loadCredentials();
  const moltnetConfig = JSON.parse(creds.moltnetJson);
  const apiHost = new URL(moltnetConfig.endpoints.api).hostname;

  // Egress policy: only LLM provider + MoltNet API
  const { httpHooks, env: secretEnv } = createHttpHooks({
    allowedHosts: [
      'api.openai.com', '*.openai.com', 'chat.openai.com',
      'chatgpt.com', '*.chatgpt.com',
      apiHost,
      'registry.npmjs.org',
      'github.com', '*.github.com', '*.githubusercontent.com',
    ],
  });

  // Resume snapshot (~35ms)
  const cp = VmCheckpoint.load(CHECKPOINT_PATH);
  const vm = await cp.resume({
    httpHooks,
    env: {
      ...secretEnv,
      PATH: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
      HOME: '/home/agent',
      NODE_NO_WARNINGS: '1',
    },
  });

  try {
    // Fix TLS: append Gondolin MITM CA to system trust store
    await vm.exec(`sh -c '
      cp /etc/gondolin/mitm/ca.crt /usr/local/share/ca-certificates/gondolin-mitm.crt
      update-ca-certificates
      cat /etc/gondolin/mitm/ca.crt >> /etc/ssl/certs/ca-certificates.crt
    '`);

    // Inject credentials (never baked in snapshot)
    await vm.fs.writeFile('/home/agent/.pi/agent/auth.json', creds.piAuthJson, { mode: 0o600 });
    await vm.fs.writeFile('/home/agent/.moltnet/moltnet.json', creds.moltnetJson, { mode: 0o600 });
    await vm.fs.writeFile('/home/agent/.moltnet/env', creds.agentEnvRaw, { mode: 0o600 });
    await vm.exec('chown -R agent:agent /home/agent/.pi /home/agent/.moltnet');

    // Write task script + run as non-root agent user
    await vm.fs.writeFile('/workspace/task.ts', buildTaskScript(), { mode: 0o644 });
    await vm.exec('chown agent:agent /workspace/task.ts');

    const result = await vm.exec(
      `su -p agent -s /bin/sh -c ${shellQuote(
        'export HOME=/home/agent && export NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt && cd /workspace && tsx task.ts 2>&1',
      )}`,
      { timeoutMs: 120_000, stdout: 'inherit', stderr: 'inherit' } as any,
    );
    console.log(`exit=${result.exitCode}`);
  } finally {
    await vm.close();
  }
}

main().catch((err) => { console.error('[fatal]', err); process.exit(1); });

Output (entries task)

[host] resumed in 35ms
[host] credentials injected
[vm] MoltNet identity: 1671-B080-99BF-4270
[vm] Diary: themoltnet
Here are the 5 most recent diary entries:

1. Agent runtime design checkpoint: from sandbox to task orchestration
   - Tags: agent-runtime, gondolin, pi, promise-theory, architecture
   - Importance: 8

2. release workflow: set GH_REPO for gh release download
   - Tags: accountable-commit, risk:medium, branch:main, scope:release
   - Importance: 5
...
exit=0

Artifacts

.exploration/pi-gondolin/
├── package.json              # @earendil-works/gondolin@0.7.0, pi-coding-agent, pi-ai, @themoltnet/sdk
├── src/
│   ├── snapshot-build.ts     # Build snapshot: Node 22 musl + tsx + pi SDK + MoltNet SDK
│   └── run-vm.ts             # Resume + inject credentials + run task
└── image/
    └── pi-agent.qcow2        # Checkpoint (not committed)

Branch: exploration/pi-gondolin-runtime (worktree at ../themoltnet-pi-gondolin)

[legreffier]

Follow-up: Pi extension model + auto-build snapshot

The standalone VM scripts (snapshot-build.ts + run-vm.ts) from the previous comment validated the full stack. The next step was adopting pi's extension model — and it turns out this is the right architecture for the agent runtime.

What changed: moltnet-gondolin.ts — a pi extension

Instead of inline task scripts injected into the VM, we now have a proper pi extension that redirects pi's built-in tools (read/write/edit/bash) into a Gondolin VM:

pi -e ./src/moltnet-gondolin.ts              # run pi with Gondolin sandbox
pi -e ./src/moltnet-gondolin.ts --agent legreffier --worktree-branch feat/task

How it works:

1. session_start → lazy-boots Gondolin VM from checkpoint (~35ms)
2. Injects pi OAuth + MoltNet identity credentials into VM filesystem
3. Mounts host working directory (or a fresh git worktree) at /workspace via RealFSProvider
4. Registers tool overrides: createReadTool(cwd, { operations: gondolinReadOps }) etc.
5. Registers MoltNet custom tools (moltnet_list_entries, moltnet_get_entry) that run on host via SDK
6. before_agent_start → patches system prompt to reflect VM workspace path
7. user_bash → redirects interactive shell commands to VM
8. session_shutdown → gracefully closes VM

Key insight: pi's extension API (registerTool, on('user_bash'), tool operations pattern) maps directly to what we need. The "task system" from #852 is pi's extension model — a registered command with tools, system prompt, and lifecycle hooks.

Next design iteration

Before committing the extension, several improvements are needed:

1. Auto-build snapshot on first run

The snapshot should build automatically when the extension first runs and no cached checkpoint exists. No manual npm run snapshot:build step.

~/.cache/moltnet/gondolin/       (Linux)
~/Library/Caches/moltnet/gondolin/  (macOS)
└── v1-<recipe-hash>/
    └── pi-agent.qcow2

The recipe hash covers: Node version, installed packages, Alpine base image version. When any of these change, a new snapshot is built and the old one can be pruned.

2. Expanded VM toolchain

The current snapshot only has Node + tsx + pi SDK + MoltNet SDK. For real dev work we need:

Tool	Why
node (unofficial-builds musl)	Runtime — Alpine's npm is broken
pnpm	Package manager (matches host monorepo)
git	Worktree operations, commits
gh	PR creation, issue management
moltnet CLI (Go binary)	Agent identity, diary operations
python3	Script execution, data processing
jq	JSON processing
Standard tools	curl, tar, xz, etc.

No Docker in VM — e2e tests that need a full stack run in CI. The VM can reach host services via Gondolin's TCP port mapping (tcp: { hosts: { 'localhost': '127.0.0.1:5432' } } with synthetic DNS).

3. Safety for external use

The extension must be safe for others to install in their own repos:

• Snapshot builds locally from public sources only (Alpine repos, npmjs, unofficial-builds.nodejs.org)
• No credentials baked in snapshot — always injected at VM start
• Egress policy is explicit and auditable (allowedHosts array)
• VFS mounts are scoped to the working directory (path escape throws)
• MoltNet custom tools run on host via SDK (no credential exposure to VM)

4. Session reflection → diary entries

Inspired by pi-self-learning: use session_tree hook to serialize the conversation, run LLM reflection, extract mistakes/fixes, and create MoltNet episodic diary entries. This connects pi sessions to MoltNet's persistent memory.

Architecture

┌─────────────────────────────────────────────────┐
│  pi CLI (host)                                  │
│  └─ moltnet-gondolin extension                  │
│     ├─ tool overrides → Gondolin VM             │
│     │   ├─ read/write/edit: VFS operations      │
│     │   └─ bash: vm.exec() as agent user        │
│     ├─ MoltNet tools → SDK on host              │
│     │   ├─ moltnet_list_entries                 │
│     │   └─ moltnet_get_entry                    │
│     ├─ lifecycle hooks                          │
│     │   ├─ session_start → ensureVm()           │
│     │   ├─ before_agent_start → patch prompt    │
│     │   ├─ user_bash → VM bash ops              │
│     │   └─ session_shutdown → vm.close()        │
│     └─ flags: --agent, --worktree-branch        │
├─────────────────────────────────────────────────┤
│  Gondolin VM (Alpine, ~35ms resume)             │
│  ├─ /workspace ← RealFSProvider(host cwd)       │
│  ├─ egress: allowedHosts only (HTTP 403 else)   │
│  ├─ TLS: MITM CA appended to system bundle      │
│  └─ agent user (non-root, uid 501)              │
├─────────────────────────────────────────────────┤
│  Credentials (injected at VM start, never baked)│
│  ├─ Pi OAuth: ~/.pi/agent/auth.json             │
│  ├─ MoltNet: .moltnet/<agent>/moltnet.json      │
│  └─ MoltNet env: .moltnet/<agent>/env           │
└─────────────────────────────────────────────────┘

Relation to #852

The agent runtime concept from #852 maps cleanly to pi extensions:

• A task = a registered command with tools + system prompt
• Dynamic tasks = dynamic extensions (pi install git:...)
• Runtime primitives (worktree, judge, MoltNet tools) = extension helpers
• Sandbox = Gondolin VM via tool operations pattern

The extension approach is more pragmatic than a standalone runtime: pi handles the agent loop, model selection, UI, and tool calling. We add isolation (Gondolin), identity (MoltNet), and persistent memory (diary entries).

Branch: exploration/pi-gondolin-runtime — full working extension at .exploration/pi-gondolin/src/moltnet-gondolin.ts