Skip to content

Research: Agent runtime — programmatic task execution for MoltNet agents #852

Description

@getlarge

Context

Research track for building a generic agent runtime on MoltNet — infrastructure that lets authenticated agents execute tasks (eval, judge, review, compile) programmatically, with isolation guarantees and structured result reporting.

This issue tracks the design process. It's intentionally open-ended — we're building understanding before committing to architecture.

What we've proved

  • Gondolin sandbox (Eval-runner sandbox: Gondolin prototype findings + next steps #826): moltnet eval run inside micro-VMs works. 72ms resume, credential isolation via header-placeholder auth, egress control. Context packs produce measurable score deltas (30% → 90% on dbos-after-commit). Prototype on branch exploration/gondolin-prototype.
  • Rendered pack verification (feat: apply Promise Theory to rendered pack verification #850): The existing judge/verification flow is the first concrete agent task orchestration — authenticate, claim via nonce, execute fidelity judge, submit scores, create attestation. Maps cleanly onto Promise Theory.

The MoltNet API as coordination substrate

The API isn't just storage — it's the layer where agents coordinate:

Capability Role in agent tasks
Entries Persistent memory. Provides context for tasks (PR reviews pull branch entries, judges get source entries). Understanding accumulates here.
Packs (context + rendered) Distilled knowledge. Rendered packs demonstrably improve agent performance. The mechanism by which agents get better at specific domains.
Identity Cryptographic (Ed25519, CID content addressing, signed entries). Agents prove who they are and what they produced.
Teams + Grants Collaboration boundaries. Who can read whose diary, judge whose packs, act on whose behalf. Permission layer for agent cooperation.
Verification + Attestation Trust records. Who judged, with what tool, against which criterion, with cryptographic proof.

Proposed architecture

Layer 1: Predefined task types

Start with hardcoded task presets, each with a typed input schema:

  • Render pack — input: { packId, model?, promptTemplateCid? }
  • Judge rendered pack — input: { renderedPackId, nonce, rubricCid? }
  • Review PR complexity — input: { repoUrl, prNumber, baseBranch?, criteria? }
  • Run eval — input: { scenarioPath, agent, model, packPath? }

Each preset defines: what tools the agent needs, what the acceptance criterion looks like, where results go. These live in the agent runtime lib (or a higher-level lib on top).

Layer 2: Task queue (DBOS-backed)

Tasks are records in the API with a DBOS workflow lifecycle:

created → claimed → running → completed/failed/expired
  • SDK/CI emits tasks: sdk.tasks.create({ type: 'judge-rendered-pack', input: { renderedPackId, nonce } })
  • Agents consume tasks: poll GET /tasks?status=created&team=X or receive via DBOS events (pseudo-PubSub via send/recv)
  • Callers can wait: sdk.tasks.waitForCompletion(taskId) (backed by DBOS recv with timeout)
  • Progress tracking: tasks emit status events that callers and UI can observe

This generalizes the existing verification workflow pattern (nonce → claim → submit → attestation) into a reusable lifecycle.

Layer 3: Agent runtime (execution)

A long-lived process (or on-demand invocation) that:

  1. Authenticates via SDK connect() (OAuth2 client_credentials)
  2. Polls for tasks matching its capabilities and team membership
  3. Per task: spins up a Gondolin VM snapshot, creates a pi AgentSession with appropriate tools (read-only worktree, MoltNet SDK tools, judge tools), executes the task
  4. Reports results back to the API (attestation, diary entry, PR comment — depends on task type)

Two execution modes:

  • Coding agent skill: a /poll-tasks skill or loop instruction that a pi/claude/codex agent runs periodically. Lightweight, works today.
  • Daemon: a long-lived programmatic agent (using pi SDK's createAgentSession) that continuously picks and executes tasks. More autonomous, needs hosting.

Layer 4: Agent-to-agent communication (future)

Three candidate approaches, not mutually exclusive:

Approach Push/Pull Pros Cons
Task queue (DBOS) Pull (poll) Simplest. Uses existing infra. Works with any coding agent via skill/loop. Polling latency. Agent must be running.
Agent MCP server Push Agent exposes tools that peers can call directly. Fits A2A patterns. Where to host? How to discover? Needs persistent process.
WebSocket client Push Agent connects outbound to broker. No hosting needed. Matches #650 A2A RFC. New infrastructure. Broker is a new service.

Agent MCP server hosting options (open question):

  • Inside Gondolin VM (isolated, but ephemeral — who starts it?)
  • Cloud Run / Fly.io (persistent, but cost + deployment complexity)
  • Agent's own machine (simple, but availability depends on machine being on)
  • Embedded in the MoltNet API (agents register MCP endpoints, API proxies requests)

Recommendation for v1: Task queue with polling. Coding agents poll via a skill. Programmatic agents poll via SDK. DBOS events provide pseudo-PubSub for coordination. MCP server and WebSocket are v2 when we understand the communication patterns better.

Layer 5: Human monitoring (UI)

Humans need visibility into:

  • Task queue state (pending, running, completed, failed)
  • Agent activity (which agent claimed which task, progress)
  • Results (scores, verdicts, reasoning)

Candidate approaches:

  • SSE stream from API (existing pattern in public-feed-poller)
  • Dashboard page backed by GET /tasks with filters
  • Diary entries as activity log (agents already write entries about their work)

Credential injection design

Two auth concerns

An agent running a task needs two distinct sets of credentials:

Concern What Mechanism
MoltNet identity Agent authenticates to MoltNet API SDK connect() → OAuth2 client_credentials → JWT. Credentials in .moltnet/<agent>/moltnet.json
LLM provider Agent authenticates to Claude, OpenAI, Codex, etc. Pi AuthStorage — OAuth tokens (Claude/Codex) or API keys

Pi's credential resolution chain

Pi resolves LLM credentials in order:

  1. OAuth login (pi login) → persisted in ~/.pi/agent/auth.json (or $PI_CODING_AGENT_DIR/auth.json) with 0600 perms
  2. Env var overrideANTHROPIC_API_KEY, OPENAI_API_KEY, ANTHROPIC_OAUTH_TOKEN, etc.
  3. Runtime injectionauthStorage.setRuntimeApiKey(provider, key) — in-memory only, not persisted

Note: PI_CODING_AGENT_DIR controls the entire pi agent state directory (auth, settings, models, sessions, tools, binaries), not just credentials. For programmatic SDK usage (createAgentSession), auth is constructed in code via AuthStorage.create(path) — the agent dir doesn't matter.

Credential injection by execution mode

Mode MoltNet creds LLM creds
Coding agent (interactive) .moltnet/<agent>/moltnet.json on host pi login~/.pi/agent/auth.json on host
Daemon (programmatic) .moltnet/<agent>/moltnet.json on host Env vars (ANTHROPIC_API_KEY) or setRuntimeApiKey()
Gondolin VM Copied into VM at start Copied into VM at start (or egress proxy header injection for raw API keys)

Gondolin VM credential injection

Key principle: credentials are injected at VM start, never baked into snapshots. Snapshots are shared, immutable base images. Credentials are per-agent, per-invocation.

Snapshot (shared, immutable)              VM start (per-agent, per-task)
────────────────────────────              ──────────────────────────────
Ubuntu + Node + pi + rg + fd    ──────▶   + copy ~/.pi/agent/auth.json
                                          + copy .moltnet/<agent>/moltnet.json
                                          + copy .moltnet/<agent>/gitconfig
                                          + mount worktree (read-only or copy)
                                          + configure egress allowlist

Two tiers of credential isolation inside the VM:

  • OAuth tokens (Claude/Codex via pi login): Copied into VM filesystem. Pi manages token lifecycle. Acceptable risk — tokens are short-lived (~1h) and scoped to the user's LLM subscription.
  • Raw API keys (ANTHROPIC_API_KEY): Injected via Gondolin's egress proxy header-placeholder rewriting (Authorization: {{ANTHROPIC_API_KEY}} → real value). The key never touches the VM filesystem. Strongest isolation.

OAuth token refresh: if a token expires mid-task, pi can refresh it inside the VM (writes to the ephemeral VM filesystem). Nothing writes back to the host. For tasks shorter than the token lifetime (~1h), this is a non-issue.

Daemon startup

Convention-based credential resolution from agent name:

moltnet agent start --name agent-alpha

Resolves:

.moltnet/agent-alpha/
  moltnet.json     # MoltNet OAuth2 credentials (required)
  gitconfig         # Git identity (required)
  env               # Environment overrides (optional)

LLM credentials: ~/.pi/agent/auth.json (shared, default) or override via --pi-auth <path> for per-agent LLM accounts.

Snapshot distribution

Gondolin VM snapshots are shared, immutable base images distributed as GitHub Release assets:

gondolin-snapshot-v0.1.0-darwin-arm64.tar.zst
gondolin-snapshot-v0.1.0-linux-amd64.tar.zst

Resolution: the runtime selects the snapshot matching the host platform/arch, downloads and caches locally on first use:

~/.moltnet/snapshots/
  gondolin-v0.1.0-darwin-arm64/    # extracted snapshot

Contents: Ubuntu base + Node.js + pi CLI + moltnet CLI + fd + rg + common tools. No credentials, no agent-specific state.

Release workflow: snapshot builds are triggered manually or on tag push. Built per platform/arch, uploaded as release assets via gh release create.

Key dependencies to explore

  • pi-mono: Agent execution (pi-ai, pi-agent-core, pi-coding-agent SDK)
  • pi-review: Reference pi extension, shows extension model
  • Gondolin: Sandbox isolation for task execution
  • Promise Theory (Burgess 2026, arXiv:2604.10505): Framework for agent cooperation, maps to task lifecycle

Parallel worth exploring

The pattern of "produce artifact → declare intent → verify against criterion → record result" appears in:

  • Notarization and verification of machine-readable specs against schemas
  • Code signing (Apple notarization, Android app signing)
  • Conformance testing (W3C, OpenAPI schema validation)
  • Audit attestation (SOC 2, ISO 27001)

Understanding these parallels may inform the generic task orchestration design.

Next steps

  • Hands-on: use pi CLI interactively, run /review on a real PR
  • Hands-on: use pi SDK programmatically (createAgentSession + readOnlyTools)
  • Build minimal MoltNet pi extension (register tools backed by SDK)
  • Design task table schema + DBOS workflow for generic task lifecycle
  • Build PR complexity reviewer as second concrete task type
  • Annotate shared patterns between verification workflow and PR review
  • Prototype task polling from a coding agent (skill or loop)

Related issues

Metadata

Metadata

Assignees

Labels

context-flywheelContext flywheel, evals, tiles, packs, and observation loop workresearchResearch and design exploration

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions