feat(#943): Phase 0 eval substrate dogfood — SDK, pi-extension, prompts, daemon, imposers#1151
Merged
Merged
Conversation
Contributor
|
Contributor
🚨 Dependency Audit — Vulnerabilities foundFull report |
Contributor
|
getlarge
force-pushed
the
feat/943-phase0-eval-substrate-dogfood
branch
2 times, most recently
from
May 15, 2026 12:48
54709e9 to
ac675da
Compare
The 32 KiB inline content cap on ContextRef.content blocked the #943 Phase 0 dogfood: .claude/skills/legreffier/SKILL.md is 35,225 bytes and is the planned skill-binding payload for the first proctored eval scenario set. The cap was sized for short example skills; protocol-heavy operator skills routinely exceed it. No callers asserted the old bound; tests are green. MoltNet-Diary: 17917279-5abb-4af3-bbcc-721d9958cb97
Add three files plus two script aliases: - tools/src/tasks/scenario.ts — reads evals/<skill>/scenario-N/ and projects task.md/criteria.json/eval.json into ContextRef bindings and a normalized Rubric (weights summed to 1.0, llm_checklist scoring per agentskills.io binary-first methodology). - tools/src/tasks/run-eval.ts — one-shot imposer per variant. --correlation-id REQUIRED (eval without it is pointless per #943 design). --skill <slug> reads .claude/skills/<slug>/SKILL.md as a kind: skill ContextRef. --prompt-prefix / --user-inline cover the remaining two Tier-1 ContextBinding kinds. - tools/src/tasks/judge-eval-variant.ts — single judge over N≥2 completed run_eval producers. Pre-checks producer status (exists, type=run_eval, completed, acceptedAttemptN!=null) before POST to fail fast before the daemon claims a doomed judge task. Both imposers are intentionally throwaway — retired in favor of server-side task templates once #1135 lands. MoltNet-Diary: 0632dcae-35f6-475a-9556-5511e73a9653
Extends MoltNetError with optional readonly validationErrors populated from the server's ProblemDetails.errors[] array. Imposer scripts and daemon-side error handlers can now branch on err.validationErrors?.length to print field-level rejection reasons instead of forcing operators to re-run with curl to see what was rejected. Refs: #943 MoltNet-Diary: f989af1e-0e12-4fb8-9849-fac8f59b8092
resumeCommands now accepts (string | ResumeCommand)[]. Object form takes `retries` and `retryBackoffMs` so consumers can opt brittle idempotent steps (pnpm install, go mod download) into a bounded retry budget without making the platform-critical steps retry. sandbox.json opts pnpm install into retries=2 and bumps pnpm's own --fetch-retries=8, lowering --network-concurrency=8 to reduce DNS pressure in the post-resume DHCP race window. Refs: #943 MoltNet-Diary: 45094a65-93d2-4e31-9013-75b93a139c98
…complete-rejection fallback Producer prompts now spell out that omitting `verification` when `input.successCriteria` is set causes the server to reject `submit_*` with VALIDATION_FAILED — observed during run_eval dogfooding where gpt-5.4-mini skipped the moltnet_get_task step entirely and the attempt surfaced as `lease_expired` instead of the real reason. Daemon's `finalizeTask` now catches `tasks.complete` failures and falls back to `tasks.fail` with a typed error code (`output_rejected_by_server` carrying the SDK's validationErrors, or `complete_call_failed` for everything else). Eliminates the silent-lease-expiry pathway when complete is rejected. Refs: #943 MoltNet-Diary: 2cd09391-f236-4054-96f5-590d54234300
Replaces --skill <slug> with --skill-path <file>. The slug-only form silently assumed the .claude/skills/<slug>/SKILL.md layout — brittle outside this repo. Path form derives the slug from the parent dir name and validates it against the ContextRef.slug pattern. Imposers now surface MoltNetError.validationErrors (landed in f77d3cc) so a server-side rejection prints the offending field without requiring an out-of-band curl probe. Refs: #943 MoltNet-Diary: 070f2272-166d-4f17-989d-ecb4beb3426c
MoltNet-Diary: e7e99a45-11fb-4ab0-8fc4-8dd5ee494aec Task-Group: sandbox-resume-install-retries Task-Family: bugfix
MoltNet-Diary: f0d334b3-eaac-4dba-b2e7-df691b95d46f Task-Group: eval-submit-tool-enforcement Task-Family: bugfix Task-Completes: true
MoltNet-Diary: ab38ba6c-51fc-4067-9c91-453caa8470ac Task-Group: legreffier-eval-runner-modes Task-Family: feature Task-Completes: true
MoltNet-Diary: 64fee1fb-091d-47ef-ba7d-b68c626b3a5f Task-Group: eval-runtime-dogfood-followups Task-Family: bugfix Task-Completes: false
MoltNet-Diary: 77cd5648-c090-49f9-a1c9-61fc67987038 Task-Group: eval-judge-attempt-model Task-Family: refactor Task-Completes: false
getlarge
force-pushed
the
feat/943-phase0-eval-substrate-dogfood
branch
from
May 16, 2026 12:25
ac675da to
29b529f
Compare
MoltNet-Diary: a4ff44ee-57bf-4910-933f-5bfd92715bdc Task-Group: eval-judge-attempt-model Task-Family: bugfix Task-Completes: false
MoltNet-Diary: 82b9b9b8-5a7c-459b-895a-7222b2e2603d Task-Group: eval-judge-producer-context Task-Family: bugfix Task-Completes: false
Owner
|
@legreffier /complexity-review |
MoltNet-Diary: 7ddad1a8-950a-4fac-99f4-e734ae450361 Task-Group: eval-judge-workspace-preservation Task-Family: bugfix Task-Completes: false
MoltNet-Diary: af2687b9-7fcd-49fc-a667-e946115ff0d5 Task-Group: eval-context-inline-prompt-discipline Task-Family: bugfix Task-Completes: false
Contributor
Author
Review —
|
| Where | Adds | Owns |
|---|---|---|
libs/pi-extension/src/runtime/runtime-instructor.ts (104 lines) |
identity, diary discipline, accountable commits, skill-pack disclaimer | system prompt, every task |
libs/agent-runtime/src/context-bindings.ts |
concatenates prompt_prefix → systemPromptPrefix, user_inline → userInlineSuffix, writes skills |
per-task fragment |
libs/pi-extension/src/runtime/inject-task-context.ts |
wraps for Gondolin VM | VM glue |
libs/agent-runtime/src/prompts/run-eval.ts |
scenario, correlation, self-verification, final output | first user message |
libs/pi-extension/src/runtime/execute-pi-task.ts:460–625 |
glues all four with inline conditionals | nobody — 160-line block inside a 700-line function |
When debugging a producer attempt, you can't cat one file to see "what did the model see?" — you mentally interleave four sources.
Minimum-viable refactor
One type + one assembler in libs/agent-runtime:
// libs/agent-runtime/src/prompts/assemble.ts
export interface AssembledPrompt {
systemPrompt: string; // runtime instructor + system fragments
firstUserMessage: string; // task body + user fragments
skills: Skill[]; // pi <available_skills> entries
// For debugging:
sections: Array<{ source: string; role: 'system' | 'user' | 'skill'; bytes: number }>;
}
export function assembleTaskPrompt(args: {
task: Task;
ctx: TaskUserPromptContext;
runtimeInstructor: string;
context: TaskContext;
deliverSkill: ContextDeliverer['skill'];
}): Promise<AssembledPrompt>;execute-pi-task.ts then calls one function instead of orchestrating four. The 460–625 block collapses to:
const assembled = await assembleTaskPrompt({ task, ctx, runtimeInstructor, context: rawContext, deliverSkill });
await emit('info', { event: 'prompt_assembled', sections: assembled.sections });
// pass assembled.systemPrompt, firstUserMessage, skills to piThe sections array is the unlock: dump it to the attempt's event log and "what did the model see?" becomes a SQL query on task_events instead of a debug session. It's also what eval replay needs — replay today shows bytes, not framing.
Per-task-type ordering becomes data, not control flow. run_eval declares context-first; fulfill_brief declares instructor-first; nobody re-implements the wrap logic.
Small cleanups while in the area
buildSelfVerificationBlock— inlineinput.successCriteriadirectly, drop themoltnet_get_taskround-trip. WithinputCidpinning the input and a 64 KiB cap, the round-trip costs more attention than it saves. The agent already has the bytes; making it re-fetch them is theatre. EchoinginputCidback stays useful as a "have you seen the input you're scoring against" check.- Strip rotting comments in
execute-pi-task.tsandinject-task-context.ts("// Slice 1.5 of #943 — wire …"). The diary entries on this branch are the durable record.
(Earlier draft suggested trimming buildFinalOutputBlock — withdrawn. Operator confirmed that the MUST/MUST NOT wording is load-bearing: weaker agents drop the final submit when the framing softens, which fails the attempt at the last hop. Keep it loud.)
PR-level notes
- Commit shape is good. 11 independently reviewable commits with diary trailers.
Task-Group: <slug>would have helped — the four sub-features (SDK errors, pi-extension retries, verification consequence, imposer flags) feel like four separate task chains. 439693aa fix(tasks): raise ContextRef.content cap to 64 KiB— fine, butmaxItems: 5 × 64 KiB = 320 KiBper task. With runtime instructor + scenario + verification + final-output, you're well into six-figure token counts before the model sees the question. Worth a procedural entry recording the new effective budget.88db35dc(pi-extension retries) andf7d37abe(SDK validationErrors) are orthogonal to eval prompts and could ship independently. Splitting would shrink this PR by ~30%.- Test-plan checkboxes are unchecked. At minimum item 3 (
submit_run_eval_outputwithoutverification→tasks.failwithoutput_rejected_by_server, notlease_expired) should be exercised before merge — it's the one that motivatedd9542099. - Missing episodic: no entry captures "qwen3.5 implemented Keto inside the DBOS transaction despite pack being present" with the actual model output. That's the most valuable artifact from this dogfood. Worth
episodicwithscope:agent-runtime,scope:prompts,severity:highso future eval-runner work can find it by tag.
Suggested commit chain
feat(tasks): add eval_context binding, materialize /workspace/context-pack.md—libs/tasks/src/context.tsschema +inject-task-context.tsdeliverer. Diary: semantic (decision, "why new binding vs overloading prompt_prefix"). Risk: medium.refactor(agent-runtime): centralize prompt assembly in assembleTaskPrompt— extract 460–625 fromexecute-pi-task.tsintolibs/agent-runtime/src/prompts/assemble.ts. Diary: semantic + procedural. Risk: medium.feat(eval): reorder run_eval prompt for context-first, inline successCriteria—run-eval.tsprompt + imposer emitseval_context,buildSelfVerificationBlockinlines criteria. Diary: procedural,Task-Completes: trueafter re-running scenario-0 against qwen3.5. Risk: medium.
The metric that tells you whether the prompt-shape hypothesis was right: re-run scenario-0 baseline-vs-context on qwen3.5 after commit 3 (gpt-5.4-codex now passes, so it's not a discriminating model anymore). If the score still doesn't move, e:ad605aea's last sentence was correct and you've at least falsified the prompt-shape hypothesis cleanly.
Sources: e:ad605aea · e:199a9014 · e:64fee1fb · e:cb73f622 · e:ff60dd52 — all on branch:feat/943-phase0-eval-substrate-dogfood, agent 1671.
Contributor
Author
Correction + addendum on commit
|
Contributor
Author
Two more —
|
| # | Failure | Detected | Recoverable today? |
|---|---|---|---|
| 1 | Model never called the submit tool. Session ended naturally on end_turn, model thought it was done. |
submitToolHandle.getCaptured() === null at line 911 → fail with output_missing. |
No. Whole attempt wasted. This is the one that hurts. |
| 2 | Model called the tool with invalid args (schema or cross-field). | validateTaskOutput() returns errors → tool returns isError: true with the corrective message → model retries on the next turn. |
Yes, already. This is the recoverable case the submit-tool was designed for. |
| 3 | Model called with valid args, server rejects on cross-field rule (e.g. verification required iff successCriteria). |
tasks.complete throws → finalize.ts catches and falls back to tasks.fail with output_rejected_by_server + field-level details. |
Logged, not retried. Right call — the same bytes would fail the same way on retry. |
Mode #2 already works. Mode #3 is correctly terminal. The lever is mode #1, and it's where automatic retry pays off.
Recommendation: in-session resubmit nudge before retrying the attempt
Cheapest, highest-leverage fix. Inside execute-pi-task.ts, before bailing with output_missing, insert one additional user turn:
// Around line 939 — submit tool exists, model never called it
} else if (submitToolHandle) {
// First, give the model one explicit chance to submit. Most "forgot
// to call submit" failures are simply the model ending the session
// narratively ("done!") without realizing the runtime requires the
// structured call. A single targeted nudge recovers the majority of
// these without paying the cost of a fresh attempt (new VM, fresh
// context tokens, lost workspace state).
const nudgeBudget = opts.submitNudgeBudget ?? 1;
for (let i = 0; i < nudgeBudget && !submitToolHandle.getCaptured(); i++) {
await emit('info', { event: 'submit_nudge_attempt', i });
await session.prompt(
'You ended the session without calling `' +
submitToolName(task.taskType) +
'`. The runtime requires that structured call to record your output — without it the attempt fails even though your work may be correct. Call the tool now with your final output.',
);
}
const recaptured = submitToolHandle.getCaptured();
if (recaptured) {
parsedOutput = recaptured;
parsedOutputCid = await computeJsonCid(recaptured);
recordTaskOutputParseResult({ taskType: task.taskType, model: opts.model, code: 'captured_via_nudge' });
} else {
parseError = {
code: 'output_missing',
message: 'Agent did not submit output… (after ' + nudgeBudget + ' nudge(s))',
};
}
}Properties:
- Cheap: stays in the same VM, same context window, same session. One extra turn vs. a whole new attempt (fresh VM = seconds + tokens; full context reseed = $$).
- Single source of truth: still uses the existing submit tool and its validator. No duplicate code path.
- Observable: new
captured_via_nudgecounter shows you the rate at which this fires — that's the metric for "is the prompt strict enough" vs "is this just intrinsic model floppiness." - Bounded:
submitNudgeBudgetdefaults to 1; weaker models could get 2; budget cap means it can't loop. - Eval-clean: doesn't bias quality — the model still chooses its content. Only nudges presence of the call.
Don't add task-level auto-retry yet
Tempting, but it's the wrong layer for this failure mode:
- A new attempt means a new VM, fresh prompt assembly, lost workspace state (the producer's intermediate files), and the model re-doing reasoning from scratch. For mode feat: initial MoltNet monorepo scaffolding #1 — "forgot the last step" — this is wildly expensive when an in-session nudge would have worked.
- For mode Claude/builder journal vklid #3 (server reject), retrying makes no sense; same bytes, same rejection.
- For mode feat: initial MoltNet monorepo scaffolding #2 (bad args), in-session recovery already handles it.
So there's no failure mode where attempt-level retry is the right answer at this layer. The imposer can already create a fresh task with the same input if it wants, and that's the correct place for "this whole attempt was unrecoverable, try again with a fresh sandbox" policy — eval imposers, producer imposers, and pr-review imposers all have different retry tolerances and should decide for themselves.
If you do want a daemon-side retry knob later, gate it strictly:
- Allow retry on:
output_missing(after nudge exhausted),llm_api_error(transient),bash_timeout_cap_exceeded,prompt_build_failed. - Refuse retry on:
output_rejected_by_server(deterministic),output_validation_failed(in-session recovery already failed),task_cancelled,max_turns_exceeded(model got stuck — retrying won't unstick it).
But the in-session nudge will eat 80%+ of the mode-#1 cases on its own. Ship that first, measure the rate, then decide if a second layer of retry is even needed.
Suggested patch order
feat(pi-extension): in-session resubmit nudge before output_missing failure— ~30 LoC inexecute-pi-task.ts, newsubmitNudgeBudgetopt (default 1), newcaptured_via_nudgecounter label. Diary: semantic (decision: in-session over attempt-retry) + procedural. Risk: low.Task-Completes: trueafter observingcaptured_via_nudge> 0 across a scenario run.fix(agent-runtime): inline successCriteria in buildSelfVerificationBlock— drop themoltnet_get_taskround-trip. Risk: low. Re-run scenario-0 on qwen3.5 to confirm score doesn't regress.
The nudge counter is what gives you the empirical data to decide whether to invest in (3) — daemon-side attempt retry. If captured_via_nudge recoveries are >30% of would-be failures, the eval substrate just got much more reliable for the same scoring weight.
Sources: imposer comment in tools/src/tasks/run-eval.ts, libs/pi-extension/src/runtime/submit-output-tool.ts, libs/pi-extension/src/runtime/execute-pi-task.ts:905–965, apps/agent-daemon/src/lib/finalize.ts, plus the gpt-5.4 result (1.0 vs 0.6) the operator reported in this thread.
Contributor
Author
Correction — "imposer" framing and retry layeringI leaned on "imposer" in the last comment and built a retry layering on top of it. Both were wrong, and #1158 is exactly the corrective — already filed (by this account, 2026-05-15), and the vocabulary is settled there: proposer / propose / claim. Threading the promises model back through the retry discussion: In promises theory, no party can impose work on an agent. Promises are voluntary self-bindings. A proposer publishes a proposal; an agent voluntarily claims it; the claim is the promise. There is no mechanism upstream of the claim by which anyone can force a second attempt onto whoever claims next. The agent that claims is the only party that can promise to act, and only the same agent (or another that voluntarily claims) can ever produce another attempt. So when I wrote "the proposer can already create a fresh task if it wants, and that's the correct place for retry policy" — that smuggled in coercion semantics the model explicitly rejects. A proposer re-proposing after a failed attempt is creating a new offering, not retrying the prior attempt. There is no continuity in the trust model; calling it "retry" misleads. What this changes in the recommendationThe in-session "nudge" survives, but the framing flips:
The prompt copy carries the difference:
One line of prompt copy; one big conceptual shift. The nudge becomes a feedback channel from substrate to agent, not a re-prompt that overrides agent choice. What this rules out (not "defers")Three retry layers I floated should be ruled out, not deferred:
What remains, both grounded in agent agency
The substrate's responsibility ends at emitting clear failure signals ( Concrete patch to
|
Contributor
Author
Re-sequencing pointer — filed #1176Filed #1176, which supersedes the "in-session feedback nudge" recommendation I posted earlier in this thread. Short version: the submit-tool call is part of what the agent promises when claiming the task, so it belongs in Re-sequencing impact on the patch chain I proposed:
#1176 implementation order (separate PR, not this one):
This PR can land as-is for the eval-substrate scope; #1176 is the contract-level fix that follows. |
MoltNet-Diary: 06a2c4a1-2d63-4f26-aa6e-49e2db4d7de5 Task-Group: eval-pr-ci-fixes Task-Family: bugfix Task-Completes: true
getlarge
approved these changes
May 18, 2026
MoltNet-Diary: 4d4bc580-15ca-4c0c-b7be-9853cee6991c Task-Group: agent-daemon-typecheck-fix Task-Family: bugfix Task-Completes: true
MoltNet-Diary: 897611ca-2523-4b75-b02c-96a50854523b Task-Group: agent-daemon-e2e-local-env Task-Family: bugfix Task-Completes: true
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Five commits landed while dogfooding the
run_eval+judge_eval_varianteval substrate against the legreffier skill (#943 Phase 0). Each commit is independently reviewable and self-contained.Commits
feat(sdk): surface server validationErrors on MoltNetError(f77d3ccd)MoltNetError.validationErrors(optional readonly array) populated fromProblemDetails.errors[]err.validationErrors?.lengthto print field-level rejection reasonsf989af1e-0e12-4fb8-9849-fac8f59b8092(semantic, rejected alternatives spelled out)feat(pi-extension): per-step retry policy on resumeCommands(8a00e9fb)resumeCommandsschema:(string | ResumeCommand)[]whereResumeCommand = { run, retries?, retryBackoffMs? }sandbox.jsonoptspnpm installintoretries: 2and bumps pnpm's own--fetch-retries=8, lowering--network-concurrency=8to reduce DNS pressureEAI_AGAINfrom registry.npmjs.org on freshly-resumed VMs (DHCP race)45094a65-93d2-4e31-9013-75b93a139c98fix(agent-runtime,agent-daemon): explicit verification consequence + complete-rejection fallback(badfb96e)buildSelfVerificationBlock) now spells out the server-side rejection consequence — observed during run_eval dogfooding where the agent submitted output withoutverificationand the attempt surfaced aslease_expiredinstead of the realVALIDATION_FAILEDfinalizeTaskcatchestasks.completefailures and falls back totasks.failwith a typed error code carrying the SDK'svalidationErrorscompleteis rejected2cd09391-f236-4054-96f5-590d54234300feat(tools): --skill-path on run-eval, surface SDK validationErrors(a664fb9b)--skill <slug>with--skill-path <file>. Slug derived from parent dir name (Claude-style<slug>/SKILL.mdlayout)tasks.createcall070f2272-166d-4f17-989d-ecb4beb3426cOut of scope (filed separately)
moltnetCLI does not pickendpoints.apifrom the credentials file (cost ~5 min during this session; episodic diary entry filed)Test plan
pnpm installresume retry triggers if EAI_AGAIN reappearssubmit_run_eval_outputwithoutverificationnow surfaces astasks.failwithoutput_rejected_by_server(notlease_expired)--skill-pathrejects a non-conforming parent dir name