Evals: agent-runtime prompt harness across Ollama Cloud models (#1588)#1590
Draft
legreffier[bot] wants to merge 5 commits into
Draft
Evals: agent-runtime prompt harness across Ollama Cloud models (#1588)#1590legreffier[bot] wants to merge 5 commits into
legreffier[bot] wants to merge 5 commits into
Conversation
Contributor
|
Contributor
🚨 Dependency Audit — Vulnerabilities foundFull report |
Contributor
|
legreffier
Bot
force-pushed
the
issue-1588-evals-test-agent-runtime-prompt
branch
from
edd6ddb to
7b6625e
Compare
Contributor
Author
Complexity Review JudgmentComposite Score: 1.0/1.0
Rationale: |
The @moltnet/agent-eval lib: scenario reader, run_eval/judge_eval_attempt input builders (rubric hidden from producer), the stage-1 checkGates deterministic evaluator, and shared Pi-config helpers extracted from the live-ollama e2e test. 25 unit tests, no LLM dependency. Foundation for the agent-runtime prompt eval harness (#1588). MoltNet-Diary: 0c348d10-0bfe-45b6-9255-c71cc6c24576 Task-Group: agent-eval-harness Task-Family: feature
submit-output-compliance and prompt-assembly-integrity (pure-gate), instruction-following-bullets (exercises the pinned judge rubric), and tool-use-artifact-upload (requireToolCalls gate under shared_mount). A corpus test parses every scenario so malformed ones fail CI. MoltNet-Diary: dd78066d-3b8a-447e-bdb5-7267227dcb58 Task-Group: agent-eval-harness
live-ollama-evals.e2e.test.ts runs the evals-v2 corpus against one pinned Ollama Cloud model and asserts deterministic gates only (no judge) — the per-PR regression tripwire. Refactors live-ollama.e2e.test.ts to use the shared writeAgentCredentials/writePiConfig from @moltnet/agent-eval. MoltNet-Diary: b37e089b-3bab-4b1e-ac74-05bff453b8e6 Task-Group: agent-eval-harness
runMatrix sweeps (model x scenario), gating the pinned-model judge behind stage-1 deterministic gates (gate fail => composite 0, judge skipped). eval-matrix-runner.ts wires the live effects and writes score-matrix.json, sequencing producer then judge via runOnce (warm slot is DB-backed, so it survives across calls). 4 orchestration unit tests; judge leg deferred to a live run. MoltNet-Diary: a87690d3-f426-4160-aef1-0b44dd255a3c Task-Group: agent-eval-harness
eval-matrix.yml sweeps the evals-v2 corpus across N Ollama Cloud models on a schedule + workflow_dispatch, using :ci-main images, and uploads score-matrix.json. Not per-PR (exceeds the 15-min budget). Also aligns the runner default models to the maintained .pi/models.json list (the previous defaults were retired). MoltNet-Diary: 51d57f47-e446-427f-a116-878c95ee95cf Task-Group: agent-eval-harness
legreffier
Bot
force-pushed
the
issue-1588-evals-test-agent-runtime-prompt
branch
from
7b6625e to
8f2024b
Compare
Contributor
Author
⏸️ Parked — pending runtime-prompt preset designThis draft is intentionally parked. The runtime prompt is being reworked so that it is no longer hardcoded in That reshapes the eval's variable under test: from "model, against a fixed hardcoded prompt" → "preset (± model)". Fixing the scenario prompts now would be throwaway, so scenario calibration is on hold until the preset shape lands. What stays valid (preset-agnostic, keep warm)
What will need retargeting once presets exist
Known-broken (expected, not a harness bug)The one failing CI job (E2E Tests (Agent Daemon)) is the live gate smoke: all 4 producers failed the No action needed on this PR until the preset work is further along. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1588.
What this adds
A deterministic-first eval harness for the agent-runtime prompt, so we can (a) catch prompt regressions before merge and (b) track which model/prompt is better over time — across Ollama Cloud models.
The key finding while scoping this: most of the machinery already existed (
run_eval+judge_eval_attempttask types with hidden-rubric producer/judge separation, a live-ollama CI path, the deterministic gate signals the runtime already emits). So this is wiring + a trustworthy fresh corpus, not a new eval engine.New:
libs/agent-evalreadScenario— parsesevals-v2/<slug>/{prompt.md,eval.json,rubric.json,gates.json}and fails loudly on drift (the deliberate opposite of the orphanedtools/src/tasks/scenario.ts).buildRunEvalInput/buildJudgeInput— turn a scenario intorun_eval/judge_eval_attemptinputs; the rubric is hidden from the producer, injected only on the judge side.checkGates— the stage-1 deterministic evaluator (assertsexecute_startmodel/workspace,prompt_assembledsections, tool-call presence/absence, and a schema-valid capturedRunEvalOutput) reading the attempt message stream. Depends only on a narrowGateAgentinterface, so it carries no runtime SDK dependency and is unit-tested with a fake.runMatrix/summarizeMatrix— pure sweep orchestration with injected effects.writeAgentCredentials/writePiConfig— extracted fromlive-ollama.e2e.test.tsso the smoke test and matrix runner share one copy.New:
evals-v2/corpus (4 scenarios)submit-output-compliance,prompt-assembly-integrity(pure-gate),instruction-following-bullets(exercises the pinned judge),tool-use-artifact-upload(requireToolCallsgate). A corpus guard test parses every scenario so a malformed one fails CI.evals/(the old dir) is left untouched — it has legit rendered-pack scenarios worth mining before any cleanup.New: per-PR smoke + nightly matrix
apps/agent-daemon-e2e/src/live-ollama-evals.e2e.test.ts— runs the corpus against one pinned model, gates only (no judge). Fast regression tripwire in the existinge2e-agent-daemonjob.apps/agent-daemon-e2e/src/eval-matrix-runner.ts+.github/workflows/eval-matrix.yml— nightly/on-demand sweep of N models × scenarios, gate-then-pinned-judge, emittingscore-matrix.json.Design: why gates + a pinned judge
The "inception" risk is that MoltNet's own judge runs through the same runtime prompt under test. The hybrid design breaks that loop:
Verification status
@moltnet/agent-eval), including the anti-inception path (gate fail → composite 0 → judge skipped).libs/agent-evalandapps/agent-daemon-e2e.runOncecalls (same mechanism as the continuation test, but not exercised here). Validate before trusting composites:Heads-up for reviewers / follow-ups
nx synclandmine:nx.json'ssync.applyChanges: truemakesnx syncrewrite the roottsconfig.jsonreferences from 49 →[](Nx 22.7). Kept out of every commit here, but you'll hit it locally —git diff tsconfig.jsonafter installs and restore if emptied; useMOLTNET_SKIP_NX_SYNC=1. (Corrects a stale diary claim.).pi/models.jsonids (qwen3.5:397b-cloud,glm-5.2:cloud,gpt-oss:120b-cloud). Notelive-ollama.e2e.test.tsstill hardcodesqwen3-coder:480b-cloud, which is not in the maintained list — worth a separate look.:ci-maindependency: the nightly workflow pulls:ci-mainimages; it will fail until a main build has published them.docs/use/context-pack-evals.md) — same machinery, foldable in later.🤖 Authored by LeGreffier (fingerprint
1671-B080-99BF-4270).