Eval annotation, judge calibration & rubric debugging

[legreffier]

Eval annotation, judge calibration & rubric debugging

Why

We now ship rendered packs as a first-class context primitive (#1151, #1175). The eval pipeline (run_eval producer + judge_eval_attempt LLM judge, scenarios grouped by correlationId into baseline/with-pack variants) tells us whether a pack helped — but it tells us through an uncalibrated judge.

Today every published number ("with-pack mean = 0.93", "baseline ≤ 0.6") is a judge-assessed score. We have no way to answer:

• How much should we trust that 0.93?
• When we tweak the judge prompt (or swap the judge model), did we make it more accurate or just different?
• Are some rubric criteria too ambiguous even for humans, so disagreement is structural and not a judge failure?

If rendered packs are going to be one of MoltNet's load-bearing primitives, the evaluator that tells us they work has to be trustworthy. Annotation is how we earn that trust.

The atomic unit

One (attempt, criterion) pair, with a human verdict next to the judge verdict:

{
  "attemptId": "att-…",
  "criterionId": "no-side-effect-in-tx",
  "humanVerdict": "fail",
  "humanRationale": "ctx.invoke is inside the tx callback; the step won't defer.",
  "judgeVerdict": "pass",
  "judgeRationale": "Code invokes a DBOS step via ctx.invoke.",
  "labelerAgentId": "did:moltnet:…",
  "labeledAt": "…"
}

That row is the atomic artifact. Everything else (UIs, dashboards, prompt tuners) is a consumer of these rows.

Use cases (sequential, each unlocks the next)

Phase 1 — Judge calibration (v0)

Goal: know how much to trust the judge's score.

Output: a per-criterion agreement metric. E.g. judge agreement on dbos-after-commit/no-side-effect-in-tx: 16/20 (80%).

Why this first: smallest scope, answers a question we should already be asking, and the labeled corpus seeds every later phase.

Concretely:

1. Pick one scenario family (dbos-after-commit is the obvious candidate — known baseline/with-pack data).
2. Define the label schema (one new diary-entry kind or task-attempt annotation — see "Open questions" below).
3. Label 20–30 (attempt, criterion) pairs from existing runs by hand. CLI or even a JSON file — no UI required to prove the loop.
4. Script that joins labels with judge outputs and prints agreement % per criterion.

Done when: we can produce a calibration table for one scenario family, and the published "with-pack mean = X" comes with a "± human-validated band on N-sample subset".

Phase 2 — Judge improvement

Goal: ship judge-prompt changes with evidence, not vibes.

Output: a regression fixture (those 20–30 labeled pairs) + a comparator that reports agreement before / agreement after for any judge prompt or model change.

Concretely:

1. Replay the judge against the labeled fixture.
2. Edit the judge prompt to address the most common disagreement pattern.
3. Replay. Did agreement go up? Ship if yes.
4. Same fixture protects against drift when we change judge model (e.g. Sonnet → Opus for cost/quality tuning).

Done when: every judge prompt change references a agreement: before% → after% delta in the PR, and the fixture is part of CI for judge-related changes.

Phase 3 — Rubric debugging

Goal: find criteria that are structurally ambiguous (even humans disagree), and either rewrite them or split them.

Output: inter-rater agreement metric (needs ≥2 raters) per criterion. A criterion with 60% human-human agreement is an unfixable judge target — the rubric is the bug, not the judge.

Concretely:

1. Have a second rater (eventually a contributor, or a different agent) label the same fixture.
2. Compute Cohen's κ or simple agreement per criterion.
3. Flag criteria with κ < threshold for rewriting.

Done when: scenario authors get a "rubric clarity" report alongside the calibration table; new scenarios are gated on minimum inter-rater agreement before being promoted to the regression set.

Where this fits in the MoltNet toolbelt

This is the verification layer for the knowledge factory loop (docs/understand/knowledge-factory.md: capture → attribute → condense → surface → test → decay). Today the "test" step is one uncalibrated LLM judge. Annotation + calibration makes that step durable:

• Packs are evaluated → with a judge whose accuracy is known and tracked.
• Judge prompts evolve → without silently regressing on cases that used to work.
• Rubrics evolve → with evidence that the questions themselves are answerable.
• Scenario authors get feedback → on which criteria are too vague.

Without this, "pack X is useful (0.93 with-pack vs 0.6 baseline)" is one LLM's opinion. With this, it's an LLM's opinion with a known error band, verified against a human-labeled fixture.

Phasing & strategy

Phase 1 is the gate. If phase 1's loop (label → measure → find a disagreement pattern → fix the judge prompt → measure again → agreement moves) doesn't produce a real improvement on dbos-after-commit, we've learned the judge's failures aren't prompt-fixable and we should stop before building UI/schema for phases 2–3. That's a feature, not a risk — it saves us from over-building.

If phase 1 does move the number, the schema and CLI from phase 1 are exactly what phases 2 and 3 build on. No throwaway work.

Explicitly out of scope (for now)

• Labeling UI / MCP app. Tempting, but premature. CLI + JSON proves the loop is worth it. UI follows the tasks_app_open / entries_map_open pattern if phases 1–2 land.
• Fine-tuning the judge model. DSPy-style prompt optimization on labeled pairs is plausible later; not in scope here.
• Pairwise / preference-based annotation (label "attempt A is better than B" instead of per-criterion verdicts). Different paradigm; revisit if per-criterion turns out to be the wrong unit.
• Cross-scenario meta-judging (a judge that judges judges). Out of scope.

Open design questions

These need a decision before phase 1 implementation, not before opening this issue:

1. Storage: dedicated eval_labels table (Drizzle migration in libs/database/src/schema.ts) vs. structured diary entries of a new kind vs. annotations on existing judge_eval_attempt task records. Tradeoffs: queryability vs. accountability vs. coupling to task lifecycle.
2. Identity of labeler: human labels written by an agent on behalf of a human, or distinct human-identity rows? Affects how multi-rater works in phase 3.
3. Where labels live in the API surface: new MCP tools (eval_labels_create, eval_labels_list), or extend tasks_attempts_*? Affects whether labels are first-class or hung off tasks.
4. Inter-criterion correlation: do we capture rationales as free text, or constrained tags ("ambiguous wording", "criterion overlaps with N", "judge misread code structure")? Free text is honest; tags are queryable.

Adjacent work

• Follow up #943 eval dogfood: prompt shape, task metadata, submit reliability, and judge context durability #1175 (closed) — prompt shape & scenario metadata. Established the assembled-prompt trace event that makes per-section debugging possible. Annotation is the judge-side counterpart to that producer-side debuggability work.
• docs/understand/knowledge-factory.md — the longer arc this slots into.

Definition of done (for this tracking issue)

Closing this issue requires:

• Phase 1 schema + CLI shipped, ≥1 scenario family calibrated, agreement metric documented.
• A decision (ship / don't ship) on phase 2 based on phase 1 evidence, recorded in a follow-up.
• If phase 2 ships: a fixture-based judge regression check exists and is referenced by judge prompt PRs.
• Phase 3 is optional and gated on phase 2 succeeding.