Proctored evals on the task runtime (supersedes #523)

[legreffier]

Supersedes #523. Closing #523 because the new task runtime (libs/task-service + libs/tasks) plus the rubrics resource planned in #881 collapse the original design down to a thin task-types + query feature. No new entities, no bespoke session lifecycle, no API-side telemetry pipeline.

Why #523 collapses

#523 was designing from scratch what the task runtime now provides:

#523 concept	Task runtime equivalent
eval_sessions (with/without pair)	correlation_id grouping the 4 tasks
eval_scores claim/submit + nonce	Existing task claim / complete lifecycle
Bespoke proof receipt (Ed25519)	crypto-service claim + complete attestations
Trust tiers (vitro / vivo / Gondolin)	executorTrustLevel enum (self_declared … sandbox_attested)
Criteria withheld until claim	Existing claim releases full input only to claimer
eval_tasks (immutable {task_md, criteria} CID)	input_cid (task body) + rubric_cid from #881
OTel forwarding pipeline (Phase 4)	traceparent already returned by claim; trajectory lives in OTel/Axiom

Proposed shape

Wait for #881 (first-class rubrics with CID lookup), then:

1. Register two task types in libs/tasks/src/task-types/:
   • run-eval — input { scenarioCid, variant: 'with' | 'without', packCid?, model, mode: 'vitro' | 'vivo' }. Output: transcript hash + token totals.
   • judge-eval-variant — input { runTaskId, rubric_cid }. Output: scored criteria + verdict.
2. Add traceparent to the shared task complete-output base schema. The runtime already returns it on claim; tasks echo it on completion. Trajectory queryable from Axiom by trace id; nothing eval-specific shipped through the API.
3. One correlation_id per session, grouping all 4 tasks (with-run, without-run, with-judge, without-judge).
4. Atomicity ("both variants graded or neither counts") enforced at query time, not via a new workflow:

   GROUP BY rubric_id, pack_cid, model
   HAVING count(distinct variant where status='completed') = 2
      AND every run-task has executorTrustLevel >= sandbox_attested
   

5. CLI:
   • moltnet eval push — creates/updates rubric via feat(rubrics): first-class rubrics resource with CID lookup (Phase 2) #881 SDK (no eval anchor table needed; the rubric_id chain in rubrics is the stable eval anchor).
   • moltnet eval run — creates 4 tasks under one correlation_id, awaits completion via existing task wait semantics.

What's gone

• ❌ evals table (anchor) — rubric_id already serves this role.
• ❌ eval_tasks table — input_cid + rubric_cid already content-address both halves.
• ❌ eval_sessions table — correlation_id already groups tasks.
• ❌ eval_scores table — task attempts + complete output already store this.
• ❌ Bespoke proof signature flow — claim/complete attestations already cover it.
• ❌ Phase 4 server-side OTel forwarding — traceparent carries the link; no need to re-ingest spans through the API.

Proctored guarantee

"Score counts toward published aggregates iff both run tasks have executorTrustLevel >= sandbox_attested." Enforced at query time. No new endpoint, no new entity. Sandbox-attested trust ties directly into Gondolin (#826).

Open question

Should judge-eval-variant ship before #881 with inline rubric bodies (mirrors #881 Phase 1 pattern in judge_pack), then migrate to rubric_cid? Or block this entirely on #881?

Default: block on #881 — avoids the migration churn.

Related

• research: proctored eval protocol — server-attested scores for DSPy-based eval engine #523 (closed; superseded)
• feat(rubrics): first-class rubrics resource with CID lookup (Phase 2) #881 (rubrics resource — hard dependency)
• Research: Agent runtime — programmatic task execution for MoltNet agents #852 (task runtime — substrate)
• Eval-runner sandbox: Gondolin prototype findings + next steps #826 (Gondolin sandbox — sandbox_attested trust source)
• Pi-native attribution loop: three-session pipeline (curate → render → judge) #875 (three-session pipeline — sibling task-types pattern)

[legreffier]

Eval task input/output schemas — open question surfaced from #518 discussion

Following the design discussion on #518 around treating rendered packs as installable skills (AgentSkills format). One thing that fell out: this issue sets up the proctored task types (run-eval, judge-eval-variant) but the input/output schemas for those task types are still TBD. Filing this as a placeholder so it's not lost.

What's missing

run-eval input as currently specified:

{ scenarioCid, variant: 'with' | 'without', packCid?, model, mode: 'vitro' | 'vivo' }

This works for "scenario-vs-pack" comparisons but doesn't yet say:

1. What's a scenarioCid? Concretely, what content does it address? AgentSkills' evaluating-skills doc defines a scenario as { id, prompt, expected_output, files?, assertions? } stored in evals/evals.json. Adopting that shape directly would give us a standard format and a documented methodology. Question: is scenarioCid the CID of one entry in that array, or of the whole evals.json?

2. Where do scenario input files live? AgentSkills stores them at evals/files/* inside the skill directory. If we ship skill bundles per Context pack injection: server-side bindings, conditional resolution, and mid-session probing #518, eval input files ship inside the bundle. The runner needs to fetch the bundle (or a subset) before executing.

3. run-eval output: transcript hash + token totals — what's the canonical transcript format? The eval methodology benchmarks need to compare with-skill vs without-skill outputs. The runner needs to capture not just tokens but the actual produced files / artifacts. Lean: store outputs to a content-addressed location, return the root CID + token totals + duration.

4. judge-eval-variant input: rubric_cid is clear, but what does the judge actually receive? The runner output (transcript hash) + the source artifacts (output CID) + the rubric. Schema should be explicit so judges are reproducible.

Proposed shape (for discussion, not for implementation yet)

run-eval input:

{
  scenarioCid: string,         // CID of an evals.json entry
  variant: 'with' | 'without' | 'old-skill',
  skillBundleCid?: string,     // present iff variant === 'with' or 'old-skill'
  model: string,
  mode: 'vitro' | 'vivo',
  inputFilesCid?: string       // CID of the evals/files/ tree for this scenario
}

run-eval output:

{
  outputsCid: string,          // CID of produced artifacts (matches AgentSkills outputs/ directory)
  transcriptCid: string,       // CID of the full agent execution log
  totalTokens: number,
  durationMs: number,
  traceparent: string          // already in base schema
}

judge-eval-variant input:

{
  runTaskId: string,
  rubricCid: string,
  // resolved server-side from runTaskId: outputsCid, transcriptCid
}

judge-eval-variant output:

{
  assertionResults: Array<{
    text: string,
    passed: boolean,
    evidence: string
  }>,
  passRate: number,
  holisticNotes?: string,      // optional blind-comparison rubric output
  traceparent: string
}

This maps directly to AgentSkills' grading.json format, which means:

• Skill authors who already use AgentSkills evals can reuse their evals.json as scenarios.
• Our judge-eval-variant output is portable to anything else that consumes the AgentSkills grading format.
• The benchmark aggregation step (benchmark.json in their methodology) becomes the "did rendered pack X pass quality bar Y?" surface.

Why this matters for #518

#518 plans to drop the moltnet skill review / moltnet skill optimize CLI commands and delegate eval work to this task runtime. That requires the task input/output schemas to be defined first — otherwise the CLI has nothing concrete to dispatch. So this issue's TBD becomes a soft blocker on #518's "skip review/optimize commands" decision.

Not asking for resolution now

Just noting the gap. Probably needs a separate design pass once #881 lands and the rubrics CID story is concrete. Filing here so the AgentSkills alignment isn't forgotten when this gets picked up.

[legreffier]

Reinforcing the binary-with-evidence direction + three-lane refinement

Picking up on the schemas proposed in the previous comment, plus context from #999 (rendered-pack fidelity judge redesign). Two methodology points worth folding in before the schemas freeze.

1. The output shape is right; make the rationale explicit in the schema

The proposed judge-eval-variant output —

assertionResults: Array<{ text: string, passed: boolean, evidence: string }>

— is exactly what agentskills.io's grading guide prescribes, and it matches the binary-rubric methodology popularized by Hamel Husain. The discipline that makes this format work, encoded as schema-level requirements:

• evidence must be required, not optional — for both PASS and FAIL. agentskills.io: "Don't give the benefit of the doubt. If an assertion says 'includes a summary' and the output has a section titled 'Summary' with one vague sentence, that's a FAIL — the label is there but the substance isn't." If we make evidence optional, judges will skip it on PASS and we lose the cluster-analysis dataset.
• No composite field, no Likert axes. A numeric rubric averages "13 grounded claims + 1 fabricated claim" into 0.93 and ships hallucinations. Pass-rate from binary assertions is the only aggregate that survives.
• passRate should be derived, not authored. Computed as passed_count / total_count. Keeps the judge honest.

2. Three lanes, not one rubric — and the lanes correspond to different judges

agentskills.io makes a distinction the original Hamel post understates: not everything is binary, but nothing meaningful is a Likert.

Lane	What it scores	Mode	Where it lives
1. Binary assertions with evidence	Verifiable properties: grounding, faithfulness, signature integrity, file-exists checks	assertionResults	judge-eval-variant, plus judge_pack for fidelity (#999)
2. Graded with threshold	Coverage-style fractions where summarization is legitimate	coverage + threshold → derived passed: bool	Same task; one assertion per graded axis
3. Holistic quality	Style, organization, "feels right"	NOT scored — blind LLM comparison or human review	Separate task type (compare-pack-quality?)

agentskills.io on Lane 3: "writing style, visual design, whether the output 'feels right' — are hard to decompose into pass/fail checks. These are better caught during human review."

The holisticNotes?: string field in the proposed output is a soft trap. It invites the judge to smuggle Lane 3 quality assessments into a Lane 1 task, which dilutes both signals. Recommend dropping it and tracking holistic comparison as a separate task type.

3. Crucial split: usefulness (this issue) vs fidelity (#999)

The methodology guide is explicit that the with-skill / without-skill baseline is structural. That's exactly what run-eval + judge-eval-variant provides — and it's structurally a usefulness measurement: did the pack help the agent do its task better than no pack?

But fidelity is a different question — does the pack faithfully represent its source entries? — with no meaningful "without" baseline. #999 redesigns the fidelity judge (judge_pack) using the same three-lane principle.

• judge-eval-variant (this issue) — judges with-pack vs without-pack execution outputs. Lanes 1 + 2. Usefulness.
• judge_pack (Fidelity judge rubric: numeric gradients smooth away the failures we most need to see #999) — judges rendered pack vs source entries. Lanes 1 + 2. Fidelity.
• Both share the assertion-result shape. Define AssertionResult once in libs/tasks/src/rubric.ts and reuse.

4. Variance check before locking the schema

Pre-flight, mirroring #999: pick 3 fixed scenarios, run judge-eval-variant 5× each on the same runTaskId, measure assertion-flip rate (same assertion flips PASS↔FAIL across runs). High flip rate → assertions are vague, fix them before declaring the schema stable.

Schema delta suggestion

// shared in libs/tasks/src/rubric.ts
type AssertionResult = {
  id: string;
  text: string;
  passed: boolean;
  evidence: string;        // REQUIRED for PASS and FAIL alike
};

// judge-eval-variant output (usefulness)
{
  assertionResults: AssertionResult[];
  passRate: number;        // derived
  traceparent: string;
}

// judge_pack output (#999, fidelity)
{
  groundingAssertions: AssertionResult[];
  faithfulnessAssertions: AssertionResult[];
  coverage: { value: number; threshold: number; passed: boolean };
  pass: boolean;
  traceparent: string;
}

TL;DR

• Direction is right; tighten by requiring evidence and removing holisticNotes.
• Define AssertionResult once, reuse across judge-eval-variant and judge_pack (Fidelity judge rubric: numeric gradients smooth away the failures we most need to see #999).
• Lane 3 is a separate task type, not a field.
• Variance pre-flight on a few scenarios before declaring schema stable.

Cross-linked: #999, agentskills.io eval guide, Hamel on LLM judges.