Grilling: what should the eval loop score, and how?

[antejavor]

Part of #297

Question

What should the eval loop (component 5) score, and how? Covers: what the score measures, corpus source and shape, task-generation method, scoring mechanism, and how a score feeds back into schema/design decisions.

Resolved via /grilling with the driving user before this ticket was filed (same session that charted the map). Answer recorded below.

[antejavor]

Resolution

• What eval scores: blended end-to-end score — mostly (a) recall quality (agent queries the emerged graph via retrieval, answer scored against a known ground truth), with (b) structural graph-shape checks folded in but weighted lighter. No attribution split between "bad graph" vs. "bad retrieval" yet — one blended score for now; attribution machinery is future work once the blended score is non-trivially wrong often enough to be worth separating.
• Corpus source: synthetic-first. Real historical-session replay has no reliable ground truth (you know an answer was given, not that it was correct); synthetic lets the answer key be fixed up front. Real replay is a later, harder-to-score second wave, not this ticket's scope.
• Corpus shape: one unified corpus (~1000 questions, per the driving user's thesis note), not split per component. Questions are framed as organizational signals/realistic-employee-style questions against what got translated from raw data — they may span multiple components or land on one incidentally, but are not authored component-by-component. This is deliberate: a siloed corpus can't exercise the multi-use-case thesis (same bucket serving wildly different query shapes concurrently).
• Task generation: hybrid during development — bulk of the corpus skips the real harness (synthetic content injected straight into reconciliation, cheap/fast iteration), plus a small "gold slice" (~20-50) that must flow through a real live harness session (same pattern already proven for graph-model verification, map Map: Live Claude Code verification for the SPAWNED subagent-nesting inference rule #288) so capture-layer surprises aren't invisible to eval. Harden later: grow the real-harness share over time as the loop matures.
• Scoring mechanism: graded rubric — coverage (all necessary facts present, not just any) + efficiency (cost to retrieve: hops/tool-calls/tokens) — judged by an LLM-as-judge against the known answer key. Binary correct/incorrect was rejected (throws away the "how well" signal schema evolution needs); human-in-loop grading was rejected (doesn't scale across repeated runs per schema iteration).
• Feedback loop: human-gated for this effort. Eval runs produce a comparison report across candidate schema/decay-rule versions; the driving user decides whether to promote. Fully automatic schema promotion (score crosses a threshold → auto-promote) is the long-term direction from the driving user's context-graph-memory-thesis note, but not something this map builds — every other consequential state change in this codebase's history (both closed graph-model maps, the Write Gate proposal, Grilling: retention/compression policy for raw collection-tier data #267's confirmation-gated deletion) is human-gated by design, and auto-promotion needs a track record of human-gated runs first to calibrate against.