Grilling: how does the LLM-as-judge scoring mechanism get built?

[antejavor]

Part of #297

Question

How does the LLM-as-judge scoring mechanism (coverage + efficiency rubric, per #299) actually get built? deepeval (this repo's established eval framework — see the corpus-format ticket) offers two paths: its built-in GEval metric (custom natural-language criteria, no code) or a hand-written BaseMetric subclass, the same pattern evals/coherence.py already uses. Resolve which path, what the criteria/prompt actually says for "coverage" vs. "efficiency," which judge model, and whether any consistency/calibration check (e.g. repeat-and-compare) is needed before trusting a single judge run.

Depends on the corpus format ticket (need the LLMTestCase-or-otherwise shape settled first).

[antejavor]

Resolution

1. Split the rubric by mechanism — LLM judges quality, code measures cost

#299 decided a graded coverage+efficiency rubric "via LLM-as-judge." Refined here: only the quality axes go to the judge. Efficiency as #299 defined it — hops, tool calls, tokens — is directly countable, so it's a deterministic metric with no LLM involved. Asking a judge to grade a number you can count adds cost and run-to-run variance to the headline score for zero information gain, and makes efficiency regressions arguable rather than factual.

2. LLM-judged side: a minimal pair, not built-ins-only and not one big custom metric

The ticket framed this as GEval vs. a hand-written BaseMetric subclass. Both were the wrong frame — deepeval ships built-ins that already cover part of the rubric, verified against the installed package:

• ContextualRecallMetric for retrieval-side coverage. Its _required_params are exactly INPUT + RETRIEVAL_CONTEXT + EXPECTED_OUTPUT — precisely the Golden fields locked in #302. Zero prompt-authoring, maintained upstream.
• One GEval with a coverage rubric (GEval takes a real rubric param: Rubric(score_range, expected_outcome), 0–10) for the answer itself, because no built-in asks "does actual_output contain every fact in expected_output" — which is the actual question when you hold an answer key.

Deliberately skipped at v1: FaithfulnessMetric and AnswerRelevancyMetric. Both exist primarily for the no-ground-truth case, and each extra LLM metric is another judge call per question — at ~1000 questions, four metrics is 4000+ calls per run, multiplied again by #299's per-schema-candidate re-runs. Two metrics keeps a full run cheap enough to actually get run.

A hand-written BaseMetric subclass (the evals/coherence.py pattern) stays reserved for genuinely non-LLM metrics — which is exactly what the deterministic efficiency metric will be.

Free side benefit: ContextualRecall scores retrieval while the GEval scores the answer, so rough stage attribution falls out for nothing — even though #299 deliberately deferred building attribution machinery.

3. Judge model: Anthropic, pinned to an explicit dated model ID

• Judge independence is the deciding reason. Extraction (reconciliation via LightRAG, rag.llm_model_func) is OpenAI-backed. An OpenAI judge shares the extractor's blind spots — an entity the extractor systematically drops is plausibly one the judge also treats as unimportant, so the score flatters the pipeline exactly where it's weakest. A different provider decorrelates those failure modes.
• ANTHROPIC_API_KEY is already a repo secret from #289 (map Map: Live Claude Code verification for the SPAWNED subagent-nesting inference rule #288's work), so CI needs no new provisioning.
• deepeval's AnthropicModel accepts _anthropic_api_key explicitly, so it composes with the config-file-only key resolution from ADR 0002 / PR context-graph: propagate resolved Memgraph/LLM config to the reconcile subprocess #253 rather than needing ad-hoc env vars. It already defaults temperature=0.
• Pin the version. deepeval's default is the moving alias claude-3-7-sonnet-latest; a moving judge silently breaks cross-version score comparability, which is the entire point of Grilling: what should the eval loop score, and how? #299's human-gated comparison — the same reasoning that put the corpus in git rather than a mutable store. Pin a dated ID, bump deliberately, and treat a judge bump as invalidating prior baselines.

4. Calibration: two small checks, run occasionally — not per-run

Noise and bias are different failures and need different instruments:

• Repeat-and-compare, for noise: re-run a fixed ~50-question subset 3× and check the score spread. temperature=0 is not truly deterministic on hosted APIs, so single-run stability is assumed rather than established. This is what bounds "is a 2-point delta real?" on every cross-version comparison.
• ~20–30 human-graded items, as a bias anchor: a judge can be perfectly stable and consistently wrong; a variance check cannot detect that by construction. Only human labels can.

Both run on judge-model bumps and rubric changes, not on every eval run.

Not a reversal of #299: that ticket rejected human grading as the scoring mechanism for the corpus (doesn't scale across repeated runs). ~25 items calibrating the instrument once is a different thing from humans scoring 1000 questions every run.

Spun out

The deterministic efficiency metric needs its counts to come from somewhere, and the obvious source — the eval agent's own harness session — is itself captured by context-graph's hooks, writing into the same graph being measured. Ticketed separately as Grilling: how is the deterministic efficiency metric instrumented, and how do eval runs avoid polluting the graph under test?.