eval: the judge scores identical answers differently, which is what makes the passing set unstable

[antejavor]

What

The GEval judge gives different scores to the same answer text on comparable questions, and the disagreement straddles the pass threshold.

From a single run's abstention questions — same rubric, same judge, same model, one run:

score	verdict	answer
0.7	PASS	'0'
0.7	PASS	'0'
0.7	PASS	'0'
0.7	PASS	'not in memory'
0.7	PASS	'not in memory'
0.2	fail	'not in memory'
0.2	fail	'not in memory'
0.0	fail	'You lead a team of 4 engineers…' (genuine fabrication)

The exact string not in memory scored 0.2, 0.2, 0.7, 0.7. Half passed, half failed, on questions where the expected output is the same shape of "this is not in memory".

The 0.0 is correct and shows the judge is not simply broken — it distinguishes a fabricated answer from a refusal. It just doesn't score refusals consistently.

Why this matters more than it looks

This is the missing explanation for the calibration result. Three repeat runs on an identical, frozen graph gave:

3 runs: 15%, 10%, 15%   noise floor +/-5pp
stable passes: 0 of 6 questions that passed at least once

Zero questions passed in all three runs, and the first two runs' passing sets were entirely disjoint. That was previously attributed to retrieval nondeterminism. At least part of it is the judge: with the graph fixed and retrieval producing the same answer, the score still moves across the gate.

The consequence is that a pass/fail flip does not imply a behaviour change, so:

• coverage deltas below the floor are judge noise, not retrieval noise
• the efficiency median — taken over whichever questions passed — is resampled by the judge, not by retrieval
• compare() can attribute a regression to a schema change that never happened

Notes on the threshold

Scores cluster heavily: across four runs, 42/80 rubric scores are exactly 0.0 and 9 are exactly 1.0, with a thin spread between. Six land exactly on the 0.7 gate — i.e. the passes are at the threshold, not above it. A metric whose passes sit exactly on the boundary will flip on the smallest judge variation.

Directions worth considering

• Score each question more than once and take a median, at the cost of N× judge calls. Expensive but it directly attacks the variance.
• Move the gate off the cluster point — if passes land exactly on 0.7, a 0.7 threshold is the worst possible choice.
• Pin sampling if the judge exposes temperature/seed; worth checking whether deepeval's AnthropicModel passes it through, since a non-zero temperature would explain much of this.
• Report a per-question pass rate over repeats rather than a single pass/fail, making the instability visible instead of averaging it into a headline.

The first check should be the cheapest: whether the judge is being called with non-zero temperature. If so, most of this may be configuration rather than an inherent limit.

_{Found while diagnosing abstention 0/8 (map #297, PR #311).}

[antejavor]

Filed #329, which blocks part of the diagnosis here.

Diagnosing this issue largely means swapping the judge and seeing whether the instability follows the model or the rubric — if a different judge on the same corpus and frozen graph produces a stable passing set, the problem is the model; if it churns the same way, the problem is the rubric or the threshold.

That experiment is currently unexpressible: _build_model hardcodes the judge to Anthropic and the agent to OpenAI, so a judge can only be varied within one vendor.

Also worth recording here, since it removes the cheapest hypothesis: both models already run at temperature=0. deepeval's AnthropicModel and GPTModel both default to it and we do not override, so the churn is not sampling temperature. Runs 2 and 3 used --skip-reconcile against run 1's graph, so all three scored an identical graph with deterministic settings — and still produced disjoint passing sets.

One correction to the evidence in the body above, which I overstated. I wrote that the judge "scores identical answers differently", citing not in memory scoring 0.2, 0.2, 0.7, 0.7. Those were four different questions with different inputs and expected outputs, and GEval scores all three fields — so different scores there are not self-contradiction. The real evidence is the cross-run churn on identical inputs, which stands and is the more serious finding.

[antejavor]

Diagnosed for real, and the root cause is not quite what this issue's title says.

The judge itself is stable. A fixed LLMTestCase (same input/actual_output/expected_output, no retrieval involved) scored by the same judge 5x in a row: perfectly identical scores every time, for both an Anthropic judge and an OpenAI judge (#329's provider resolution is what made testing the OpenAI side possible at all — previously --judge-model could only select an Anthropic model).

The real cause: retrieval-agent non-determinism. ceb54acb's retrieved answer text was byte-identical across 6 independent runs against a frozen (--skip-reconcile) graph, and its covered status still flipped 0.0/0.0/1.0/0.917/0.971/0.0. Traced it directly by capturing the agent's actual Cypher queries across repeat calls against the same frozen graph, same question: it issues different LIMIT/ORDER BY/column-aliasing choices every time, even at temperature=0 — a documented limitation of hosted LLM APIs (server-side batching/floating-point non-determinism), not something fixable in this codebase. That changes retrieval_context's content and literal rendering run to run, which flips ContextualRecallMetric's score even when the final natural-language answer never changes.

What this rules out: none of the judge-side mitigations this issue originally proposed (swap judge provider, move the 0.7 gate, median-of-N judge calls on the same input) touch the actual cause, since it's upstream of the judge entirely.

What's actually implemented (this issue's own last suggestion, which does match the real cause): calibrate now reports each question's own pass rate across repeats, not just the aggregate stability ratio — naming the specific flaky questions (neither always nor never pass) instead of only counting how many are stable. describe_stability's aggregate "X of Y stable" can't tell a question that always fails from one that flips half the time; per_question_pass_rate/describe_per_question_rates can.

Full write-up and the fix history: #329 (PR #335, unblocked testing the OpenAI side), #340 (PR, GLiNER2 backend + the embedding-concurrency bug found along the way + this diagnosis). The per-question pass-rate addition is up for review; will close this once it's merged.

Remaining open question, probably its own issue if it's worth pursuing: does the retrieval agent's own Cypher-generation non-determinism have a cheaper fix (e.g. constraining query shape)? That's a different, narrower problem than "the judge is unstable."

[antejavor]

PR #341 merged. Closing.