Grilling: how is the deterministic efficiency metric instrumented, and how do eval runs avoid polluting the graph under test?

[antejavor]

Part of #297

Question

Two coupled questions, surfaced resolving Grilling: how does the LLM-as-judge scoring mechanism get built?:

1. Where do the efficiency counts come from? #304 decided efficiency (hops / tool calls / tokens) is measured deterministically in code, not LLM-judged — a hand-written non-LLM BaseMetric, the memgraph-toolbox evals/coherence.py pattern. But the counts have to be sourced. Candidates:

• deepeval's own LLMTestCase.tools_called / TOOLS_CALLED param, populated by whatever runs the retrieval agent;
• the agent SDK's own token/usage accounting;
• context-graph's own captured data — the eval agent's retrieval session is itself a harness session, so actions-graph already records its tool calls and agent-context-graph already emits the events. Self-instrumenting, and arguably the most honest measure of "what did answering this actually cost."

2. How does an eval run avoid polluting the graph it measures? The third option above exposes the real problem, and it applies regardless of which option is chosen: if the eval agent runs as a real harness session with hooks active, its own activity is captured into the same graph the eval is scoring. Successive eval runs would then accumulate their own sessions as evaluable content, and a corpus question could be answered from a previous eval run's traces rather than from the planted fixture data. Candidates to grill through: hooks disabled for eval runs (loses the free instrumentation from option 3); a separate Memgraph database/instance per eval run; a marker on eval-origin nodes plus filtering at retrieval time; accepting the pollution as immaterial (needs an argument, not an assumption).

Note the tension with #302's reasoning: the corpus went into git specifically to keep the ruler outside the thing being measured. The same principle plausibly applies to the eval run's own traces.

Related and worth reading first: #307 (gold-slice real-harness scripting) hits the same "eval runs inside a real harness session" territory from the fixture-planting side.

[antejavor]

Resolution

1. Efficiency is retrieval payload size, not agent-side cost

The ticket framed efficiency as "hops / tool calls / tokens" inherited from #299, and this session initially read "tokens" as the eval agent's own LLM burn. That framing was wrong. Corrected by the driving user: efficiency is how many tokens were handed back to answer the question. For the same answer quality, returning less is better than returning more.

Consequences, all simplifying:

• Efficiency = token count of retrieval_context — the exact Golden field already locked in #302. A pure deterministic count, no LLM, matching Grilling: how does the LLM-as-judge scoring mechanism get built? #304's split (code counts cost, judge grades quality).
• It is simultaneously a retrieval metric and an eval metric. deepeval's ContextualPrecisionMetric is the LLM-judged cousin of the same concern (signal-to-noise in what came back) if a graded version is ever wanted.
• No adapter work, and no dependency on hook payloads. This matters, because the investigation that prompted the ticket found the opposite would have been blocked: SessionEndEvent carries total_cost_usd/total_input_tokens/total_output_tokens, but claude_code.py:192, codex.py:142, and claude.py:287 all construct it without them — only openai.py (OpenAI Agents SDK) and the separate actions_graph/hooks.py ActionTracker path populate token totals. Had efficiency meant agent-side burn, it would have needed new adapter work on a field Claude Code's Stop payload may not even carry — the aspirational-field pattern #275 exists to prevent.
• Hop / tool-call counts remain available for free from Action nodes on every runtime, as a secondary signal only. Not the headline.

Implementation requirement: token counting needs a pinned tokenizer, for the same comparability reason #304 pinned the judge model — a tokenizer change silently shifts every efficiency number. Default assumption unless revisited: the tokenizer of the model being fed the context.

2. Coverage is a hard gate; efficiency ranks within it

Efficiency alone is trivially gameable — return nothing, score perfectly — so it is never a standalone number. A question either clears the coverage threshold or it doesn't; efficiency is compared only among those that did.

A single weighted composite was rejected: it lets a retrieval change trade real coverage for token savings and still show a flat or improved headline, which is exactly the regression the eval exists to catch. Gate-then-rank makes that trade impossible to hide, and deepeval's per-metric threshold already expresses the gate natively.

3. Isolation: one database per eval batch, reset before fixture injection

The unit of isolation is the eval batch (one run over many questions), not one database per question. All of a batch's questions share one Memgraph database.

Driving reason is reproducibility, not pollution. Comparing schema version A against B is meaningless if the graph also contains whatever ambient real sessions landed that week; a known, fixed graph state per run is required regardless — the same argument that pinned the corpus in git (#302) and the judge model (#304). Isolation makes pollution a non-issue structurally, as a side effect.

No new infrastructure needed: memgraph.database / MEMGRAPH_DATABASE already exists as a config key threaded through agent-context-graph's cli.py (default "memgraph").

Batch-shared, not per-question, is also methodologically better — not merely cheaper. One tiny database per question would present retrieval with almost no distractors, scoring near-perfect precision by construction. A batch-wide graph is what makes the efficiency metric and any ContextualPrecision measurement mean anything.

Three roles, kept deliberately distinct:

Concern	Mechanism
Isolating the graph under test	The database boundary — per batch, reset before fixtures load
Distinguishing which fixture/question a node came from	Markers on nodes — provenance and bookkeeping only
Keeping eval-agent traces out of the graph under test	The database boundary (hooks off for eval agents, or pointed at a separate throwaway database if the secondary hops signal is wanted) — not a marker filter

The last row is the important one. Marker-and-filter was rejected specifically for that job: every retrieval query would have to remember the filter, and a forgotten one fails silently toward a better-looking score. Same denylist-vs-allowlist reasoning #282 settled on for collection-tier deletion. Markers are fine for bookkeeping inside a batch, where nothing depends on them for correctness.

[antejavor]

Correction, found during implementation (2026-08-25)

Part 3 of the resolution above — "one database per eval batch, using the existing memgraph.database / MEMGRAPH_DATABASE config" — is not implementable on community Memgraph. Multi-database is multi-tenancy, an Enterprise feature. Verified directly against the dev instance:

SHOW DATABASES;
-> Memgraph.ClientError.MemgraphError: Your license has an invalid type.
   To use multi-tenancy you need to have an enterprise license.

The ticket reasoned from the existence of a MEMGRAPH_DATABASE config key that database-level isolation was free existing infrastructure. The key exists, but on a community licence it cannot select a second database — so the mechanism was never available, only its configuration surface. Recording it here rather than quietly building something else.

The intent is unchanged and still correct: a known, fixed graph state per run, so cross-version score comparison means something, with eval-agent traces structurally unable to pollute the graph under test.

Replacement mechanism: a dedicated Memgraph instance reserved for eval, cleared before each batch loads its fixtures.

• Delivers the same known-fixed-state guarantee the original decision was for.
• Clearing is safe precisely because the instance is dedicated — the destructive-wipe risk that would rule this out on a shared database doesn't apply.
• Works on a community licence, so contributors and plain CI can run eval without an Enterprise secret.
• Everything else in the original resolution stands unchanged: the batch (not the question) is still the isolation unit; batch-shared remains methodologically better than per-question because distractors are what make precision and payload-size efficiency measurable; markers inside a batch remain provenance-only and never the barrier keeping eval-agent traces out.

Accepted cost: no concurrent batches against one instance. Not required by anything decided so far. If it ever is, the escalation is an ephemeral container per batch — considered and set aside here as unnecessary container lifecycle in the runner for a need that doesn't yet exist.

Requiring an Enterprise licence was also considered and rejected: it would keep the ticket's original wording at the price of making the eval loop unrunnable for anyone without one.