Grilling: what triggers an eval run, and how often?

[antejavor]

Part of #297

Question

What triggers an eval run, and how often — per schema/decay-rule candidate on demand, a scheduled sweep, tied to a CI job, something else? Independent of the corpus/judge tickets; can resolve in parallel.

[antejavor]

Resolution

Two tiers, split on cost and determinism. Implemented in PR #311.

Tier 1 — every PR, no LLM key

tests.yaml gains a test-context-graph-eval job, triggered by changes to context-graph/eval/** (plus its dependencies: actions-graph, sessions-graph, memgraph-toolbox).

It provides no LLM key deliberately. The LLM-backed tests skip themselves without one, so the job stays fast, free and deterministic. That still covers 74 of the package's 79 tests against a real Memgraph — converter, corpus round-tripping, sampling, scoring composition, injection e2e, and retrieval plumbing.

This is the tier that catches ordinary regressions, and it is the one that runs constantly.

Tier 2 — real eval batches, workflow_dispatch only

eval-run.yaml, following the precedent already set by test-live-graph-model.yaml (map #288). Inputs: question count, judge model, and whether to skip reconciliation. It uploads the report as an artifact rather than asserting on it.

Not on push or pull_request, for three separate reasons — any one of which would be sufficient:

1. Cost. Reconciliation is roughly two LLM calls per session, and a 20-question batch injects ~900 sessions (measured: 10 questions produced 493 distinct sessions). That is untenable per-PR.
2. Non-determinism. Judged scores vary run to run. #304 already decided this is handled by repeat-and-compare calibration, precisely because a single run's number cannot be trusted to a fixed threshold. Gating on a noisy metric produces flaky CI, which teaches people to ignore it — worse than not having it.
3. It would contradict a decision already made. #299 chose human-gated promotion: eval produces a comparison report a person reads and acts on. A pass/fail CI gate on the score is automatic promotion wearing a different hat, and would quietly undo that.

Point 3 is the deciding one. The first two are about affordability and noise, and both would eventually soften — cheaper models, a smaller sampled batch, better calibration. The third does not soften: it is a standing decision about who decides, and CI is not the answer to that question until #299's human-gated stance is deliberately revisited.

Cadence, concretely

• Per PR: the fast tier, automatically.
• Real eval batches: on demand, when someone is actually evaluating a change to reconciliation, decay, retrieval, or the graph model — the moments where a score means something. Not on a timer.
• Not scheduled. A nightly or weekly run was considered and rejected for now: nothing consumes the output automatically (promotion is human-gated), so a scheduled run would spend real money producing reports nobody asked for. Revisit if and when a standing baseline is genuinely wanted for tracking drift over time — that is a different need from the per-change comparison Grilling: what should the eval loop score, and how? #299 described, and should be argued on its own terms.

Implementation notes worth carrying forward

• Both jobs start Memgraph with --schema-info-enabled. Required, not cosmetic: retrieval's schema description goes through memgraph-toolbox's SearchSchemaTool — the same implementation the MCP server's search_schema tool wraps — which refuses outright without the flag. Without it the agent silently falls back to bare label names, so CI would exercise a different code path than production.
• The dispatch job runs the retrieval agent on OpenAI and the judge on Anthropic, preserving Grilling: how does the LLM-as-judge scoring mechanism get built? #304's deliberate provider split so the judge does not inherit the extraction pipeline's blind spots and flatter it exactly where it is weakest.
• --skip-reconcile exists because reconciliation dominates run cost. Iterating on retrieval or scoring against an already-reconciled graph should not pay for it again.