Grilling: what format does the eval corpus have, and where does it live?

[antejavor]

Part of #297

Question

What format does the eval corpus (component 5) actually take, and where does it live?

Prior art found: memgraph-toolbox — a dependency every context-graph package already has — ships a deepeval-based evaluations extra (pyproject.toml: deepeval>=3.5.2), with one custom metric already built (CoherenceEmbeddingsBasedMetric in evals/coherence.py, extending deepeval.metrics.BaseMetric). deepeval is this repo's established eval framework; no other eval framework (promptfoo/braintrust/ragas) appears anywhere in the repo. deepeval's LLMTestCase (input / actual_output / expected_output / context) is a natural fit for the corpus shape already decided in #299 (question + ground-truth answer + retrieved context).

Resolve: does the corpus get stored as a deepeval-shaped dataset (and if so, where — a file in the family, a new memgraph-toolbox submodule, something else), or is there a reason to deviate from LLMTestCase's shape? This decision blocks the authorship-mechanics and judge-harness tickets, which need the shape settled first.

[antejavor]

Resolution

Format: deepeval's Golden, serialized as JSONL, committed to git. No custom corpus schema — Golden already carries everything #299's design needs, verified directly against the installed deepeval (.venv/.../deepeval/dataset/golden.py):

Golden field	Eval use
input	the question
expected_output	ground-truth answer (the answer key)
context	the facts that should be recallable
retrieval_context	what retrieval actually returned, per run
additional_metadata (free dict)	org-signal category, slice tag (bulk vs. gold), component coverage
name / comments / source_file	provenance back to the upstream benchmark item

Golden is the corpus-time primitive (question + answer key, no output yet); LLMTestCase is the run-time object (Golden + actual_output). deepeval also ships native file I/O for exactly this shape — EvaluationDataset.save_as(json|csv|jsonl, directory) and add_goldens_from_json_file/add_goldens_from_csv_file — so persistence needs no new code.

Storage medium: JSONL files in git. Rejected alternatives, with reasons:

• Memgraph nodes — rejected on "keep the ruler outside the thing being measured." The corpus is the answer key, and the schema-evolution loop's whole purpose is mutating the graph; an answer key living in that same graph can be silently invalidated or corrupted by a migration, leaving no way to distinguish a real score regression from a corrupted fixture. Git also gives free proof the corpus didn't change between two schema versions' runs — which Grilling: what should the eval loop score, and how? #299's human-gated cross-version comparison depends on.
• Confident AI cloud (deepeval's built-in push/pull) — rejected on this project's own thesis grounds: the accumulated graph is the org's durable IP, unlike rented intelligence. Putting the core answer key in a third-party SaaS contradicts that directly.

Upstream benchmark content: fetch-and-convert, don't vendor. Corpus content is intended to derive from an existing third-party memory benchmark that ships session files, rather than being authored from scratch. Commit only our converted output (the Golden JSONL, plus injection-ready session fixtures), alongside a fetch-and-convert script pinned to a specific upstream release/commit. Rationale:

• Git was chosen for a diffable, reviewable answer key — true of converted Golden JSONL, not of hundreds of MB of upstream blobs.
• Third-party benchmarks carry their own licenses; not vendoring sidesteps that question entirely.
• The pin keeps it reproducible, and because converted output is committed, upstream disappearing degrades to "can't re-derive" rather than "corpus lost."

Accepted cost: one conversion adapter to maintain per upstream benchmark format.

Spun out: which benchmark(s) to actually derive from — and the real risk that the well-known ones are conversational (user/assistant turns) and would exercise sessions-graph's text path while barely touching actions-graph/skills-graph — is now Research: which memory benchmark should the eval corpus derive from, and what does each actually exercise?, which blocks the corpus-authorship ticket.