Map: Hybrid retrieval — the typed graph as an index into turns

[antejavor]

Destination

A retrieval design for the context graph that uses the typed graph as an index into turn text. It's implemented as the sessions-graph read path. Target: 80–90% correct end to end on a trustworthy judge, with an understanding of what each stage contributes and why it works.

The eval side exists as a starting point in #389: --retrieval-strategy hybrid, with lanes for turn vectors, turn full text, entities and their edges, facts by source sentence, and the user's facts across sessions. This map decides what of it becomes the product, how, and on what evidence.

Why this map exists

Map #344 made the graph hold typed, time-anchored facts (merged in #373). Reading it back has so far shown:

• Typed edges are a poor answer store. On 100 LongMemEval questions with 5 sessions each, answering from every typed edge of the right sessions scored 21/100; answering from their full text scored 48/100. An LLM agent writing Cypher over the typed schema scored 12–15, below plain text search (33–34).
• Typed edges are a usable index. Hybrid retrieval, with the graph finding facts and their source turns, scored 40–44 against 37–38 for turns alone, across three runs each. It helped most on knowledge-update questions, which reached 14/16 on a cleaner graph.
• Those scores predate the scoring fix. Judge results had been attributed to the wrong questions (context-graph-eval: judge scores were attributed to the wrong questions #387, fixed in context-graph-eval: match judge scores to their question, and run judges at minimum effort #388). Run totals held, but every per-question result was wrong. On the corrected scoring the full hybrid scores 45/100 (Does the graph lift hybrid retrieval on corrected scoring? #391). What each lane contributes is not yet measured.

Notes

• Domain: context-graph/eval for measurement; unstructured2graph and sessions-graph for what ships.
• Skills: /grilling, /domain-modeling and /research, as on the other maps.
• Decision criterion: end-to-end coverage of full hybrid against the current baseline (45/100, Does the graph lift hybrid retrieval on corrected scoring? #391), same graph and judge; target 80–90%. Driven by failure analysis: which questions are missed and why (retrieval, graph model, or answering). Ablations explain contributions; they don't gate. Text search and the Cypher graph agent are not re-run, since they're not the frontier.
• Cost discipline: judge calls dominate eval cost. Judges run at minimum effort with fixed rubric steps (context-graph-eval: match judge scores to their question, and run judges at minimum effort #388). Reuse extracted graphs with --skip-reconcile; re-extract only when extraction changes.
• Measurement graphs: round 2's graph (context-graph/unstructured2graph: typed relation model on a shared hygm ontology (map #344) #373's model plus the pronoun and generic-noun fixes; 100 questions, 5 sessions each) is the baseline graph until extraction changes.

Decisions so far

• Baseline and criterion (Does the graph lift hybrid retrieval on corrected scoring? #391): full hybrid on the round-2 graph scores 45/100 on corrected scoring (knowledge-update 14/16, single-session-user 11/15, single-session-assistant 9/11, temporal 7/26, multi-session 4/26, preference 0/6; abstention 7/8). The turns-only ablation no longer gates; the next step is failure analysis of the 55 misses.
• Failure analysis of the 55 misses (Does the graph lift hybrid retrieval on corrected scoring? #391 comment / this map): partial or missed retrieval 25, answering 9, no question date 8, scoring gate 7, preference 6. The eval pooled all 100 personas under one user → Eval: one user per question, retrieval scoped to it #396; the score gate → Eval: Contextual Recall fails correct computed answers #397; the answer prompt → Hybrid answer prompt: turn dates, ordering, and recommendations #398; extraction quality → Extraction quality: advice vs events, event dates, relation types #399.
• Turn provenance written at extraction (Write turn links and edge source sentences at extraction time #392): Segment.source_id; a sources list on MENTIONED_IN; one edge per turn with source_id, text and role; link_turns deleted; the eval graph rebuilt.
• One user per question (Eval: one user per question, retrieval scoped to it #396): the eval had pooled 100 personas under one user; each question now gets its own user and retrieval is scoped to it. Rebuilt graph: 48/100.
• Scoring gate (Eval: Contextual Recall fails correct computed answers #397, Eval: score retrieval against the evidence turns, not the expected answer #403): the answer rubric decides coverage; Contextual Recall and evidence recall (share of evidence turns retrieved) are reported beside it, not gated. 48 → 60.
• Answer prompt (Hybrid answer prompt: turn dates, ordering, and recommendations #398): question date always on, compute over listed evidence, turns first, recommendation questions never refused. 60 → 76; temporal 22/26.
• Preference rubric (Eval: a preference rubric for recommendation questions #404): recommendation questions judged on a preference rubric. 80/100; preference 6/6, abstention 8/8.
• LongMemEval's official judge is the headline (Eval: score every run with LongMemEval's official judge as the headline number #409): verbatim prompts, gpt-4o; our rubrics stay as diagnostics. 84/100.
• Extraction quality (Extraction quality: advice vs events, event dates, relation types #399): graph correctness as a goal of its own, not the benchmark. Speaker policy, owner identity, named-only merging, resolved dates → Eval: graph-quality measure — sampled edge labels and deterministic counts #411 measure, hygm: date datatype, owner identity scope, named-only merging #412 hygm, unstructured2graph/sessions-graph: resolve dates, enforce identity rules, speaker policy #413 extraction, Eval: rebuild the graph with #399's extraction and re-measure #414 rebuild.
• Embeddings (Where embeddings live, and what gets embedded #393): vectors live in Memgraph, computed by MAGE embeddings.text; bge-small-en-v1.5 by default; messages, entity names and edge text; messages embedded at session end; exact cosine over the user's own rows.
• Read path (The retrieval read path: API and what the answerer receives #394): recall(question) returns context, the harness model answers; served over MCP by agent-context-graph mcp (tools registered through the agent_context_graph.tools entry point), plus agent-context-graph recall [--json]; own memory only, user from the config file; widths from [recall]; a SessionStart line points the model at it. MCP results are text only: Claude Code and Codex both show the model structuredContent instead of the text when both are sent.
• Read path shipped (actions-graph: write plain text on message actions #415 → context-graph: message actions carry their plain text #421, sessions-graph: embed messages, entities and edges in Memgraph #416 → context-graph: embed messages, entities and edges inside Memgraph for recall #422, sessions-graph: recall() core, shared by the eval #417 → context-graph: sessions-graph recall(), the read path the eval benchmarks #424, agent-context-graph: MCP server with the recall tool #418 → context-graph: serve recall to Claude Code and Codex over MCP #426, actions-graph: record Claude Code assistant replies so recall can return them #425 → agent-context-graph: record each turn's assistant reply so recall can return it #427): message text on actions, in-Memgraph embeddings, the recall() core shared by the eval, the MCP server in both plugins, and assistant replies recorded at turn end. Parity run on the shipped code: 89/100 on the official judge, evidence recall 79/92; recall takes 0.6 s on eval users, 0.7 s at 10k messages, 6.7 s at 100k. The 80–90% target is met.

Not yet specified

Not pursued, since the target is met; reopen only if a new measurement calls for it:

• Whether retrieval should iterate (answer, find what's missing, retrieve again) or stay single-pass.
• How the lanes weight and budget against each other.
• Answering over aggregates: largely closed by the answer prompt (multi-session 4/26 → 13–15/26, temporal 7/26 → 22/26); the remaining misses split between partial retrieval and answer noise.

Out of scope

• Extraction vocabulary and its derivation (Observe value relations with their intended tail during derivation #386, the evolution loop). This map measures on context-graph/unstructured2graph: typed relation model on a shared hygm ontology (map #344) #373's hand vocabulary.
• Eval-harness correctness beyond what the ablation needs: map Map: Context-graph emergence pipeline — reconciliation, retrieval, eval #297.
• LightRAG: retained, not invested in.

[ViridianAIQualityAssurance]

One thing I'd add to the ablation criterion: I'd treat the corrected reruns after #387/#388 as a new evidence generation rather than as continuity with the old per-question results.

For the hybrid vs hybrid-without-graph comparison, it may be worth freezing and recording the code revision, graph/extraction identity, retrieval configuration, judge identity/config, and exact question-set identity for each run. I'd also retain the per-question artifacts and missing/failed cases rather than only the aggregate score.

Then the promotion criterion could be declared before rerunning: repeated hybrid lift outside the observed run-to-run noise, with the same comparison identities across arms.

Given #387, I'd also make result ↔ question association itself a regression invariant in the harness. Aggregate totals surviving while per-question attribution was wrong is exactly the sort of failure that can make an otherwise sensible ablation look trustworthy.

That would make the eventual “graph lane earned its place” decision reconstructable from the evidence, rather than depending on the aggregate result alone.