Does the graph lane help on coding sessions, not just LongMemEval?

[antejavor]

Part of #390

Question

Every measurement so far is on LongMemEval: personal-assistant chat, one user, facts about the user. The context graph's own target is coding-agent sessions. Is there a corpus, even a small hand-labelled one from real Claude Code sessions, on which the typed graph's lanes can be measured? Does the hand vocabulary even fit it?

Output

A corpus proposal with a handful of questions and gold answers, or a finding that the graph lanes can't be judged on coding sessions yet.

[antejavor]

Unlinked from map #390 (closed). The one open question it leaves: every number so far (89/100) is LongMemEval chat; this checks whether it holds on coding sessions.

[antejavor]

BEAM as a second data point (PR #438)

BEAM's 100K set has 20 long chats across coding, math, writing, advice, legal and lifestyle, probed on ten memory abilities. It ran through the shipped read path (injection, GLiNER2 reconciliation, recall, the LongMemEval answering prompt) and was judged by BEAM's own rubric judge.

100 balanced questions (10 per ability, 5 per chat):

ability	hybrid with passages	turns-only
abstention	1.00	0.70
contradiction_resolution	0.63	0.68
event_ordering	0.18	0.17
information_extraction	0.90	0.93
instruction_following	0.73	0.90
knowledge_update	0.70	0.50
multi_session_reasoning	0.70	0.63
preference_following	0.93	0.80
summarization	0.65	0.56
temporal_reasoning	0.68	0.35
BEAM score (mean of abilities)	0.708	0.622

Before the passage fix, hybrid scored 0.693.

Findings

1. The hand vocabulary doesn't fit. With the LongMemEval vocabulary, GLiNER2 typed about 21.5k entities as the catch-alls Topic/Quantity. The relations were personal-life ones (works_for, lives_in, purchased), with no types for libraries, features, bugs or requirements. That answers this issue's "does the hand vocabulary even fit?": no. It led to map Map: An ontology that learns from the user's sessions #431, an ontology that learns from the user's sessions.
2. The graph still helps. With the graph lanes on, hybrid won 14 questions outright against 4 for turns-only; the biggest gaps were temporal and knowledge update. Some of that may be extra context: hybrid's payload is about 3× larger (median 7,338 vs 2,586 tokens). A payload-matched control is still pending.
3. Long turns were cut twice: bge-small embeds only the first 512 tokens, and recall showed only 1,500 characters. Fixed in context-graph: BEAM eval and passage-level recall #438 with passages. The fix is mechanically sound, but it moved BEAM only within noise. LongMemEval parity held (88 vs 89).
4. Remaining failure modes:
   • Event ordering questions span the whole conversation, which top-k recall misses.
   • Same-day updates. Every turn in a batch shares one date, so "most recent wins" can't order updates made on the same day.
   • The answering prompt's "Working:" lines are scored as extra events on event ordering.
5. BEAM data and judge caveats:
   • One question in our 100 (100K-12-knowledge_update-0) has no update evidence in the chat; it exists only in BEAM's generation plan.
   • BEAM's judge is noisy both ways. It once scored near-identical answers 1.0 and 0.0, so read per-question verdicts before trusting a small delta.

Still open on this issue: a corpus from real Claude Code sessions. BEAM's coding chats are planning and assistance conversations, not agent sessions with tool calls.