LoCoMo protocol fixes + Supermemory provider on v5

[sohamd22]

LoCoMo numbers out of memorybench weren't comparable to anything published, and a lot of our "failures" were the harness. This fixes the LoCoMo side and moves the Supermemory provider to the v5 API.

LoCoMo

• Category ids match the dataset: 1 multi-hop, 2 temporal, 3 open-domain, 4 single-hop, 5 adversarial (they were shifted by one).
• Adversarial gold is "Not mentioned in the conversation" (the judge was getting undefined).
• Photo captions go into the message text, like LoCoMo's own eval (some answers only live in photos).
• One container per conversation, ingested once and shared by its ~200 questions: 10 containers / 272 docs instead of 1,986 / 55,014. Every question's evidence is inside its own conversation (checked all 2,824 evidence refs).
• Answer + judge prompts are Mem0's paper prompts, verbatim with commit links (same ones Zep, Memobase, MemOS, LangMem use). Every provider uses them on LoCoMo and only formats its own search results.
• Headline score leaves out adversarial (1,540 Qs), adversarial is still scored per type.

Supermemory provider (v5)

• One doc per session via /ns/{ns}/document: "Name: message" transcript, real session date in date (no more user/assistant roles or upload-day dates).
• Waits for memory extraction (v3 dreamingStatus) before searching. v5 says done once chunks are indexed, before memories exist.
• Retries 429/5xx, honoring retry-after.
• Search settings (limit, threshold, searchMode, rerank, rewriteQuery, include, dreaming, supportingContext) come from --provider-config '<json or path>' and get saved with the run. Nothing set = API defaults.

Harness

• --provider-config flag.
• Benchmarks can optionally set a container key, their own answer/judge prompts, and types left out of the headline.

Results (cats 1–4)

Run	Score
Oct 3 run on main (gpt-5, old judge, all 1,540 Qs)	52.8%
This branch, same Oct 3 data, v5 search limit 30, gpt-4o-mini + Mem0 prompts	68.8%
This branch, re-ingested, conv-26 + conv-49 (308 Qs)	69.5% → 76.6% (40 fixed / 18 broke, p ≈ 0.005)

For reference Mem0's paper has Mem0 at 66.9% and Zep at 66.0% under the same protocol.

Run it

bun run src/index.ts run -p supermemory -b locomo -m gpt-4o-mini -j gpt-4o-mini \
  --provider-config '{"limit":30}' -r locomo-b-limit30-<date>

Needs SUPERMEMORY_API_KEY and OPENAI_API_KEY. Old LoCoMo checkpoints use per-question containers, so start a fresh run ID.

🤖 Generated with Claude Code