Entity-aware ranking boost: cross-conversation entity confusion dominates hard retrieval misses in LoCoMo benchmark #951
Description
Context: fresh benchmark run (2026-06-10)
Same run as #950: full LoCoMo via basic-memory-benchmarks (run 3e11241b9d56, 1,986 queries), BM main @ 0.21.6 (de53e0e) vs mem0ai 2.0.5. See #950 for headline numbers and setup.
The finding
Of the 281 queries BM uniquely missed at recall@5, 146 are true retrieval misses (gold doc absent from top-10). The dominant pattern is cross-conversation entity confusion — proper nouns in the query don't pull their weight against generic semantic similarity, so documents from the wrong conversation outrank the gold doc:
- "What are Joanna's hobbies?" — gold is
locomo-c03-s01/s02; BM's top hits lead withlocomo-c07-*docs (different people entirely) - "Who is Anthony?" — gold
locomo-c04-s04; BM's top 5 are spread across c06/c01/c05, none from the right conversation - "What symbolic gifts do Deborah and Jolene have from their mothers?" — right conversation, but generic Deborah/Jolene chatter outranks the session that answers it
This concentrates in the single-hop category (entity-attribute lookups), BM's weakest: R@5 0.486 vs mem0's 0.592.
Proposal
This is where the knowledge graph should be an unfair advantage — entities are first-class in Basic Memory. Boost search candidates whose extracted entities match proper nouns / entity mentions in the query, so "Joanna" strongly prefers documents whose entity set actually contains Joanna. Possible shapes:
- a ranking boost term when query-detected entities intersect the doc/chunk's linked entities
- or an entity-filtered first pass with semantic fallback, fused with the existing hybrid score
Complementary to the rerank stage proposed in #950: reranking fixes near-misses already in the candidate set; entity boosting fixes the candidate set itself. Both measurable commit-to-commit with the benchmarks repo's worktree workflow.
🤖 Generated with Claude Code
Activity
claude commentedon Jun 11, 2026
Claude finished @groksrc's task in 2m 7s —— View job
Issue Triage
enhancementapplied; other proposed labels don't exist yet in the repo)Classification
search_service.py,search_repository.py, and thesearch_notesMCP toolNot a Duplicate
This is complementary to #950, not a duplicate:
Both are needed; neither subsumes the other.
Architecture Notes
Reading
search_service.pyandsearch_repository.py, the current search pipeline is:SearchService.search()normalizes the query into_PreparedSearchQuery(text, filters, retrieval mode)SearchRepository.search()— FTS, vector, or hybrid depending onretrieval_modeThe knowledge graph has exactly the structure needed for entity boosting: every entity has a
title(the entity name) andpermalink(slugified), and docs link to each other via relations. The bottleneck is that the query side has no awareness of which tokens are entity references — "Joanna" and "hobbies" are weighted equally.Two feasible shapes mentioned in the issue:
Option A — Ranking boost (lower risk, no schema change):
After initial retrieval, detect proper-noun-like tokens in the query (capitalized, not stop-words), look up whether each candidate doc's
titleor its linked entity names contain any of those tokens, and apply a score multiplier. This is a pure post-processing step on the existingList[SearchIndexRow].Option B — Entity-filtered first pass (higher recall improvement potential):
Extract entity candidates from the query, retrieve a set of docs whose titles/relations match those candidates, then fuse this set with the standard hybrid results before re-ranking. Needs a new repository method but no schema changes if entity lookups hit the existing
Entitytable.Option A is lower-blast-radius and measurable commit-to-commit; Option B is more powerful but needs more careful integration with the hybrid scoring.
Suggested Next Steps
FTS_RELAXED_STOPWORDS) before reaching for a full NER modelSearchService.search()as a post-retrieval boost passThis is well-scoped and measurable — the benchmark harness gives a direct signal. The knowledge graph's entity-first design makes Option A implementable without schema changes, making it a good starting point.
docs(core): record entity-boost benchmark findings; keep default off
feat(core): add config-gated entity-aware ranking boost for hybrid se…
docs(core): record entity-boost benchmark findings; keep default off
Merge upstream/main (v0.22.1) into fork
phernandez commentedon Aug 11, 2026
Worth revisiting now that the rerank stage from #950 has landed (#1143, hardened in #1232,
verification tracked in #1231).
One material change to this issue's premise: the reranker's candidate pool is up to ~100
rows deep (
max(semantic_vector_k, reranker_candidates * 4)), so a subset of the 146"gold doc absent from top-10" misses analyzed here may already be recoverable — the gold
doc only needs to reach the candidate pool, not the top-10, for the cross-encoder to lift
it. The remaining misses (gold outside the top ~100) are the true candidate-set failures
that entity boosting targets and reranking cannot fix.
Next step before sizing this: re-run LoCoMo with the reranker enabled (fastembed default)
and re-segment the single-hop misses into rerank-recovered vs still-absent-from-pool.
#1231 Phase 3 sets up the quality harness; the still-absent set is this issue's real
target population and will tell us whether the boost term or the entity-filtered first
pass is the right shape.
phernandez commentedon Aug 11, 2026
Re-test plan now that reranking is landed and instrumented (#1231, #1232, #1233)
The 2026-06-10 segmentation predates both the reranker and everything else since 0.21.6,
and the reranker's ~100-deep candidate pool changes what "true retrieval miss" means. The
146 gold-absent-from-top-10 misses split into three populations with different remedies:
reranker_candidates20→50, larger reranker model) before any new machinery.
entity-aware boost this issue proposes.
Runs (basic-memory-benchmarks, LoCoMo, current main):
(June: R@5 0.486).
reranker_candidates=50— sizes how much of (b) the cheapknob buys.
Then re-segment the remaining misses by the discriminating question: was the gold doc
anywhere in the candidate pool? Decision rule: implement the boost only if (c) is large
and still dominated by cross-conversation entity confusion; otherwise close or
deprioritize this issue. Results to be recorded here.
phernandez commentedon Aug 11, 2026
LoCoMo rerank comparison results (2026-08-11)
Two full retrieval runs (1,986 queries, uncorrected key — same as the June baseline),
current main (
38a2ed67), identical config except the reranker: semantic search enabledboth runs, run B adds
reranker_enabled=true(fastembedjinaai/jina-reranker-v1-tiny-en,default 20-candidate prefix over the ~100-candidate pool). Run artifacts:
main-rerank-off-long-r1/main-rerank-on-long-r1in basic-memory-benchmarks; envprovenance in
benchmarks/rerank-comparison.log.* adversarial excluded from headline per LoCoMo protocol; shown for completeness.
What this means for this issue
The cross-encoder is doing precisely what the entity-confusion analysis predicted a
query-document interaction model would do: "What are Joanna's hobbies?" now weighs the
proper noun against each candidate instead of relying on generic embedding similarity.
The June gap to mem0 (0.486 vs 0.592) is substantially closed at 0.537 from a lower
baseline (0.444 on current main).
R@10 moved little (+1.7pp overall). Single-hop R@10 of 0.665 means ~1/3 of gold docs
still never reach the candidate pool — that pool-absence set is the only population an
entity-aware boost can still help, and it is much smaller and better-defined than the
June miss analysis suggested.
conversation chunks and the full ~100-candidate pool gets scored; this is far above the
~90ms micro-benchmark overhead on short notes.
reranker_max_document_charsand asmaller candidate pool are the existing mitigations if this matters for a deployment.
Recommendation: downgrade this issue to a low-priority follow-up scoped to the
pool-absence residue (entity-filtered first pass fused with hybrid, per the original
proposal), to be picked up only if post-rerank quality complaints point at single-hop
lookups. The reranker has captured the majority of the practical win.
phernandez commentedon Aug 11, 2026
I think the boosting part will need to wait until we can do a better graph search in postgres 19
phernandez commentedon Sep 14, 2026
Closing as superseded/deprioritized by the measured reranker work.
The concrete ranking problem that motivated this issue was addressed by #1143 and hardened/measured through #1231, #1232, and #1233. In the full LoCoMo comparison, reranking improved overall R@5 by 5.0 points and MRR by 9.7 points; single-hop—the category called out here—improved most, with R@5 +9.3 points and MRR +11.9 points.
What remains is not another reranker feature. It is the unmeasured subset whose gold document never enters the roughly 100-row retrieval candidate universe. We did not complete the planned top-100 segmentation, so there is not enough evidence that this residue is still dominated by cross-conversation entity confusion or that an entity boost is the right remedy. The proposed graph-first retrieval path was also deliberately deferred pending better PostgreSQL graph-search support.
Future embedding/retrieval benchmarks should include reranker-off/on arms so model and candidate-generation effects remain distinguishable. If a fresh benchmark shows a material pool-absence population with a specific mechanism, open a new bounded candidate-generation issue from that evidence rather than retaining this pre-reranker proposal. basicmachines-co/basic-memory-cloud#2409 remains separate because it selects multilingual embedding models and plans Cloud migration, not entity-aware LoCoMo retrieval.