Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950

Activity

1. claude commented on Jun 11, 2026

   

   claudebot

   on Jun 11, 2026 – with Claude

   Last edited by claude

   Contributor

   More actions

   Claude finished @groksrc's task in 2m 36s —— View job

   ---

   Issue Triage

   • Analyze issue content and classify
   • Explore relevant codebase (search/ranking code)
   • Check for duplicate issues
   • Apply appropriate labels
   • Provide triage summary

   ---

   Classification

   Field	Assessment
   Type	Enhancement — new optional rerank stage on top of existing hybrid search
   Priority	High — directly addresses ~48% of retrieval quality loss (135/281 unique misses), backed by quantitative benchmark data
   Component	Search pipeline (search_repository_base.py, search_service.py, fastembed_provider.py)
   Complexity	Complex — requires new cross-encoder model integration, configuration surface, pipeline hook, and benchmark validation

   Labels applied: enhancement

   ---

   Related Issues

   This issue overlaps with two existing open issues:

   • Add reranking step to search pipeline (local cross-encoder) #618 ("Add reranking step to search pipeline (local cross-encoder)") — Most closely related. Add reranking step to search pipeline (local cross-encoder) #618 proposes a tiered approach: cloud reranker for subscribers, OpenAI-compatible API for local, and no-op default. This issue (Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950) is complementary but takes a different angle: use fastembed's built-in cross-encoder models (already a dep, zero new user-facing config) rather than external APIs/cloud. The benchmark data here provides the quantitative motivation that Add reranking step to search pipeline (local cross-encoder) #618 lacked.
   • Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall #666 ("Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall") — Broader scope; reranking is one item in a larger list. Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950 is more actionable and focused.

   Verdict: Not a duplicate — #950 is a focused, evidence-backed proposal for a specific implementation path (fastembed local cross-encoder, ~50–150ms overhead, zero new dependencies) that complements but doesn't duplicate the tiered approach in #618. The benchmark data makes this standalone.

   ---

   Codebase Context

   The current hybrid search pipeline in search_repository_base.py:

   1. FTS (BM25/ts_rank) + vector (BGE small via fastembed) → scores in [0, 1]
   2. Fusion: max(vec, fts) + 0.3 * min(vec, fts) (fixed formula, no reranking hooks)
   3. Sort descending → paginate

   No existing reranking code or extension hooks. The natural insertion point is after fusion in _search_hybrid() / SearchService.search(), before the final sort + paginate.

   fastembed is already a dependency and ships cross-encoder/reranker models (BAAI/bge-reranker-v2-m3, jina-reranker). This means the implementation path the issue proposes — local reranking with no new install-time dependencies — is feasible.

   ---

   Implementation Notes (for whoever picks this up)

   1. Candidate expansion: Widen retrieval to top-20 (or configurable rerank_candidates, default 20) before reranking, then return top_k from the reranked set.
   2. Provider: fastembed ships reranker models via fastembed.rerank.cross_encoder.TextCrossEncoder — same ONNX/CPU path as the embedder.
   3. Config: New semantic_reranker_enabled / semantic_reranker_model config keys; default disabled to avoid latency regression for existing users.
   4. Extension point: Add optional reranker hook in SearchRepositoryBase parallel to the existing embedding_provider.
   5. Fusion weight tuning (mentioned in the issue): The FUSION_BONUS = 0.3 constant in search_repository_base.py is worth revisiting — short queries may benefit from higher semantic weight.
   6. Benchmark gate: The issue correctly notes this is measurable commit-to-commit with the benchmarks repo. Target: recover most of the 135 rank-6–10 misses without regressing latency beyond ~200ms mean.
2. claude

   added

   enhancementNew feature or requestNew feature or request

   on Jun 11, 2026
3. phernandez commented on Aug 5, 2026

   

   phernandez

   on Aug 5, 2026

   Member

   More actions

   implemented in #1143
4. phernandez

   closed this as completedon Aug 5, 2026
5. phernandez

   added this to the v0.23 milestone on Aug 11, 2026