feat(rag): enable MMR/BM25/relevance-floor + benchmark RLM refinement & reranker #10600

Task 4 — RAG retrieval-quality flips & hybrid search

Theme: Precision. Several retrieval-quality features are fully built but disabled by default; the hybrid path uses naive substring matching instead of the existing BM25.

Problem

• MMR diversity disabled: knowledge/search_components/reranking.py:49 mmr_lambda=0.0; with reranking ON the crude _diversify_results is also skipped (advanced_rag_optimizer.py:799-800) → top-k can be near-duplicate chunks.
• RLM adaptive query refinement off: services/rag_config.py:92 enable_rlm_refinement=False; full loop exists (advanced_rag_optimizer.py:945, rag_service.py:198-213) and refinement history is logged-then-discarded.
• Hybrid keyword half uses substring TF, not BM25: advanced_rag_optimizer.py:734-740 loads all facts into memory per query and scores via _calculate_keyword_score (:365) — while knowledge/search_components/bm25.py exists unused.
• No relevance floor on optimizer hybrid path: advanced_rag_optimizer.py:648-715 applies no min score; non-chat callers (cag_service.py:130, graph_rag_service.py:486) ingest low-relevance chunks.
• Advanced-search feature flags never enabled: knowledge/search.py:645-647 enable_query_expansion/relevance_scoring/clustering all default False; only enable_reranking=True is passed (services/knowledge/service.py:151).
• Dated reranker: services/rag_config.py:47 pins cross-encoder/ms-marco-MiniLM-L-6-v2 (2021).

Subtasks

• 4.1 Enable MMR diversity — reranking.py:49, advanced_rag_optimizer.py:799
  • Step: set a sensible mmr_lambda default; benchmark via knowledge/rag_benchmarks.py
• 4.2 Replace substring keyword scan with BM25 index — advanced_rag_optimizer.py:365,734-740, knowledge/search_components/bm25.py
  • Step: route hybrid keyword half through BM25; remove full-corpus in-memory scan
• 4.3 Add a uniform relevance floor to advanced_search — advanced_rag_optimizer.py:648-715
  • Step: configurable min_score; protects all callers
• 4.4 Enable advanced-search quality flags in chat retrieval — services/knowledge/service.py:151
  • Step: pass enable_query_expansion/enable_relevance_scoring (benchmark clustering separately)
• 4.5 Benchmark + enable RLM refinement — services/rag_config.py:92
  • Step: A/B via rag_benchmarks.py; flip default only if recall improves without latency regression
• 4.6 Benchmark a stronger reranker — services/rag_config.py:47
  • Step: try bge-reranker / larger ms-marco; config-only swap; keep if precision lifts
• 4.7 Surface reranker-active metric — reranking.py:383-440
  • Step: emit health signal when reranking silently falls back (no sentence_transformers)

Verification

• Run knowledge/rag_benchmarks.py before/after each flip; record precision/recall deltas in the issue
• Confirm no latency regression on chat retrieval

Notes

Every flip is config-gated and benchmarked — no blind enabling. Reuse existing BM25/MMR/reranker code; do not reimplement.