Enhancement: MMR (Maximal Marginal Relevance) diversity scoring in reranker #2090

Summary

Add Maximal Marginal Relevance (MMR) as a post-reranking diversity pass in ResultReranker. Currently the reranker blends cross-encoder + vector scores (#2004) but does not penalize redundant results — if 5 near-identical chunks score highly, all 5 are returned, wasting LLM context window.

Motivation

• Inspired by MMR diversity scoring patterns used in multi-agent orchestration frameworks
• AutoBot's existing diversity_threshold in advanced_rag_optimizer.py is binary dedup (drop if too similar) — MMR is a graded alternative that balances relevance vs. novelty
• Direct improvement to RAG answer quality with minimal code change

Proposed Implementation

Add mmr_lambda to RerankWeights (default 0.0 = disabled, backward-compatible):

# In reranking.py — RerankWeights
mmr_lambda: float = 0.0  # 0.0 = disabled, 0.3-0.7 = typical

# After cross-encoder reranking, apply MMR selection:
# score = λ * relevance(doc, query) - (1-λ) * max(cosine_sim(doc, already_selected))

When mmr_lambda > 0, re-order the top-N reranked results using MMR before returning.

Files Affected

• autobot-backend/knowledge/search_components/reranking.py — add MMR pass after _apply_rerank_scores()
• autobot-backend/services/rag_config.py — add mmr_lambda to RAGConfig
• autobot-backend/constants/model_constants.py — add RAG_MMR_LAMBDA default

Compatibility

• Default off (mmr_lambda=0.0) — zero behavior change unless explicitly enabled
• Slots into existing RerankWeights dataclass alongside edge and recency weights
• Natural fit with Neural Mesh Phase 3 — enable MMR when graph-expanded results increase redundancy

Acceptance Criteria

• mmr_lambda configurable via RAGConfig
• MMR pass applied after cross-encoder reranking when lambda > 0
• Unit tests covering: disabled (lambda=0), moderate diversity (0.5), high diversity (0.9)
• No regression when disabled (existing tests pass unchanged)