Add RAG FiQA MRR optimization community notebook

[kamran-rapidfireAI]

Summary

• Adds rag_fiqa_mrr_optimization.ipynb to the community_notebooks/ folder
• Sourced from the AI Winter 2025 competition notebooks

Test plan

• Verify notebook renders correctly on GitHub
• Confirm notebook cells execute without errors

Made with Cursor

---

Note

Low Risk
Adds a standalone notebook only; no library/runtime code paths are modified, with risk limited to notebook execution/dependency assumptions.

Overview
Adds a new community Colab notebook, community_notebooks/rag_fiqa_mrr_optimization.ipynb, that runs a RapidFire AI multi-config RAG evaluation on the FiQA dataset.

The notebook installs/initializes RapidFire AI, downsamples and filters FiQA queries/corpus, grid-searches over RAG chunking and reranker top_n settings with a vLLM Qwen generator, computes retrieval metrics (including MRR), and outputs a results DataFrame plus simple metric plots/log viewing helpers.

^{Written by Cursor Bugbot for commit 4099c2b. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[cursor]

Set conversion destroys ordering for rank-sensitive metrics

High Severity

The ordered list of retrieved documents (pred) is converted to a Python set via set(pred), which destroys the ranking order. This retrieved_set is then passed to compute_ndcg_at_k and compute_rr, both of which are rank-sensitive metrics that depend on document position. Iterating a set yields arbitrary order, so NDCG and MRR — the notebook's primary optimization target — produce meaningless, non-deterministic values.

Additional Locations (1)

• community_notebooks/rag_fiqa_mrr_optimization.ipynb#L348-L369

 

[cursor]

NDCG uses mismatched relevance scales in DCG vs IDCG

Medium Severity

In compute_ndcg_at_k, the actual DCG is computed with binary relevance values (0 or 1), but the ideal DCG (idcg) uses a relevance value of 3 for each relevant document. This mismatch means the NDCG score is systematically scaled down by a factor of ~3, making the metric incorrect. Both DCG and IDCG need to use the same relevance scale.