📊 RAG Evaluation Framework

Automatically evaluates RAG pipelines using Claude as the judge across 5 critical metrics. Includes a Streamlit dashboard, pipeline comparison, and automated test suite. This is the skill that separates mid-level from senior AI engineers.

Why This Project Gets You Hired

Every serious AI company asks in interviews:

"How do you know your RAG system is actually working?" "How do you measure hallucination rate?" "How do you compare two RAG configurations objectively?"

This project answers all three — with a production-quality evaluation system.

The 5 Metrics

Metric	What It Measures	Threshold
Faithfulness	Answer only uses facts from context	≥ 0.70
Answer Relevance	Answer directly addresses the question	≥ 0.70
Context Precision	Retrieved chunks are relevant to query	≥ 0.60
Context Recall	Context contains the needed information	≥ 0.60
Hallucination Rate	Answer contains fabricated facts (lower = better)	≤ 0.30

Architecture — LLM-as-Judge Pattern

RAG Sample
(question + context + answer)
          │
          ▼
┌─────────────────────────────────────┐
│         Claude Judge (Haiku)        │  ← fast + cheap for eval
│                                     │
│  ┌───────────┐  ┌────────────────┐  │
│  │Faithfulness│  │Answer Relevance│  │
│  └───────────┘  └────────────────┘  │
│  ┌────────────┐  ┌──────────────┐   │
│  │Ctx Precision│  │Ctx Recall    │   │
│  └────────────┘  └──────────────┘   │
│  ┌──────────────────────────────┐   │
│  │   Hallucination Detector     │   │
│  └──────────────────────────────┘   │
└─────────────────────────────────────┘
          │
          ▼
MetricScore(score, reasoning, passed)
          │
          ▼
Overall Score = mean(faith, rel, prec, recall, 1-halluc)
          │
          ▼
Pipeline Report + Comparison Dashboard

Key Features

• ✅ 5 automated metrics — all using Claude as judge
• ✅ LLM-as-Judge pattern — same approach used at Anthropic and OpenAI
• ✅ Pipeline comparison — benchmark different RAG configurations head-to-head
• ✅ Radar chart visualization — instant visual quality assessment
• ✅ Downloadable reports — markdown eval reports per pipeline
• ✅ Configurable thresholds — adjust passing criteria per use case
• ✅ Streamlit dashboard — no-code interface for non-engineers

Quick Start

# 1. Install
pip install -r requirements.txt

# 2. Set API key
export ANTHROPIC_API_KEY=your_key_here

# 3. Run CLI evaluation
cd src && python evaluator.py

# 4. Run Streamlit dashboard
streamlit run src/dashboard.py

Code Example

from evaluator import RAGEvaluator, RAGSample

evaluator = RAGEvaluator()

sample = RAGSample(
    question="What is RAG?",
    context=["RAG combines retrieval with generation..."],
    answer="RAG is a framework that retrieves relevant documents before generating answers.",
)

result = evaluator.evaluate_sample(sample)
print(f"Faithfulness:  {result.faithfulness.score:.3f}")
print(f"Hallucination: {result.hallucination_rate.score:.3f}")
print(f"Overall:       {result.overall_score:.3f}")

# Compare two pipelines
pipeline_a = evaluator.evaluate_pipeline(samples_a, "naive-rag")
pipeline_b = evaluator.evaluate_pipeline(samples_b, "reranked-rag")
comparison = evaluator.compare_pipelines([pipeline_a, pipeline_b])
print(f"Winner: {comparison['winner']}")

Sample Dashboard

The Streamlit dashboard shows:

• Radar chart comparing your pipeline against passing thresholds
• Per-sample bar charts to identify which questions the RAG struggles with
• Pipeline history for comparing different configurations
• Detailed reasoning from Claude judge for each metric score

What I Learned

• LLM-as-Judge — using a fast, cheap model (Haiku) to evaluate outputs from a larger model — the industry standard approach at scale
• Metric design — why hallucination needs to be inverted (lower = better) and weighted in overall score
• Structured prompts for evaluation — each metric needs a precisely worded rubric to get consistent scores
• Pipeline comparison — ranking configurations by overall score vs individual metrics tells different stories
• Production eval thinking — evaluation is not a one-time task, it's a continuous feedback loop

Tech Stack

Python 3.11 · Anthropic Claude (Haiku as judge) · Streamlit · Plotly · Pandas · NumPy