rag-evalkit

Offline-first evaluation harness for RAG pipelines: score faithfulness, relevance, and retrieval quality on every commit, and fail the build when they regress.

What this solves

• RAG regressions ship silently: a prompt tweak or a chunking change degrades grounding and nobody notices until users complain. A --fail-under faithfulness=0.7 gate in CI catches it at the pull request.
• LLM-as-judge evaluation is nondeterministic and costs money per run, so teams stop running it. The default judge here is deterministic, free, and evaluates roughly 5,900 cases per second, so there is no excuse to skip it.
• Retrieval and generation fail differently. Separate metrics (recall@k, MRR, nDCG for the retriever; faithfulness, answer and context relevance for the generator) tell you which half of the pipeline to fix.

Architecture

flowchart LR
    P[Your RAG pipeline] -- "questions + contexts + answers" --> J[JSONL dataset]
    J --> R[rag-evalkit run]
    R --> H[Heuristic judge<br/>deterministic, offline]
    R -.optional.-> L[LLM judge<br/>caller-supplied client]
    R --> RM["recall@k, MRR, nDCG"]
    H & L & RM --> REP[Markdown report + per-case scores]
    REP --> G{"--fail-under gates"}
    G -- below threshold --> F[CI build fails]
    G -- pass --> OK[Merge]

Loading

Tech stack

Technology	Why it was chosen for this project
Python stdlib only (core)	Every metric is arithmetic over token lists; zero dependencies means instant installs and no version conflicts inside anyone's pipeline (ADR 0002)
Deterministic lexical judge	Same dataset, same scores, every time; a score change always means the pipeline changed, never the judge (ADR 0001)
Pluggable LLM judge	A one-argument completion callable is the entire integration contract; the library never handles API keys
FAISS + TF-IDF (demo extra)	Shows the full index-retrieve-evaluate loop running locally with no external services
GitHub Actions	CI runs lint, tests, the demo pipeline, and an mrr=0.9 quality gate on every push, eating the project's own dog food

Quickstart

pip install .
rag-evalkit run examples/generated_eval.jsonl -o report.md

As a CI quality gate:

rag-evalkit run eval.jsonl --fail-under faithfulness=0.7 --fail-under mrr=0.8

Dataset format (JSON Lines, retrieval fields optional):

{"id": "q1",
 "question": "How does RAG ground a language model's output?",
 "contexts": ["Retrieval augmented generation grounds a model by..."],
 "answer": "RAG grounds the model by injecting retrieved chunks...",
 "ranked_ids": ["d3", "d6", "d1"],
 "relevant_ids": ["d3", "d6"]}

Full local demo, building a FAISS index and evaluating retrieval end to end:

pip install .[demo]
python examples/faiss_demo.py
rag-evalkit run examples/generated_eval.jsonl

Using the LLM judge with any OpenAI-compatible client:

from openai import OpenAI
from rag_evalkit.judge import LLMJudge
from rag_evalkit.runner import evaluate, load_dataset

client = OpenAI()
judge = LLMJudge(lambda prompt: client.chat.completions.create(
    model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}]
).choices[0].message.content)
results = evaluate(load_dataset("eval.jsonl"), judge)

What the metrics catch

Running the demo dataset, which includes one deliberately hallucinated answer, produces this discrimination:

Case	Faithfulness	Notes
q1 to q4 (grounded)	0.53 to 0.92	Claims traceable to retrieved chunks
q5 (hallucinated)	0.10	"Invented at Bell Labs in 1962 with quantum annealing" matches almost nothing retrieved

Mean faithfulness across the set is 0.6386 precisely because the hallucinated case drags it down, which is the point: the aggregate moves when quality moves.

Throughput

Measured on a shared x86_64 container, Python 3.12, heuristic judge. Reproduce with python benchmark/run_benchmark.py.

Cases	Sentences per answer	Chunks	Total	Cases per second
1,000	3	3	0.17 s	~5,800
1,000	10	5	0.45 s	~2,200
10,000	3	3	1.69 s	~5,900

Design decisions

• ADR 0001: why the default judge is deterministic and where the LLM judge belongs
• ADR 0002: why the core has zero dependencies and FAISS lives in an extra

The Pivot

There is no embedding-based semantic similarity metric in v0.1. Neural embeddings would catch paraphrase that lexical matching misses, but they drag in a model download and a framework dependency, and they reintroduce the nondeterminism the default judge exists to avoid. If paraphrase-heavy pipelines show systematic underscoring, the right addition is an optional embedding extra behind the same Judge protocol.

Known limitations, stated plainly

Lexical faithfulness is a regression tripwire, not a truth oracle. An answer that inverts meaning while reusing context tokens will score higher than it deserves. Use the heuristic judge to catch regressions cheaply on every commit, and the LLM judge adapter for periodic deeper audits.

Failure modes

• Malformed dataset lines fail loudly with the file and line number, never silently skipped.
• Empty datasets are rejected rather than reporting vacuous perfect scores.
• An LLM judge returning non-JSON or missing keys raises immediately instead of coercing garbage into a score.
• Judge scores outside [0, 1] are clamped and never propagate out of range.

Project layout

src/rag_evalkit/   metrics, retrieval metrics, judges, runner, CLI
tests/             23 tests covering metrics, judges, runner, CLI gates
examples/          FAISS demo corpus and generated evaluation dataset
benchmark/         reproducible throughput benchmark
docs/adr/          architecture decision records

Roadmap

• Optional embedding-based similarity judge behind the same protocol
• JUnit XML output so gate failures annotate pull requests natively
• Pairwise A/B comparison mode for prompt and retriever experiments

License

MIT