MM-RAGBench

MM-RAGBench is a benchmark dataset for evaluating multimodal Retrieval-Augmented Generation systems. It is built as a native companion to mmeval-vrag, so multimodal RAG pipelines can be loaded, evaluated, and compared in a few lines of Python.

MM-RAGBench focuses on the failure modes that matter in real systems: weak retrieval, unsupported visual claims, cross-modal inconsistencies, hallucinated attributes, and answers that are fluent but not grounded in the retrieved evidence.

---

Overview

MM-RAGBench provides a standardized benchmark for evaluating systems that retrieve and reason over both text and images. It is designed for researchers and practitioners building multimodal RAG systems, vision-language models, agentic RAG pipelines, and evaluation infrastructure.

The benchmark supports end-to-end evaluation of retrieval quality, answer faithfulness, hallucination rate, cross-modal grounding, and multimodal consistency through direct integration with mmeval-vrag.

---

Key Features

• 3,000 multimodal queries across 6 domains requiring cross-modal reasoning
• 12,000 candidate documents as image-text pairs from open-license sources
• Fine-grained annotations for hallucination traps, faithfulness criteria, difficulty, and modality requirements
• Native mmeval-vrag integration with EvalSample and QueryItem
• All 11 mmeval-vrag metrics supported out of the box
• Per-domain and per-difficulty analysis for granular model comparison
• Benchmark artifacts suitable for regression testing, failure analysis, and post-training data generation

---

The Problem

Existing RAG benchmarks such as RAGBench, ViDoRe, and NoMIRACL primarily focus on text-only retrieval, text-heavy settings, or retrieval without full multimodal generation evaluation.

Real-world RAG systems often retrieve images alongside text: medical scans with clinical notes, product photos with descriptions, diagrams with documentation, landmark images with historical context, or cooking images with recipe instructions.

A strong multimodal RAG system must answer questions using both modalities while avoiding unsupported visual claims. MM-RAGBench is designed to test exactly that.

---

What MM-RAGBench Evaluates

Capability	What it measures
Multimodal RAG evaluation	Whether a system can retrieve and reason over both text and images.
Retrieval quality	Whether relevant image-text evidence appears in the top-K retrieved results.
Answer faithfulness	Whether generated claims are supported by the retrieved evidence.
Hallucination rate	Whether the model introduces unsupported visual or textual details.
Cross-modal grounding	Whether the answer is aligned with both visual and textual evidence.
Multimodal consistency	Whether image-text pairs are internally consistent and relevant to the query.
Failure modes	Object, attribute, relation, and fabrication hallucinations.

---

Architecture

flowchart LR
    A[MM-RAGBench Dataset<br/>Queries, images, text, annotations] --> B[User RAG System]

    B --> C[Retriever<br/>Text / Image / Hybrid]
    C --> D[Retrieved Evidence<br/>Image-text candidates]

    D --> E[Generator<br/>LLM / VLM / RAG Pipeline]
    E --> F[Generated Answer]

    A --> G[mmeval-vrag Evaluation Engine]
    D --> G
    F --> G

    G --> H[Retrieval Metrics<br/>Precision, Recall, MRR, NDCG]
    G --> I[Generation Metrics<br/>Faithfulness, Hallucination, Relevance]
    G --> J[Cross-modal Metrics<br/>Alignment, Grounding, Consistency]

    H --> K[Evaluation Report]
    I --> K
    J --> K

    K --> L[Leaderboard / Regression Testing]
    K --> M[Failure Analysis]
    M --> N[Corrected Examples / Preference Data]

Loading

---

Evaluation-to-Improvement Loop

MM-RAGBench can be used not only as a benchmark, but also as part of an evaluation-driven model improvement workflow.

flowchart LR
    A[Run RAG / Agent System] --> B[Log Retrieved Evidence + Answer]
    B --> C[Evaluate with MM-RAGBench + mmeval-vrag]
    C --> D[Identify Failures]
    D --> E[Hallucinations]
    D --> F[Weak Retrieval]
    D --> G[Cross-modal Mismatch]

    E --> H[Corrected Answers]
    F --> H
    G --> H

    H --> I[SFT Examples]
    D --> J[Preference Pairs]
    J --> K[DPO Data]
    I --> L[LoRA / QLoRA Fine-tuning]
    K --> L
    L --> M[Re-evaluate Checkpoint]
    M --> C

Loading

Typical artifacts:

Artifact	Use case
traces.jsonl	Stores query, retrieved evidence, generated answer, and metadata.
eval_report.json	Contains retrieval, generation, and cross-modal metrics.
failures.jsonl	Captures hallucinations, weak retrieval, missing evidence, or unsupported claims.
sft.jsonl	Supervised fine-tuning examples from corrected outputs.
dpo_pairs.jsonl	Preference pairs comparing stronger and weaker grounded answers.

---

Install

pip install mm-ragbench

This pulls in mmeval-vrag automatically.

---

Quick Start

Option 1: Evaluate pre-computed answers

from mm_ragbench import load_eval_samples
from mmeval_vrag import MultimodalRAGEvaluator, EvalConfig

# Load benchmark as mmeval-vrag EvalSample objects
samples = load_eval_samples(split="test", max_samples=100)

# Your system fills in generated_answer for each sample
for sample in samples:
    docs = your_retriever(sample.query_text, sample.query_image)
    sample.retrieved = docs
    sample.generated_answer = your_generator(sample.query_text, docs)

# Evaluate with mmeval-vrag
evaluator = MultimodalRAGEvaluator(config=EvalConfig(metrics=["all"]))
results = evaluator.evaluate(samples)

print(results.summary())

Option 2: Evaluate a live pipeline

from mm_ragbench import load_query_items
from mmeval_vrag.evaluators.pipeline import EvalPipeline
from mmeval_vrag import EvalConfig

# Load benchmark as mmeval-vrag QueryItem objects
queries = load_query_items(split="test")

# Plug in your retriever + generator
pipeline = EvalPipeline(
    retriever=my_retriever,   # (query_text, query_image, top_k) -> List[RetrievedItem]
    generator=my_generator,   # (query_text, contexts) -> str
    config=EvalConfig(metrics=["all"]),
)

results = pipeline.run(queries)
results.to_json("my_system_results.json")

Option 3: Use via mmeval-vrag loader registry

from mmeval_vrag.datasets import load_dataset
import mm_ragbench  # registers the "mm_ragbench" loader

samples = load_dataset(
    "mm_ragbench",
    "EmmanuelleB985/mm-ragbench",
    split="test",
)

---

Metrics via mmeval-vrag

All 11 mmeval-vrag metrics are supported.

Category	Metric	What it measures
Retrieval	retrieval_precision	Fraction of top-K items that are relevant
Retrieval	retrieval_recall	Fraction of relevant items found in top-K
Retrieval	retrieval_mrr	Reciprocal rank of the first relevant item
Retrieval	retrieval_ndcg	Normalized DCG accounting for rank
Generation	faithfulness	Whether generated claims are supported by context
Generation	hallucination_rate	Fraction of unsupported claims; lower is better
Generation	answer_relevance	Similarity between answer and query
Generation	context_relevance	Relevance of retrieved passages to query
Cross-modal	cross_modal_alignment	CLIP similarity between images and query
Cross-modal	visual_grounding	CLIP similarity between images and answer
Cross-modal	multimodal_consistency	CLIP similarity within image-text pairs

---

Example Evaluation Report

MM-RAGBench is designed to compare retrievers, generators, prompts, agent settings, and model checkpoints with the same evaluation harness.

System / Configuration	Retrieval R@5 ↑	Faithfulness ↑	Hallucination ↓	Cross-modal ↑	Overall ↑
Text-only RAG baseline	0.61	0.58	0.31	0.42	0.55
Multimodal RAG baseline	0.72	0.67	0.24	0.63	0.67
Hybrid retrieval + VLM generator	0.79	0.73	0.18	0.70	0.74
Agentic RAG + self-check	0.82	0.77	0.14	0.74	0.78

---

Dataset Schema

Each sample maps directly to mmeval-vrag types.

MM-RAGBench field          -> mmeval-vrag type
──────────────────────────────────────────────────────
query / query_image        -> EvalSample.query_text / query_image
gold_doc_texts/images      -> EvalSample.retrieved / List[RetrievedItem]
gold_answer                -> EvalSample.reference_answer
domain, difficulty, ...    -> EvalSample.metadata
hallucination_traps        -> EvalSample.metadata["hallucination_traps"]
faithfulness_criteria      -> EvalSample.metadata["faithfulness_criteria"]
gold_doc_ids               -> QueryItem.relevant_ids for EvalPipeline

Annotation Fields

Each query includes:

• hallucination_traps: known failure modes, for example:

  {
    "type": "attribute",
    "description": "May confuse bridge completion year with construction start year."
  }

• faithfulness_criteria: verifiable checks, for example:

  Must identify bridge type from visual features.
  

• answer_modality: whether the answer requires:

  • text_only
  • image_only
  • cross_modal

• difficulty:

  • easy: 40%
  • medium: 35%
  • hard: 25%

---

Domains

Domain	Queries	Sources
Science & Nature	500	Wikipedia diagrams, species photos, experiments
Geography & Travel	500	Landmarks, maps, cultural sites
History & Art	500	Historical photos, artworks, architecture
Technology	500	Product images, diagrams, interfaces
Food & Cooking	500	Recipe images, ingredients, techniques
Daily Life	500	Everyday objects, how-to guides, sports

---

Build the Dataset from Scratch

from mm_ragbench import MMRAGBenchBuilder

builder = MMRAGBenchBuilder(
    output_dir="data/mm-ragbench",
    storage_mode="urls",                   # "urls" | "thumbnails" | "full"
    llm_provider="anthropic",              # or "openai"
    llm_model="claude-sonnet-4-20250514",  # or "gpt-4o"
)

builder.collect_sources()          # Pull from WIT + COCO
builder.generate_queries()         # Generate queries and annotations
builder.generate_hard_negatives()  # Same-domain distractors
builder.verify_and_balance()       # Balance to 3,000 queries
builder.export_jsonl()             # Export JSONL compatible with mmeval-vrag
builder.push_to_hub("EmmanuelleB985/mm-ragbench")

Storage Modes

Mode	Disk	What's saved	Image access
"urls"	~10 MB	Text + Wikimedia/COCO image URLs	Fetched on demand during eval
"thumbnails"	~300 MB	224px JPEG thumbnails	Local files
"full"	~50 GB	Original-resolution images	Local files

The default is "thumbnails", which provides a good balance between quality and size.

Use "urls" if disk is tight; images are fetched lazily when mmeval-vrag cross-modal metrics need them.

---

Per-Domain Analysis

MM-RAGBench metadata enables granular analysis by domain, difficulty, answer modality, and hallucination-trap type.

from mm_ragbench import load_eval_samples

samples = load_eval_samples(split="test")

# Assume results was produced by mmeval-vrag
by_domain = {}

for sample, result in zip(samples, results.results):
    domain = sample.metadata["domain"]
    by_domain.setdefault(domain, []).append(result.scores)

for domain, scores in sorted(by_domain.items()):
    faith = [s["faithfulness"] for s in scores if "faithfulness" in s]
    halluc = [s["hallucination_rate"] for s in scores if "hallucination_rate" in s]

    avg_faith = sum(faith) / len(faith) if faith else float("nan")
    avg_halluc = sum(halluc) / len(halluc) if halluc else float("nan")

    print(
        f"{domain}: "
        f"faithfulness={avg_faith:.3f}  "
        f"hallucination={avg_halluc:.3f}"
    )

---

Failure Analysis

MM-RAGBench is designed to expose common multimodal RAG failure modes.

Failure type	Example
Object hallucination	The answer mentions an object that is not visible in the image.
Attribute hallucination	The answer gives the wrong color, size, material, year, or count.
Relation hallucination	The answer misstates spatial or semantic relationships between objects.
Text-image mismatch	The answer relies on text evidence but ignores contradictory visual evidence.
Weak retrieval	The correct evidence is missing from the top-K retrieved items.
Overconfident answer	The model gives a definitive answer despite insufficient evidence.

These failures can be logged and converted into corrected answers, SFT examples, or DPO-style preference pairs.

---

Example Trace Format

A typical evaluation trace can be exported as JSONL.

{
  "query_id": "mmrb_test_0001",
  "query": "What type of bridge is shown, and what visual features support the answer?",
  "modalities": ["text", "image"],
  "answer_modality": "cross_modal",
  "domain": "Geography & Travel",
  "difficulty": "medium",
  "retrieved_evidence": [
    {
      "id": "doc_042",
      "type": "image_text_pair",
      "score": 0.87,
      "text": "The Golden Gate Bridge is a suspension bridge...",
      "image_url": "https://..."
    }
  ],
  "generated_answer": "The image shows a suspension bridge, supported by the tall towers and hanging cables.",
  "reference_answer": "It is a suspension bridge, identifiable from the vertical suspenders and main cables between the towers.",
  "metrics": {
    "retrieval_recall_at_5": 1.0,
    "faithfulness": 0.84,
    "hallucination_rate": 0.05,
    "cross_modal_alignment": 0.78
  }
}

---

Example DPO Pair Format

Failure cases can be transformed into preference data for post-training workflows.

{
  "prompt": "Answer using only the retrieved evidence: What type of bridge is shown?",
  "chosen": "The image shows a suspension bridge, supported by visible towers and hanging cables.",
  "rejected": "The image shows a stone arch bridge built in the medieval period.",
  "metadata": {
    "query_id": "mmrb_test_0001",
    "failure_type": "visual_hallucination",
    "source": "mm-ragbench"
  }
}

---

Leaderboard

System	Retrieval R@5 ↑	Faithfulness ↑	Hallucination ↓	Cross-modal ↑	Overall ↑
Submit yours	—	—	—	—	—

To submit:

1. Run your system on the MM-RAGBench test split.

2. Export results with:

   results.to_json("my_system_results.json")

3. Open a PR with:

   • my_system_results.json
   • system description
   • model names
   • retriever details
   • generator details
   • prompts or decoding settings, if applicable

---

Suggested Repository Structure

mm-ragbench/
  mm_ragbench/
    __init__.py
    loaders.py
    builder.py
    schema.py
    analysis.py

  data/
    README.md
    sample.jsonl

  examples/
    quickstart_eval_samples.py
    quickstart_eval_pipeline.py
    per_domain_analysis.py
    export_failures_for_post_training.py

  docs/
    architecture.md
    dataset_card.md
    evaluation.md
    post_training_loop.md

  tests/
    test_loaders.py
    test_schema.py
    test_mmeval_integration.py

---

Data Sources & Licensing

Source	License	Used for
Wikipedia via WIT	CC-BY-SA 3.0	Article text and images
Wikimedia Commons	CC-BY / CC-BY-SA	Images
COCO	CC-BY 4.0	Everyday scene images
Generated annotations	CC-BY 4.0	Queries, answers, traps, criteria

All source-image licensing should be stored at the per-sample level when possible.

---

Citation

@software{bourigault2026mmragbench,
  author = {Bourigault, Emmanuelle},
  title = {MM-RAGBench: Multimodal Benchmark for Evaluating RAG Systems},
  year = {2026},
  url = {https://github.com/EmmanuelleB985/mm-ragbench},
}

@software{bourigault2025mmeval,
  author = {Bourigault, Emmanuelle},
  title = {mmeval-vrag: Evaluation Framework for Multimodal Vision-Language RAG Systems},
  year = {2025},
  url = {https://github.com/EmmanuelleB985/mmeval-vrag},
}

---

License

Code: Apache 2.0
Dataset annotations: CC-BY 4.0
Source images: per-sample license, all CC-BY or CC-BY-SA