Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

We construct ClinMM-Bench, the largest benchmark for multi-turn multimodal diagnostic reasoning on challenging real-world clinical cases to date. We evaluates how MLLMs synthesize progressively disclosed clinical information and images to arrive at a diagnosis and produce clinically grounded reasoning. ClinMM-Bench covers 8 specialties including Dermatology, Emergency Medicine, Internal Medicine, Nephrology, Neurology, Oncology, Ophthalmology, and Radiology, containing 1,089 challenging real-world cases.

Repository Structure

run_data_pipeline.py          # Build ClinMM data from PMCOA
run_diagnosis_eval.py         # Evaluate diagnosis accuracy
run_reasoning_eval.py         # Evaluate reasoning quality

source/pmc/                   # PMCOA matching, download, extraction, inspection, validation, conversion, quality control
source/eval/                  # Data loading, model evaluation, diagnosis judging, reasoning judging

tmp_test/                     # Minimal test data

Installation

Create and activate a Python environment:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Create a local .env file from .env.example and add API keys:

cp .env.example .env

OPENAI_API_KEY=
ANTHROPIC_API_KEY=

Minimal Test Data

The repository includes a small test fixture under tmp_test/ so that users can verify the pipeline without downloading the full benchmark data:

tmp_test/data/pmid/pmid_internalmed.txt
tmp_test/data/oa_list/oa_list.csv

Data Pipeline

The data pipeline starts from curated PMID lists and PMCOA metadata, then constructs standardized multimodal clinical cases.

Pipeline steps:

match      Match specialty PMID lists with PMCOA metadata
download   Download PMCOA article packages
extract    Extract XML text and medical images
check      Exclude invalid cases with videos, and low-resolution images
validate   Use LLMs to identify diagnostically suitable case reports
convert    Convert validated case reports into standardized JSON format
control    Run quality control on converted cases

Run a dry run with the minimal test fixture:

python3 run_data_pipeline.py \
  --specialties internalmed \
  --steps all \
  --dry-run \
  --data-dir tmp_test/data

Run the minimal test fixture for real:

python3 run_data_pipeline.py \
  --specialties internalmed \
  --steps all \
  --data-dir tmp_test/data

Use --help to inspect all options:

python3 run_data_pipeline.py --help

Evaluation Framework

ClinMM-Bench uses a two-level evaluation framework.

1. Diagnosis Accuracy Evaluation

Run diagnosis evaluation from saved model outputs:

python3 run_diagnosis_eval.py \
  --specialty internalmed \
  --model-name gpt-5-medium

2. Reasoning Quality Evaluation

The reasoning metrics are:

fact recall       = matched ground-truth facts / all ground-truth facts
factual density   = matched predicted facts / all predicted facts
hallucination     = hallucinated predicted facts / all predicted facts

Run reasoning evaluation from saved model outputs:

python3 run_reasoning_eval.py \
  --specialty internalmed \
  --model-name gpt-5-medium

Citation

Citation information will be added after the manuscript is publicly available.