[ICML 2026] FreeRet: MLLMs as Training-Free Retrievers

Yuhan Zhu · Xiangyu Zeng · Chenting Wang · Xinhao Li · Chunxu Liu

Yicheng Xu · Ziang Yan · Yi Wang · Limin Wang

Nanjing University · Shanghai AI Lab · Shanghai Jiaotong University

Institute of Science Tokyo · Zhejiang University

FreeRet turns any off-the-shelf multimodal large language model (MLLM) into a two-stage retriever — without additional training. It first extracts semantically grounded embeddings for fast candidate search, then leverages the model's reasoning for precise MCQ-based reranking.

Built on Qwen2.5-VL as the backbone, this repository provides:

Module	Description
demo/	Minimal examples for embedding search and reranking
mmeb_evaluation/	Full evaluation pipeline on the MMEB benchmark (36 datasets)

Overview

Query  ──►  Stage 1: One-word Embedding Search  ──►  Top-K candidates
                                                      │
                                                      ▼
              Stage 2: MCQ-based Reranking  ◄─────────┘
                      │
                      ▼
                 Final ranking

Key ideas

• Bypass the final MLP's lexicalization pressure to obtain semantically faithful embeddings
• Inject explicit priors via controlled one-word summarization prompts
• Mitigate framing effects in reranking with neutral multiple-choice framing (A/B)

Installation

Install dependencies following the Qwen2.5-VL setup guide, especially qwen-vl-utils.

Our tested environment:

Package	Version
Python	3.10
PyTorch	2.5.0+cu121
transformers	4.57.0.dev0
flash-attn	2.7.3
qwen-vl-utils	0.0.11

Download a Qwen2.5-VL checkpoint (3B / 7B / 32B) from Hugging Face.

Quick Start — Demo

A self-contained example walks through a composed image retrieval case (search → top-K → rerank):

cd demo
# Set checkpoint_path in demo.py to your local Qwen2.5-VL weights
python demo.py

See demo/README.md for environment details and checkpoint configuration.

Pipeline in demo.py

1. Stage 1 (mode="Search"): encode query & targets with one-word prompts → cosine similarity
2. Stage 2 (mode="Rerank"): MCQ reranking on top-K candidates → final choice

MMEB Evaluation

1. Prepare data

Follow the MMEB instructions to download evaluation images and metadata. Then update the paths in mmeb_evaluation/run_mmeb.py:

image_root_path = "/path/to/MMEB-eval/eval_images"
# load_dataset("/path/to/MMEB-eval/meta", ...)

2. Run evaluation

Evaluate a single subset:

cd mmeb_evaluation
# Edit CKPT and paths in run_mmeb.sh
bash run_mmeb.sh ImageNet-1K

Run all 36 subsets (requires a cluster scheduler; adapt run_all.sh to your environment):

bash run_all.sh

Results are saved under mmeb_evaluation/outputs/<model_name>/embed_rerank/.

Argument	Description
--model_path	Path to Qwen2.5-VL checkpoint
--subset_name	MMEB dataset name (e.g. CIRR, MSCOCO)
--rerank_num	Number of candidates to rerank (default: 50)
--out_dir	Output directory for per-subset JSON results

Repository Structure

FreeRet/
├── demo/
│   ├── demo.py                  # Embedding search + reranking example
│   ├── modeling_qwen2_5_vl.py   # Modified Qwen2.5-VL forward (Search / Rerank modes)
│   └── sources/                 # Example images
├── mmeb_evaluation/
│   ├── run_mmeb.py              # Per-subset evaluation script
│   ├── run_mmeb.sh              # Single-subset launcher
│   ├── run_all.sh               # Batch launcher for all subsets
│   ├── input_format_oneword.py  # Task-specific prompts for MMEB
│   └── modeling_qwen2_5_vl.py
└── README.md

Citation

If you find this work useful, please cite:

@article{zhu2025freeret,
  title={Freeret: Mllms as training-free retrievers},
  author={Zhu, Yuhan and Zeng, Xiangyu and Wang, Chenting and Li, Xinhao and Liu, Chunxu and Xu, Yicheng and Yan, Ziang and Wang, Yi and Wang, Limin},
  journal={arXiv preprint arXiv:2509.24621},
  year={2025}
}

Acknowledgments

This codebase builds upon Qwen2.5-VL and the MMEB benchmark. We thank the authors for their open-source contributions.