mmmu_pro_vision fails: MMMU/MMMU_Pro re-uploaded on 2026-10-05; datasets-server no longer serves pinned revision 563f3e84 #134

Summary

mmmu_pro_vision now fails on every run with:

DatasetError: Failed to map MMMU/MMMU_Pro record(s) at offset 0: TypeError: MMMU Pro image test_History_1 changed; regenerate media manifest for the current dataset revision

Cause

• The dataset changed upstream. MMMU/MMMU_Pro was re-uploaded on 2026-10-05: commits 46a29aca, a22c001a and 571c96a8 ("Upload dataset"), then the new head 0d7426df ("Update README.md"). Every vision/test-0000x-of-00004.parquet file changed between the pinned revision 563f3e84bb3b90893083a1f039cfa13077f2302b and 0d7426df (the sizes and LFS oids differ).
• datasets-server ignores the pin. It now serves rows only from the new revision, even when revision=563f3e84… is passed. For example, the image src for row 0 has the path /cached-assets/MMMU/MMMU_Pro/--/0d7426df…/--/vision/test/0/image/image.png.
• The guard therefore always fires. mirroredMmmuProImage requires the path for the pinned revision, so it rejects every row.

The guard is doing its job: the new rows aren't the pinned rows. But the benchmark can't currently run at the pinned revision through datasets-server at all.

Possible fixes

• Read the pinned revision's parquet files directly (resolve/563f3e84…/vision/test-*.parquet, for example with hyparquet range reads), keeping the existing mirror manifest. This keeps results comparable with runs made before 2026-10-05.
• Or re-pin to the new revision and regenerate the media manifest and mirror. Scores would then no longer be comparable with earlier published MMMU-Pro results.

Environment

benchmark-harness d22d57b (and 78154b8); bun 1.3.14.