Skip to content

Commit 2cb59cd

Browse files
YinyiLuoYinyiLuo
committed
add mmstar
1 parent 7438da5 commit 2cb59cd

43 files changed

Lines changed: 2789 additions & 6 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,18 @@
1818
<a href="https://huggingface.co/aifronter/post_train">🤗 Post-Training Weights</a>
1919
</p>
2020

21+
---
22+
23+
## News
24+
25+
- **Jul 2026**: Added MMStar evaluation benchmark support.
26+
27+
---
2128

2229
## Table of Contents
2330

2431
- [Table of Contents](#table-of-contents)
32+
- [News](#news)
2533
- [Introduction](#introduction)
2634
- [Supported Models](#supported-models)
2735
- [Repository Structure](#repository-structure)
@@ -210,6 +218,8 @@ cd -
210218

211219
**MMMU** — auto-downloaded from HuggingFace (`MMMU/MMMU`) at evaluation time, cached in `data/MMMU/`. No manual download needed.
212220

221+
**MMStar** — auto-downloaded from HuggingFace (`Lin-Chen/MMStar`) at evaluation time, cached in `data/MMStar/`. No manual download needed.
222+
213223
### Generation Benchmarks Data
214224

215225
These benchmarks include their data in the repository:
@@ -250,6 +260,9 @@ PYTHONPATH=src python -m umm.cli.main eval --config configs/eval/ueval/ueval_bag
250260

251261
# MME on Bagel
252262
PYTHONPATH=src python -m umm.cli.main eval --config configs/eval/mme/mme_bagel.yaml
263+
264+
# MMStar on Bagel
265+
PYTHONPATH=src python -m umm.cli.main eval --config configs/eval/mmstar/mmstar_bagel.yaml
253266
```
254267

255268
**Post-Training**
@@ -531,7 +544,7 @@ PYTHONPATH=src python -m umm.cli.main eval --config configs/eval/ueval/ueval_bag
531544
PYTHONPATH=src python -m umm.cli.main eval --config configs/eval/ueval/ueval_bagel_score.yaml
532545
```
533546

534-
**Single-stage benchmarks** (DPG Bench, MME, MMMU, MMBench, MM-Vet) run generation and scoring in one step:
547+
**Single-stage benchmarks** (DPG Bench, MME, MMMU, MMBench, MMStar, MM-Vet) run generation and scoring in one step:
535548

536549
```bash
537550
PYTHONPATH=src python -m umm.cli.main eval --config configs/eval/mme/mme_bagel.yaml
@@ -691,4 +704,3 @@ If you find TorchUMM useful in your research, please consider citing:
691704
url={https://arxiv.org/abs/2604.10784},
692705
}
693706
```
694-

‎configs/eval/mmstar/mmstar_bagel.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: bagel
6+
backbone_cfg:
7+
bagel_root: ${UMM_CODEBASE}/src/umm/backbones/bagel/Bagel
8+
model_path: ${UMM_MODEL_CACHE}/bagel/BAGEL-7B-MoT
9+
max_mem_per_gpu: 80GiB
10+
offload_folder: ./tmp/offload
11+
seed: 42
12+
13+
understanding_cfg:
14+
max_think_token_n: 20
15+
do_sample: false
16+
text_temperature: 0.3
17+
18+
mmstar:
19+
datasets: [MMStar]
20+
root: Lin-Chen/MMStar
21+
split: val
22+
cache_dir: data/MMStar
23+
out_dir: output/mmstar/bagel
24+
score_output_path: output/mmstar/bagel/summary.json
25+
max_new_tokens: 20
26+
resume: true
27+
answer_instruction: "Answer with the option's letter from the given choices directly."

‎configs/eval/mmstar/mmstar_emu3.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: emu3
6+
backbone_cfg:
7+
emu_root: ${UMM_CODEBASE}/model/Emu3
8+
model_path: ${UMM_MODEL_CACHE}/emu3/Emu3-Chat
9+
vq_hub: ${UMM_MODEL_CACHE}/emu3/Emu3-VisionTokenizer
10+
device_map: cuda:0
11+
device: cuda:0
12+
torch_dtype: bfloat16
13+
attn_implementation: flash_attention_2
14+
15+
generation_cfg:
16+
max_new_tokens: 1024
17+
18+
mmstar:
19+
datasets: [MMStar]
20+
root: Lin-Chen/MMStar
21+
split: val
22+
cache_dir: data/MMStar
23+
out_dir: output/mmstar/emu3
24+
score_output_path: output/mmstar/emu3/summary.json
25+
max_new_tokens: 20
26+
resume: true
27+
answer_instruction: "Answer with the option's letter from the given choices directly."

‎configs/eval/mmstar/mmstar_emu3_5.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: emu3_5
6+
backbone_cfg:
7+
model_path: ${UMM_MODEL_CACHE}/emu3_5/Emu3.5
8+
vq_path: ${UMM_MODEL_CACHE}/emu3_5/Emu3.5-VisionTokenizer
9+
emu3_5_root: ${UMM_CODEBASE}/model/Emu3.5
10+
use_vllm: true
11+
tensor_parallel_size: 2
12+
gpu_memory_utilization: 0.7
13+
classifier_free_guidance: 2.0
14+
image_area: 518400
15+
16+
generation_cfg:
17+
classifier_free_guidance: 2.0
18+
max_new_tokens: 512
19+
20+
understanding_cfg:
21+
image_area: 518400
22+
23+
mmstar:
24+
datasets: [MMStar]
25+
root: Lin-Chen/MMStar
26+
split: val
27+
cache_dir: data/MMStar
28+
out_dir: output/mmstar/emu3_5
29+
score_output_path: output/mmstar/emu3_5/summary.json
30+
max_new_tokens: 20
31+
resume: true
32+
answer_instruction: "Answer with the option's letter from the given choices directly."

‎configs/eval/mmstar/mmstar_janus_pro.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: janus_pro
6+
backbone_cfg:
7+
model_path: ${UMM_MODEL_CACHE}/janus_pro/Janus-Pro-7B
8+
janus_root: ${UMM_CODEBASE}/model/janus_pro/Janus
9+
seed: 42
10+
torch_dtype: bfloat16
11+
understanding_cfg:
12+
max_new_tokens: 2048
13+
do_sample: false
14+
use_cache: true
15+
16+
mmstar:
17+
datasets: [MMStar]
18+
root: Lin-Chen/MMStar
19+
split: val
20+
cache_dir: data/MMStar
21+
out_dir: output/mmstar/janus_pro
22+
score_output_path: output/mmstar/janus_pro/summary.json
23+
max_new_tokens: 20
24+
resume: true
25+
answer_instruction: "Answer with the option's letter from the given choices directly."

‎configs/eval/mmstar/mmstar_mmada.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: mmada
6+
backbone_cfg:
7+
model_path: ${UMM_MODEL_CACHE}/mmada/MMaDA-8B-Base
8+
mmada_root: ${UMM_CODEBASE}/model/MMaDA
9+
vq_model_path: ${UMM_MODEL_CACHE}/mmada/magvitv2
10+
seed: 42
11+
12+
understanding_cfg:
13+
max_new_tokens: 2048
14+
steps: 256
15+
block_length: 128
16+
17+
mmstar:
18+
datasets: [MMStar]
19+
root: Lin-Chen/MMStar
20+
split: val
21+
cache_dir: data/MMStar
22+
out_dir: output/mmstar/mmada
23+
score_output_path: output/mmstar/mmada/summary.json
24+
max_new_tokens: 20
25+
resume: true
26+
answer_instruction: "Answer with the option's letter from the given choices directly."

‎configs/eval/mmstar/mmstar_omnigen2.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: omnigen2
6+
backbone_cfg:
7+
omnigen2_root: ${UMM_CODEBASE}/model/OmniGen2
8+
model_path: ${UMM_MODEL_CACHE}/omnigen2/OmniGen2
9+
device: cuda:0
10+
device_map: cuda:0
11+
torch_dtype: bfloat16
12+
enable_cpu_offload: false
13+
enable_sequential_cpu_offload: false
14+
generation_cfg:
15+
num_inference_steps: 50
16+
seed: 42
17+
18+
understanding_cfg:
19+
max_new_tokens: 2048
20+
21+
mmstar:
22+
datasets: [MMStar]
23+
root: Lin-Chen/MMStar
24+
split: val
25+
cache_dir: data/MMStar
26+
out_dir: output/mmstar/omnigen2
27+
score_output_path: output/mmstar/omnigen2/summary.json
28+
max_new_tokens: 20
29+
resume: true
30+
answer_instruction: "Answer with the option's letter from the given choices directly."

‎configs/eval/mmstar/mmstar_ovis_u1.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: ovis_u1
6+
backbone_cfg:
7+
model_path: ${UMM_MODEL_CACHE}/ovis_u1/Ovis-U1-3B
8+
9+
understanding_cfg:
10+
max_new_tokens: 2048
11+
do_sample: false
12+
13+
mmstar:
14+
datasets: [MMStar]
15+
root: Lin-Chen/MMStar
16+
split: val
17+
cache_dir: data/MMStar
18+
out_dir: output/mmstar/ovis_u1
19+
score_output_path: output/mmstar/ovis_u1/summary.json
20+
max_new_tokens: 20
21+
resume: true
22+
answer_instruction: "Answer with the option's letter from the given choices directly."

‎configs/eval/mmstar/mmstar_show_o.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: show_o
6+
backbone_cfg:
7+
version: 1
8+
model_path: ${UMM_MODEL_CACHE}/show_o/show-o-w-clip-vit-512x512
9+
show_o_root: ${UMM_CODEBASE}/model/Show-o
10+
vq_model_path: ${UMM_MODEL_CACHE}/show_o/magvitv2
11+
seed: 42
12+
torch_dtype: bfloat16
13+
understanding_cfg:
14+
max_new_tokens: 2048
15+
do_sample: false
16+
use_clip_vit: true
17+
18+
mmstar:
19+
datasets: [MMStar]
20+
root: Lin-Chen/MMStar
21+
split: val
22+
cache_dir: data/MMStar
23+
out_dir: output/mmstar/show_o
24+
score_output_path: output/mmstar/show_o/summary.json
25+
max_new_tokens: 20
26+
resume: true
27+
answer_instruction: "Answer with the option's letter from the given choices directly."

‎configs/eval/mmstar/mmstar_show_o2.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
eval:
2+
benchmark: mmstar
3+
4+
inference:
5+
backbone: show_o2
6+
backbone_cfg:
7+
vae_path: ${UMM_MODEL_CACHE}/show_o2/Wan2.1_VAE.pth
8+
version: 2
9+
model_path: ${UMM_MODEL_CACHE}/show_o2/show-o2-7B
10+
show_o_root: ${UMM_CODEBASE}/model/Show-o
11+
seed: 42
12+
torch_dtype: bfloat16
13+
understanding_cfg:
14+
max_new_tokens: 2048
15+
do_sample: false
16+
use_clip_vit: true
17+
18+
mmstar:
19+
datasets: [MMStar]
20+
root: Lin-Chen/MMStar
21+
split: val
22+
cache_dir: data/MMStar
23+
out_dir: output/mmstar/show_o2
24+
score_output_path: output/mmstar/show_o2/summary.json
25+
max_new_tokens: 20
26+
resume: true
27+
answer_instruction: "Answer with the option's letter from the given choices directly."

0 commit comments

Comments
 (0)