Skip to content

Commit

Permalink
Update mmmu-pro
Browse files Browse the repository at this point in the history
  • Loading branch information
ShuaiBai623 authored Oct 4, 2024
1 parent dcc62fe commit 3bf7dbd
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ We have open-sourced Qwen2-VL models, including Qwen2-VL-2B and Qwen2-VL-7B unde
| Benchmark | Previous SoTA<br><sup>(Open-source LVLM)<sup> | Claude-3.5 Sonnet | GPT-4o | **Qwen2-VL-72B**<br><sup>([🤗](https://huggingface.co/Qwen/Qwen2-VL-72B-Instruct) [🤖](https://modelscope.cn/models/qwen/Qwen2-VL-72B-Instruct) |**Qwen2-VL-7B**<br><sup>([🤗](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) [🤖](https://modelscope.cn/models/qwen/Qwen2-VL-7B-Instruct)) |**Qwen2-VL-2B**<br><sup>([🤗](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct)[🤖](https://modelscope.cn/models/qwen/Qwen2-VL-2B-Instruct))
| :--- | :---: | :---: | :---: | :---: |:---: |:---: |
| MMMU<sub>val</sub> | 58.3 | 68.3 | **69.1** | 64.5 | 54.1|41.1
| MMMU-Pro | 46.9 | 51.5 | **51.9** | 46.2 | 43.5 | 37.6
| DocVQA<sub>test</sub> | 94.1 | 95.2 | 92.8 | **96.5** | 94.5| 90.1
| InfoVQA<sub>test</sub> | 82.0 | - | - | **84.5** | 76.5|65.5
| ChartQA<sub>test</sub> | 88.4 | **90.8** | 85.7 | 88.3 |83.0| 73.5
Expand All @@ -74,7 +75,6 @@ We have open-sourced Qwen2-VL models, including Qwen2-VL-2B and Qwen2-VL-7B unde
| MathVista<sub>testmini</sub> | 67.5 | 67.7 | 63.8 | **70.5** |58.2| 43.0
| MathVision | 16.97 | - | **30.4** | 25.9 | 16.3| 12.4


### Video Benchmarks

| Benchmark | Previous SoTA<br><sup>(Open-source LVLM)<sup> | Gemini 1.5-Pro | GPT-4o | **Qwen2-VL-72B**<br><sup>([🤗](https://huggingface.co/Qwen/Qwen2-VL-72B-Instruct) [🤖](https://modelscope.cn/models/qwen/Qwen2-VL-72B-Instruct)) |**Qwen2-VL-7B**<br><sup>([🤗](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) [🤖](https://modelscope.cn/models/qwen/Qwen2-VL-7B-Instruct)) |**Qwen2-VL-2B**<br><sup>([🤗](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct)[🤖](https://modelscope.cn/models/qwen/Qwen2-VL-2B-Instruct))
Expand Down

0 comments on commit 3bf7dbd

Please sign in to comment.