[Version: 4.49.0] Qwen2.5-VL is not supported in vLLM because of transformers

[usun1997]

System Info

error: Model architectures Qwen2 5 VLForConditionalGeneration' failed to be inspected. Please check the logs for more details.

right now people say that they are using methods like:

1. pip install --upgrade git+https://github.com/huggingface/transformers.git@336dc69d63d56f232a183a3e7f52790429b871ef ([Bug]: Qwen2.5-VL broke due to transformers upstream changes vllm-project/vllm#13285)
2. pip install --force-reinstall git+https://github.com/huggingface/transformers.git@9985d06add07a4cc691dc54a7e34f54205c04d40 ([Bug] ValueError: Model architectures ['Qwen2_5_VLForConditionalGeneration'] failed to be inspected. Please check the logs for more details. vllm-project/vllm#12932)
3. There is a breaking change in transformers dev. You need to update vLLM to latest dev and also redownload the HF model repo. ([Bug] ValueError: Model architectures ['Qwen2_5_VLForConditionalGeneration'] failed to be inspected. Please check the logs for more details. vllm-project/vllm#12932)

I mean. This is not supposed to be like this. I can't connect to your github without a vpn, and with a vpn, I can't connect to my workspace. Could transformers team just fix the problem instead of letting people solve it by some weird method. Thanks!

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

use Xinference, download the newest vLLM so that you get transformers 4.49.0. You download Qwen2.5 VL and deploy in vLLM and get the error:

Expected behavior

NO MORE ERROR DURING DEPLOYMENT OF QWEN2.5 VL 7B IN VLLM

[glamourzc]

me too.

[hayreenlee]

me too.

[yynickyeh]

me too.

[ppkliu]

mee too

[Rocketknight1]

cc @zucchini-nlp, but it would help a lot if someone could post the logs/error to help us figure out what's going on here!

[zucchini-nlp]

Hey all! The issue is being fixed on vllm side with vllm-project/vllm#13592 afaik, the team will check compatibility with v4.49 release

cc @Isotr0py

[Isotr0py]

That's because the qwen2.5-vl implementation in vllm 0.7.2 (the latest release, not latest commit) is still trying import Qwen2_5_VLImageProcessor which has been removed in transformers 4.49.0 release.

vLLM team is planning to make a new release including the corresponding fix (vllm-project/vllm#13286), perhaps today or tomorrow.

[ywang96]

Hello! We have released vllm==0.7.3 - Although vllm-project/vllm#13602 didn't make it to the release, we have confirmed that our docker image and PyPI package both have transformers==4.49.0 installed and should be compatible with it!

[igorpereirabr1]

Still not working after upgrate to vllm==0.7.3

[ywang96]

Still not working after upgrate to vllm==0.7.3

Can you try specifying the huggingface ID (e.g, Qwen/Qwen2.5-VL-3B-Instruct) directly?

[igorpereirabr1]

Still not working after upgrate to vllm==0.7.3

Can you try specifying the huggingface ID (e.g, Qwen/Qwen2.5-VL-3B-Instruct) directly?

Unfortunately, I can't.. my GPU cluster do not have access to huggingface.. but I've the latest version of the model stored in this path.. .

[usun1997]

Updates:

I managed to use the mentioned "weird" method to solve the case by using:

pip install --force-reinstall git+https://github.com/huggingface/transformers.git@9985d06add07a4cc691dc54a7e34f54205c04d40

Do you know why it can be solved? Transformers 4.49.0dev version is helpful. Here, the version 4.49.0 dev can support Qwen2.5-VL-7B, but "pip install transformers --upgrade" to get 4.49.0 version does not help.

[usun1997]

Hey all! The issue is being fixed on vllm side with vllm-project/vllm#13592 afaik, the team will check compatibility with v4.49 release

cc @Isotr0py

GOOD NEWS