[Performance]: InternVL multi image speed is not improved compare to original

[luohao123]

Your current environment

The output of `python collect_env.py`

latest vllm 0.6.1

Model Input Dumps

tt

🐛 Describe the bug

InternVL multi image speed is slower than original

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Could you elaborate more? What do you mean by the original speed?

[luohao123]

Compare with torch, same device, same dtype (float16, V100). (torch means hf with flashattn default)

Single image faster about 20%, while multiple image are slower, A100 got same result.

[DarkLight1337]

Can you show the scripts you used to measure the performance of HF vs vLLM?

[luohao123]

Hi, the test based on internvl 8b model, have u guys tested vllm speed improvement on multiple images? I am not lying. Multiple images actually slower than torch. For some inhouse issue, I didn't got a chance to paste code here, but I think you guys might can be easily replicate the result.

[DarkLight1337]

Hi, the test based on internvl 8b model, have u guys tested vllm speed improvement on multiple images? I am not lying. Multiple images actually slower than torch. For some inhouse issue, I didn't got a chance to paste code here, but I think you guys might can be easily replicate the result.

No, we have not tested the speed for multiple images (benchmarking work for multi-modal models is still in the early stages). Since vLLM was originally designed around language generation, most of vLLM's optimizations don't currently work on the vision encoder part of the model, which may explain the decrease in speed when more images are passed. There may also be CPU bottlenecks associated with image preprocessing.

We are still busy with making multi-modal support feature complete, so it may take a while before we can focus on optimization - any help is welcome!

[luohao123]

Hi, am not expert in accelerate, but as far as I can understand, why encoder-decoder can not use flashattn?

[luohao123]

How about using flashatten2 package, or using torch inside sdpa?

[Jeremy-J-J]

Same problem

[torinchen]

noooop

when diving into the code, i found that internvl2 uses xformer's attention not the naive one, so, the slow speed maybe come from other part.