[Feature]: Inquiry about Multi-modal Support in VLLM for MiniCPM-V2.6

[Dong148]

🚀 The feature, motivation and pitch

I am currently exploring the capabilities of the VLLM library and am interested in understanding its support for multi-modal inputs, particularly for models like MiniCPM-V2.6. I would like to know if VLLM is designed to handle multi-image and video inputs for such models.

Alternatives

1. Model of Interest: MiniCPM-V2.6
2. Types of Input: Multi-image and video
3. Current Understanding:
   • I have reviewed the documentation and initial examples provided with VLLM.

• It seems that both multiple 'image_url' input and list value in image_url is currently not supported.
• However, I am not sure if it supports the processing of multiple images or videos as input to a model like MiniCPM-V2.6.

Questions

1. Does VLLM support the integration of MiniCPM-V2.6 for processing multi-image and video inputs?
2. If yes, could you provide an example or a guide on how to set up and use this feature?
3. If not, are there any plans to extend VLLM's capabilities to support such inputs in the future?

Additional context

[DarkLight1337]

Multi-image input is currently supported for MiniCPU-V specifically (#7122), with some caveats:

• It only works in offline inference, not the OpenAI API-compatible server.
• Until the next release, you have to build from source (main branch) to use it.

We are actively working on extending the support for multi-image input - please refer to #4194 for details.

[Dong148]

Multi-image input is currently supported for MiniCPU-V specifically (#7122), with some caveats:

• It only works in offline inference, not the OpenAI API-compatible server.
• Until the next release, you have to build from source (main branch) to use it.

We are actively working on extending the support for multi-image input - please refer to #4194 for details.

Thank you for your assistance and for taking the time to help me out. I look forward to exploring more features of VLLM and potentially contributing to its development in the future.

[Patrick10203]

• It only works in offline inference, not the OpenAI API-compatible server.
• Until the next release, you have to build from source (main branch) to use it.

Are you sure that building the main branch supports multi image over the open ai api? Because the line https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/chat_utils.py#L179 is still in the main branch

[DarkLight1337]

• It only works in offline inference, not the OpenAI API-compatible server.
• Until the next release, you have to build from source (main branch) to use it.

Are you sure that building the main branch supports multi image over the open ai api? Because the line https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/chat_utils.py#L179 is still in the main branch

I was referring to multi-modal support for MiniCPM-V specifically, not for multi-modal models (+OpenAI server) in general.

[mogazheng]

The vllm OpenAI API-compatible server deployment of MiniCPM-V still does not support video understanding.

[DarkLight1337]

You should be able to pass a video via multi-image input now.

[syafiqmuda]

Hello, can I know the status on this? and can you give me some kind of doc or guide on how to use MiniCPM-V model?

[DarkLight1337]

Yes, it's supported. Please see examples in examples/offline_inference/vision_language.py

[syafiqmuda]

so for the online is still not supported yet?

[DarkLight1337]

Both online and offline inference are supported. Please check out the documentation for more details

[syafiqmuda]

Do I need to install certain vLLM version? because I see some of document specify to install:

pip install vllm==0.5.4

this doc were last update were on 27 Sep

[DarkLight1337]

Which documentation are you referring to? You should use the latest vLLM version

[syafiqmuda]

MiniCPM doc https://modelbest.feishu.cn/wiki/LZxLwp4Lzi29vXklYLFchwN5nCf