[Bug]: minicpmv crash on 1x1 image

[sfyumi]

Your current environment

Model Input Dumps

No response

🐛 Describe the bug

The channel dimension is ambiguous. Got image shape (1, 1, 3). Assuming channels are the first dimension.
ERROR 09-29 21:38:19 image.py:48] Failed to process image ([<PIL.Image.Image image mode=RGB size=1x1 at 0x7FF3D44B3E80>, <PIL.Image.Image image mode=RGB size=1x1 at 0x7FF3D44B22C
0>, <PIL.Image.Image image mode=RGB size=918x1850 at 0x7FF3D44B2C20>])
ERROR 09-29 21:38:19 async_llm_engine.py:58] Engine background task failed
ERROR 09-29 21:38:19 async_llm_engine.py:58] Traceback (most recent call last):
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 48, in _log_task_completion
ERROR 09-29 21:38:19 async_llm_engine.py:58]     return_value = task.result()
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 733, in run_engine_loop
ERROR 09-29 21:38:19 async_llm_engine.py:58]     result = task.result()
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 673, in engine_step
ERROR 09-29 21:38:19 async_llm_engine.py:58]     request_outputs = await self.engine.step_async(virtual_engine)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 340, in step_async
ERROR 09-29 21:38:19 async_llm_engine.py:58]     outputs = await self.model_executor.execute_model_async(
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 185, in execute_model_async
ERROR 09-29 21:38:19 async_llm_engine.py:58]     output = await make_async(self.driver_worker.execute_model
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR 09-29 21:38:19 async_llm_engine.py:58]     result = self.fn(*self.args, **self.kwargs)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 303, in execute_model
ERROR 09-29 21:38:19 async_llm_engine.py:58]     inputs = self.prepare_input(execute_model_req)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 291, in prepare_input
ERROR 09-29 21:38:19 async_llm_engine.py:58]     return self._get_driver_input_and_broadcast(execute_model_req)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 253, in _get_driver_input_and_broadcast
ERROR 09-29 21:38:19 async_llm_engine.py:58]     self.model_runner.prepare_model_input(
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/embedding_model_runner.py", line 142, in prepare_model_input
ERROR 09-29 21:38:19 async_llm_engine.py:58]     model_input = self._prepare_model_input_tensors(
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1123, in _prepare_model_input_tensors
ERROR 09-29 21:38:19 async_llm_engine.py:58]     builder.add_seq_group(seq_group_metadata)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 720, in add_seq_group
ERROR 09-29 21:38:19 async_llm_engine.py:58]     per_seq_group_fn(inter_data, seq_group_metadata)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 658, in _compute_multi_modal_input
ERROR 09-29 21:38:19 async_llm_engine.py:58]     mm_kwargs = self.multi_modal_input_mapper(mm_data)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/multimodal/registry.py", line 126, in map_input
ERROR 09-29 21:38:19 async_llm_engine.py:58]     input_dict = plugin.map_input(model_config, data_value)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/multimodal/base.py", line 263, in map_input
ERROR 09-29 21:38:19 async_llm_engine.py:58]     return mapper(InputContext(model_config), data)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/multimodal/image.py", line 44, in _default_input_mapper
ERROR 09-29 21:38:19 async_llm_engine.py:58]     batch_data = image_processor \
ERROR 09-29 21:38:19 async_llm_engine.py:58]     result = self.fn(*self.args, **self.kwargs)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 303, in execute_model
ERROR 09-29 21:38:19 async_llm_engine.py:58]     inputs = self.prepare_input(execute_model_req)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 291, in prepare_input
ERROR 09-29 21:38:19 async_llm_engine.py:58]     return self._get_driver_input_and_broadcast(execute_model_req)
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 253, in _get_driver_input_and_broadcast
ERROR 09-29 21:38:19 async_llm_engine.py:58]     self.model_runner.prepare_model_input(
ERROR 09-29 21:38:19 async_llm_engine.py:58]   File "/usr/local/python3/lib/python3.10/site-packages/vllm/worker/embedding_model_runner.py", line 142, in prepare_model_input
ERROR 09-29 21:38:19 async_llm_engine.py:58]     model_input = self._prepare_model_input_tensors(

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Are you able to process the image using their image processor on HuggingFace? If the problem lies with their image processor, you should raise the issue to their repository instead of ours. I doubt that most image processors are designed to deal with this case though.

[sfyumi]

Are you able to process the image using their image processor on HuggingFace? If the problem lies with their image processor, you should raise the issue to their repository instead of ours. I doubt that most image processors are designed to deal with this case though.

I'll test it later. @DarkLight1337

[alex-jw-brooks]

Hey, thanks for the bug report! I went ahead and reproduced this error with minicpm 2.6. Some repro cases below:

VLLM - The minicpm preprocessing changed a little in the last week, so it's not using the default mapper anymore, but it's still wrapping the same HF image processor, so it doesn't matter for this case

from vllm import LLM, SamplingParams
import numpy as np
from PIL import Image

model_name = "openbmb/MiniCPM-V-2_6"
llm = LLM(
    model=model_name,
    dtype="half",
    trust_remote_code=True,
)
sampling_params = SamplingParams()

IMG_DIMS = [1, 1, 3]

# Make a random image of the specified dims
img = Image.fromarray((np.random.rand(*IMG_DIMS) * 255).astype('uint8'))

outputs = llm.generate(
    {
        "prompt": "<image>./</image>",
        "multi_modal_data": {"image": img}
    },
    sampling_params=sampling_params
)

In transformers, it's pretty much equivalent to this:

import transformers
import numpy as np
from PIL import Image

model_name = "openbmb/MiniCPM-V-2_6"
proc = transformers.AutoImageProcessor.from_pretrained(model_name, trust_remote_code=True)
IMG_DIMS = [1, 1, 3]

# Make a random image of the specified dims
img = Image.fromarray((np.random.rand(*IMG_DIMS) * 255).astype('uint8'))
proc.preprocess(img, return_tensors="pt")

Running both gives ValueError: mean must have 448 elements if it is an iterable, got 3, so it's not a VLLM issue.

I did open a PR to tweak the logging error for the default mapper a bit here, although it won't be logged for this specific model now that there is a custom mapper

[sfyumi]

When the vllm server encounters this error, the system stops processing new requests. I think vllm should ignore this error and processes new reqeusts normally.

[DarkLight1337]

@njhill how can we let LLM engine recover from such errors?

[sfyumi]

#9141
This pull request may fix this bug.

[njhill]

@njhill how can we let LLM engine recover from such errors?

@DarkLight1337 I was looking into this in general before, I will try to get back to it. There are some classes of errors where we don't need to fail the whole server and some cases not even the whole batch.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!