[Bug]: AttributeError: module 'cv2.dnn' has no attribute 'DictValue'

[eyuansu62]

Your current environment

Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.2
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-26-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A800-SXM4-80GB
GPU 1: NVIDIA A800-SXM4-80GB
GPU 2: NVIDIA A800-SXM4-80GB
GPU 3: NVIDIA A800-SXM4-80GB
GPU 4: NVIDIA A800-SXM4-80GB
GPU 5: NVIDIA A800-SXM4-80GB
GPU 6: NVIDIA A800-SXM4-80GB
GPU 7: NVIDIA A800-SXM4-80GB

Nvidia driver version: 535.154.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 57 bits virtual
Byte Order:                      Little Endian
CPU(s):                          128
On-line CPU(s) list:             0-127
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
CPU family:                      6
Model:                           106
Thread(s) per core:              2
Core(s) per socket:              32
Socket(s):                       2
Stepping:                        6
Frequency boost:                 enabled
CPU max MHz:                     3400.0000
CPU min MHz:                     800.0000
BogoMIPS:                        5200.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear pconfig flush_l1d arch_capabilities
Virtualization:                  VT-x
L1d cache:                       3 MiB (64 instances)
L1i cache:                       2 MiB (64 instances)
L2 cache:                        80 MiB (64 instances)
L3 cache:                        96 MiB (2 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-31,64-95
NUMA node1 CPU(s):               32-63,96-127
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] mypy==1.10.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cudnn-frontend==1.3.0
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-dali-cuda120==1.37.1
[pip3] nvidia-ml-py==12.555.43
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvimgcodec-cu12==0.2.0.7
[pip3] nvidia-nvjitlink-cu12==12.6.68
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] nvidia-pyindex==1.0.9
[pip3] onnx==1.16.0
[pip3] optree==0.11.0
[pip3] pytorch-quantization==2.1.2
[pip3] pytorch-triton==3.0.0+989adb9a2
[pip3] pyzmq==26.0.3
[pip3] sentence-transformers==3.0.0
[pip3] torch==2.4.0
[pip3] torch-tensorrt==2.4.0a0
[pip3] torchvision==0.19.0
[pip3] transformers==4.44.2
[pip3] triton==3.0.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0           py310h7f8727e_0
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0
[conda] mkl_random                1.2.2           py310h00e6091_0
[conda] numpy                     1.23.5          py310hd5efca6_0
[conda] numpy-base                1.23.5          py310h8e6c178_0
[conda] numpydoc                  1.5.0           py310h06a4308_0
[conda] pytorch                   1.12.1          cpu_py310hb1f1ab4_1
[conda] pyzmq                     23.2.0          py310h6a678d5_0
[conda] transformers              4.24.0          py310h06a4308_0
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: 5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	NIC8	NIC9	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV8	NV8	NV8	NV8	NV8	NV8	NV8	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	0-31,64-95	0		N/A
GPU1	NV8	 X 	NV8	NV8	NV8	NV8	NV8	NV8	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	0-31,64-95	0		N/A
GPU2	NV8	NV8	 X 	NV8	NV8	NV8	NV8	NV8	SYS	SYS	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	0-31,64-95	0		N/A
GPU3	NV8	NV8	NV8	 X 	NV8	NV8	NV8	NV8	SYS	SYS	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	0-31,64-95	0		N/A
GPU4	NV8	NV8	NV8	NV8	 X 	NV8	NV8	NV8	SYS	SYS	SYS	SYS	SYS	SYS	PXB	PXB	SYS	SYS	32-63,96-127	1		N/A
GPU5	NV8	NV8	NV8	NV8	NV8	 X 	NV8	NV8	SYS	SYS	SYS	SYS	SYS	SYS	PXB	PXB	SYS	SYS	32-63,96-127	1		N/A
GPU6	NV8	NV8	NV8	NV8	NV8	NV8	 X 	NV8	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PXB	PXB	32-63,96-127	1		N/A
GPU7	NV8	NV8	NV8	NV8	NV8	NV8	NV8	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PXB	PXB	32-63,96-127	1		N/A
NIC0	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC1	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	PIX	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC2	SYS	SYS	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PIX	SYS	SYS	SYS	SYS	SYS	SYS
NIC3	SYS	SYS	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	PIX	 X 	SYS	SYS	SYS	SYS	SYS	SYS
NIC4	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PIX	SYS	SYS	SYS	SYS
NIC5	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	 X 	SYS	SYS	SYS	SYS
NIC6	SYS	SYS	SYS	SYS	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PIX	SYS	SYS
NIC7	SYS	SYS	SYS	SYS	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	 X 	SYS	SYS
NIC8	SYS	SYS	SYS	SYS	SYS	SYS	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PXB
NIC9	SYS	SYS	SYS	SYS	SYS	SYS	PXB	PXB	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PXB	 X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9

Model Input Dumps

No response

🐛 Describe the bug

$ vllm serve /share/project/huggingface/models/DeepSeek-V2.5 --tensor-parallel-size 8 --trust-remote-code --gpu_memory_utilization 0.9 --max_model_len 8192 --enable_chunked_prefill
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 5, in <module>
    from vllm.scripts import main
  File "/usr/local/lib/python3.10/dist-packages/vllm/__init__.py", line 4, in <module>
    from vllm.engine.async_llm_engine import AsyncLLMEngine
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 15, in <module>
    from vllm.engine.llm_engine import (DecoderPromptComponents, LLMEngine,
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 24, in <module>
    from vllm.engine.output_processor.interfaces import (
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/output_processor/interfaces.py", line 6, in <module>
    from vllm.engine.output_processor.stop_checker import StopChecker
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/output_processor/stop_checker.py", line 6, in <module>
    from vllm.transformers_utils.tokenizer import AnyTokenizer
  File "/usr/local/lib/python3.10/dist-packages/vllm/transformers_utils/tokenizer.py", line 13, in <module>
    from vllm.transformers_utils.tokenizers import (BaichuanTokenizer,
  File "/usr/local/lib/python3.10/dist-packages/vllm/transformers_utils/tokenizers/__init__.py", line 2, in <module>
    from vllm.transformers_utils.tokenizers.mistral import MistralTokenizer
  File "/usr/local/lib/python3.10/dist-packages/vllm/transformers_utils/tokenizers/mistral.py", line 9, in <module>
    from mistral_common.tokens.tokenizers.mistral import ChatCompletionRequest
  File "/usr/local/lib/python3.10/dist-packages/mistral_common/tokens/tokenizers/mistral.py", line 32, in <module>
    from mistral_common.tokens.tokenizers.multimodal import (
  File "/usr/local/lib/python3.10/dist-packages/mistral_common/tokens/tokenizers/multimodal.py", line 6, in <module>
    import cv2
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 175, in bootstrap
    if __load_extra_py_code_for_module("cv2", submodule, DEBUG):
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 28, in __load_extra_py_code_for_module
    py_module = importlib.import_module(module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py", line 171, in <module>
    LayerId = cv2.dnn.DictValue
AttributeError: module 'cv2.dnn' has no attribute 'DictValue'

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[hzhua]

I also met this issue. I fixed this by
pip install opencv-python-headless==4.5.4.58

[ywang96]

@patrickvonplaten Sorry for pinging you directly again please take a look at this dependency issue. Thanks!

[patrickvonplaten]

Uff yeah this is a long-standing opencv issue - referring to opencv/opencv-python#884 here.

Solutions:

• 1.) You don't know what's going on in your env, you don't care and want an easy fix. Follow what @hzhua says here or this

• 2.) You already have a opencv-python or opencv in your PyPI environment that you don't need, but you would like to install the last opencv package from mistral common. Then do:

pip uninstall opencv-python
pip uninstall opencv
pip install --upgrade mistral_common

• 3.) You already have a opencv-python or opencv in your PyPI environment that you do need. In this case just uninstall the just installed headless pypi:

pip uninstall opencv-python-headless

Also it might be good to leave a comment or bump opencv/opencv-python#884 so that opencv sees that more people are struggling with the opencv package inconsistencies & so that it might be fixed globally faster ;-)

[patrickvonplaten]

@ywang96 I'm not sure we can do much here sadly as this can happen whenever mistral_common is installed in a package that has a misconfigured, already existing opencv package. If you install vllm into a clean env this can't happen.

What do you think? To me advertising the solutions as written here: #8650 (comment) is the best thing we can do

[youkaichao]

@patrickvonplaten is it possible to lazily import cv2 from mistral_common side?

many users don't use the vision part, they may just use text LLM. it does not make sense to bother them by opencv.

[patrickvonplaten]

That makes sense - we'll do a patch release tomorrow with this PR: mistralai/mistral-common#56 so that cv2 is not automatically installed anymore

[patrickvonplaten]

Patch release 1.4.3 to make cv2 install optional is out: https://github.com/mistralai/mistral-common/releases/tag/v1.4.3

[floschne]

Hi all,

Unfortunately, I think not installing cv2 per default leads to an issue when using the vLLM docker containers with multimodal models (e.g. Pixtral).

Here's an excerpt of the error log.

vllm-1  | INFO 09-30 05:18:03 model_runner.py:1025] Loading model weights took 23.6552 GB
vllm-1  | WARNING 09-30 05:18:04 model_runner.py:1196] Computed max_num_seqs (min(256, 32768 // 40960)) to be less than 1. Setting it to the minimum value of 1.
vllm-1  | Process SpawnProcess-1:
vllm-1  | Traceback (most recent call last):
vllm-1  |   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
vllm-1  |     self.run()
vllm-1  |   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
vllm-1  |     self._target(*self._args, **self._kwargs)
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 388, in run_mp_engine
vllm-1  |     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
vllm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 138, in from_engine_args
vllm-1  |     return cls(
vllm-1  |            ^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 78, in __init__
vllm-1  |     self.engine = LLMEngine(*args,
vllm-1  |                   ^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 339, in __init__
vllm-1  |     self._initialize_kv_caches()
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 474, in _initialize_kv_caches
vllm-1  |     self.model_executor.determine_num_available_blocks())
vllm-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/gpu_executor.py", line 114, in determine_num_available_blocks
vllm-1  |     return self.driver_worker.determine_num_available_blocks()
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
vllm-1  |     return func(*args, **kwargs)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 223, in determine_num_available_blocks
vllm-1  |     self.model_runner.profile_run()
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
vllm-1  |     return func(*args, **kwargs)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1228, in profile_run
vllm-1  |     model_input = self.prepare_model_input(
vllm-1  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1519, in prepare_model_input
vllm-1  |     model_input = self._prepare_model_input_tensors(
vllm-1  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1141, in _prepare_model_input_tensors
vllm-1  |     builder.add_seq_group(seq_group_metadata)
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 728, in add_seq_group
vllm-1  |     per_seq_group_fn(inter_data, seq_group_metadata)
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 660, in _compute_multi_modal_input
vllm-1  |     mm_kwargs = self.multi_modal_input_mapper(mm_data)
vllm-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/registry.py", line 126, in map_input
vllm-1  |     input_dict = plugin.map_input(model_config, data_value)
vllm-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/base.py", line 279, in map_input
vllm-1  |     return mapper(InputContext(model_config), data, **mm_processor_kwargs)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/pixtral.py", line 96, in input_mapper_for_pixtral
vllm-1  |     encoding = tokenizer.instruct.mm_encoder(image)
vllm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/mistral_common/tokens/tokenizers/multimodal.py", line 142, in __call__
vllm-1  |     processed_image = transform_image(image, new_image_size)
vllm-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/mistral_common/tokens/tokenizers/multimodal.py", line 96, in transform_image
vllm-1  |     raise ImportError("OpenCV is required for this function. Install it with 'pip install mistral_common[opencv]'")
vllm-1  | ImportError: OpenCV is required for this function. Install it with 'pip install mistral_common[opencv]'
vllm-1  | Traceback (most recent call last):
vllm-1  |   File "<frozen runpy>", line 198, in _run_module_as_main
vllm-1  |   File "<frozen runpy>", line 88, in _run_code
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 571, in <module>
vllm-1  |     uvloop.run(run_server(args))
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
vllm-1  |     return __asyncio.run(
vllm-1  |            ^^^^^^^^^^^^^^
vllm-1  |   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
vllm-1  |     return runner.run(main)
vllm-1  |            ^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
vllm-1  |     return self._loop.run_until_complete(task)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
vllm-1  |     return await main
vllm-1  |            ^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 538, in run_server
vllm-1  |     async with build_async_engine_client(args) as engine_client:
vllm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
vllm-1  |     return await anext(self.gen)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 105, in build_async_engine_client
vllm-1  |     async with build_async_engine_client_from_engine_args(
vllm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
vllm-1  |     return await anext(self.gen)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 192, in build_async_engine_client_from_engine_args
vllm-1  |     raise RuntimeError(
vllm-1  | RuntimeError: Engine process failed to start
vllm-1 exited with code 0

And here is the respective docker-compose.yaml:

services:
  vllm:
    image: vllm/vllm-openai:latest
    entrypoint: python3
    command: "-m vllm.entrypoints.openai.api_server --port=8000 --host=0.0.0.0 --model mistralai/Pixtral-12B-2409 --limit-mm-per-prompt 'image=10' --max-model-len 32768 --tokenizer-mode mistral --load-format mistral --config-format mistral"
    env_file:
      - .env
    ports:
      - "${VLLM_EXPOSED}:8000"
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}
      - LOG_LEVEL=DEBUG
    volumes:
      - ./cache:/workspace/.cache
      - ./templates:/workspace/templates
    restart: always
    shm_size: "64gb"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://0.0.0.0:8000/v1/models"]
      interval: 30s
      timeout: 5s
      retries: 20
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

[ywang96]

This is now fixed by #8951 but you’ll need to wait for our next release.