[Bug][V1]: Kernel crashed when running qwen2.5_vl model

[dashanji]

Your current environment

The output of `python collect_env.py`

Your output of `python collect_env.py` here

🐛 Describe the bug

I'm using the vllm 0.7.3 (enable_vllm_v1) to run qwen2_5_vl model.

Initializing a V1 LLM engine (v0.7.3) with config: model='/qwen2_5-vl-72b', speculative_config=None, tokenizer='/qwen2_5-vl-72b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32000, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/qwen2_5-vl-72b, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}

Then I run it with a trace. It works fine for a while (maybe around 100 requests succeed), but after that, the cuda kernel crashed. It's reproducible when re-run the same trace.

../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [1,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374] WorkerProc hit an exception: %s
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374] Traceback (most recent call last):
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in worker_busy_loop
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     output = func(*args, **kwargs)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     return func(*args, **kwargs)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 227, in execute_model
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     output = self.model_runner.execute_model(scheduler_output)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     return func(*args, **kwargs)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 957, in execute_model
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     sampler_output = self.model.sample(
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1091, in sample
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     return self.language_model.sample(logits, sampling_metadata)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 505, in sample
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     next_tokens = self.sampler(logits, sampling_metadata)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     return forward_call(*args, **kwargs)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 55, in forward
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     sampled = self.sample(logits, sampling_metadata)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 111, in sample
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     random_sampled = self.topk_topp_sampler(
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     return forward_call(*args, **kwargs)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 62, in forward_native
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     logits = apply_top_k_top_p(logits, k, p)
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]   File "/usr/local/python-3.10.14/lib/python3.10/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 100, in apply_top_k_top_p
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374]     top_k_mask = logits_sort.gather(1, top_k_mask.unsqueeze(dim=1))
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374] RuntimeError: CUDA error: device-side assert triggered
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(VllmWorker rank=3 pid=271990) ERROR 03-04 01:21:00 multiproc_executor.py:374] 
../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [1,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of 
bounds"../aten/src/ATen/native/cuda/ScatterGatherKernel.cu` failed.
:144../aten/src/ATen/native/cuda/ScatterGatherKernel.cu: operator():144: block: [0: operator(),0: block: [0,0,0], thread: [0,0,0], thread: [1,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds",0` failed.
] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"../aten/src/ATen/native/cuda/ScatterGatherKernel.cu` failed.
:144: operator(): block: [0,0,0], thread: [1,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f551c56c446 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f551c5166e4 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f551d0a5a18 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f54cdfc7726 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f54cdfcc3f0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f54cdfd3b5a in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f54cdfd561d in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x145c0 (0x7f551d4c95c0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: <unknown function> + 0x93fb (0x7f551ebd63fb in /usr/lib64/libpthread.so.0)
frame #9: clone + 0x43 (0x7f551e51be83 in /usr/lib64/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
[rank2]:[E304 01:21:01.867394363 ProcessGroupNCCL.cpp:1595] [PG ID 2 PG GUID 3 Rank 2] Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f9272b6c446 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9272b166e4 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f92732a5a18 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f92241c7726 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f92241cc3f0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f92241d3b5a in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f92241d561d in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x145c0 (0x7f9274e225c0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: <unknown function> + 0x93fb (0x7f9274ebc3fb in /usr/lib64/libpthread.so.0)
frame #9: clone + 0x43 (0x7f927471be83 in /usr/lib64/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
[rank3]:[E304 01:21:01.867443735 ProcessGroupNCCL.cpp:1595] [PG ID 2 PG GUID 3 Rank 3] Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f782c96c446 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f782c9166e4 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f782d4a5a18 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f77de3c7726 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f77de3cc3f0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f77de3d3b5a in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f77de3d561d in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x145c0 (0x7f782d86f5c0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: <unknown function> + 0x93fb (0x7f782ebbb3fb in /usr/lib64/libpthread.so.0)
frame #9: clone + 0x43 (0x7f782e8c1e83 in /usr/lib64/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG ID 2 PG GUID 3 Rank 0] Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f551c56c446 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f551c5166e4 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f551d0a5a18 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f54cdfc7726 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f54cdfcc3f0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f54cdfd3b5a in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f54cdfd561d in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x145c0 (0x7f551d4c95c0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: <unknown function> + 0x93fb (0x7f551ebd63fb in /usr/lib64/libpthread.so.0)
frame #9: clone + 0x43 (0x7f551e51be83 in /usr/lib64/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f551c56c446 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe4271b (0x7f54cdc4271b in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x145c0 (0x7f551d4c95c0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x93fb (0x7f551ebd63fb in /usr/lib64/libpthread.so.0)
frame #4: clone + 0x43 (0x7f551e51be83 in /usr/lib64/libc.so.6)

  what():  [PG ID 2 PG GUID 3 Rank 2] Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f9272b6c446 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9272b166e4 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f92732a5a18 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f92241c7726 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f92241cc3f0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f92241d3b5a in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f92241d561d in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0x145c0 (0x7f9274e225c0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: <unknown function> + 0x93fb (0x7f9274ebc3fb in /usr/lib64/libpthread.so.0)
frame #9: clone + 0x43 (0x7f927471be83 in /usr/lib64/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f9272b6c446 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe4271b (0x7f9223e4271b in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x145c0 (0x7f9274e225c0 in /usr/local/python-3.10.14/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x93fb (0x7f9274ebc3fb in /usr/lib64/libpthread.so.0)
frame #4: clone + 0x43 (0x7f927471be83 in /usr/lib64/libc.so.6)

  what():  [PG ID 2 PG GUID 3 Rank 3] Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Looks like a problem with the sampler. cc @robertgshaw2-redhat @njhill

[DarkLight1337]

When you start up the model, do you get any warning like this?

INFO 03-03 12:14:20 [gpu_model_runner.py:1216] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
Token indices sequence length is longer than the specified maximum sequence length for this model (65536 > 32768). Running this sequence through the model will result in indexing errors

[dashanji]

@DarkLight1337 No, the complete log is as follows.

INFO 03-03 20:35:11 core.py:50] Initializing a V1 LLM engine (v0.7.3) with config: model='/qwen2.5-vl', speculative_config=None, tokenizer='/qwen2.5-vl', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32000, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/qwen2.5-vl, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 03-03 20:35:11 multiproc_worker_utils.py:300] Reducing Torch parallelism from 90 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 03-03 20:35:11 custom_cache_manager.py:19] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 03-03 20:35:11 shm_broadcast.py:258] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0, 1, 2, 3], buffer_handle=(4, 10485760, 10, 'psm_96166cd4'), local_subscribe_port=49709, remote_subscribe_port=None)
INFO 03-03 20:35:15 __init__.py:207] Automatically detected platform cuda.
WARNING 03-03 20:35:16 utils.py:2262] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,list_loras,load_config,pin_lora,remove_lora,scheduler_config not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f5a9afcdd20>
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:16 shm_broadcast.py:258] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_b7bf1f0b'), local_subscribe_port=58587, remote_subscribe_port=None)
INFO 03-03 20:35:20 __init__.py:207] Automatically detected platform cuda.
WARNING 03-03 20:35:22 utils.py:2262] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,list_loras,load_config,pin_lora,remove_lora,scheduler_config not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fced23f9d20>
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:22 shm_broadcast.py:258] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_dbab7036'), local_subscribe_port=59107, remote_subscribe_port=None)
INFO 03-03 20:35:26 __init__.py:207] Automatically detected platform cuda.
WARNING 03-03 20:35:27 utils.py:2262] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,list_loras,load_config,pin_lora,remove_lora,scheduler_config not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f7c627edde0>
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:27 shm_broadcast.py:258] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_e95704e5'), local_subscribe_port=45577, remote_subscribe_port=None)
INFO 03-03 20:35:31 __init__.py:207] Automatically detected platform cuda.
WARNING 03-03 20:35:33 utils.py:2262] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,list_loras,load_config,pin_lora,remove_lora,scheduler_config not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fee52bbdd50>
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:33 shm_broadcast.py:258] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_5b0606ce'), local_subscribe_port=49049, remote_subscribe_port=None)
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:33 utils.py:916] Found nccl from library libnccl.so.2
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:33 utils.py:916] Found nccl from library libnccl.so.2
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:33 utils.py:916] Found nccl from library libnccl.so.2
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:33 utils.py:916] Found nccl from library libnccl.so.2
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:33 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:33 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:33 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:33 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:35 custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3.json
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:35 custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3.json
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:35 custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3.json
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:35 custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3.json
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:35 shm_broadcast.py:258] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer_handle=(3, 4194304, 6, 'psm_1fe4aed1'), local_subscribe_port=58735, remote_subscribe_port=None)
(VllmWorker rank=3 pid=231003) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(VllmWorker rank=1 pid=230932) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(VllmWorker rank=2 pid=230967) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(VllmWorker rank=0 pid=230902) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:36 gpu_model_runner.py:1049] Starting to load model /qwen2.5_vl...
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:36 gpu_model_runner.py:1049] Starting to load model /qwen2.5_vl....
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:36 gpu_model_runner.py:1049] Starting to load model /qwen2.5_vl....
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:36 gpu_model_runner.py:1049] Starting to load model /qwen2.5_vl...
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:36 config.py:3054] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 264, 272, 280, 288, 296, 304, 312, 320, 328, 336, 344, 352, 360, 368, 376, 384, 392, 400, 408, 416, 424, 432, 440, 448, 456, 464, 472, 480, 488, 496, 504, 512] is overridden by config [512, 384, 256, 128, 4, 2, 1, 392, 264, 136, 8, 400, 272, 144, 16, 408, 280, 152, 24, 416, 288, 160, 32, 424, 296, 168, 40, 432, 304, 176, 48, 440, 312, 184, 56, 448, 320, 192, 64, 456, 328, 200, 72, 464, 336, 208, 80, 472, 344, 216, 88, 120, 480, 352, 248, 224, 96, 488, 504, 360, 232, 104, 496, 368, 240, 112, 376]
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:36 config.py:3054] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 264, 272, 280, 288, 296, 304, 312, 320, 328, 336, 344, 352, 360, 368, 376, 384, 392, 400, 408, 416, 424, 432, 440, 448, 456, 464, 472, 480, 488, 496, 504, 512] is overridden by config [512, 384, 256, 128, 4, 2, 1, 392, 264, 136, 8, 400, 272, 144, 16, 408, 280, 152, 24, 416, 288, 160, 32, 424, 296, 168, 40, 432, 304, 176, 48, 440, 312, 184, 56, 448, 320, 192, 64, 456, 328, 200, 72, 464, 336, 208, 80, 472, 344, 216, 88, 120, 480, 352, 248, 224, 96, 488, 504, 360, 232, 104, 496, 368, 240, 112, 376]
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:36 config.py:3054] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 264, 272, 280, 288, 296, 304, 312, 320, 328, 336, 344, 352, 360, 368, 376, 384, 392, 400, 408, 416, 424, 432, 440, 448, 456, 464, 472, 480, 488, 496, 504, 512] is overridden by config [512, 384, 256, 128, 4, 2, 1, 392, 264, 136, 8, 400, 272, 144, 16, 408, 280, 152, 24, 416, 288, 160, 32, 424, 296, 168, 40, 432, 304, 176, 48, 440, 312, 184, 56, 448, 320, 192, 64, 456, 328, 200, 72, 464, 336, 208, 80, 472, 344, 216, 88, 120, 480, 352, 248, 224, 96, 488, 504, 360, 232, 104, 496, 368, 240, 112, 376]
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:36 config.py:3054] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 264, 272, 280, 288, 296, 304, 312, 320, 328, 336, 344, 352, 360, 368, 376, 384, 392, 400, 408, 416, 424, 432, 440, 448, 456, 464, 472, 480, 488, 496, 504, 512] is overridden by config [512, 384, 256, 128, 4, 2, 1, 392, 264, 136, 8, 400, 272, 144, 16, 408, 280, 152, 24, 416, 288, 160, 32, 424, 296, 168, 40, 432, 304, 176, 48, 440, 312, 184, 56, 448, 320, 192, 64, 456, 328, 200, 72, 464, 336, 208, 80, 472, 344, 216, 88, 120, 480, 352, 248, 224, 96, 488, 504, 360, 232, 104, 496, 368, 240, 112, 376]
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:36 cuda.py:157] Using Flash Attention backend on V1 engine.
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:36 cuda.py:157] Using Flash Attention backend on V1 engine.
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:36 cuda.py:157] Using Flash Attention backend on V1 engine.
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:36 cuda.py:157] Using Flash Attention backend on V1 engine.
(VllmWorker rank=3 pid=231003) WARNING 03-03 20:35:36 topk_topp_sampler.py:46] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=3 pid=231003) WARNING 03-03 20:35:36 rejection_sampler.py:47] FlashInfer is not available. Falling back to the PyTorch-native implementation of rejection sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=2 pid=230967) WARNING 03-03 20:35:36 topk_topp_sampler.py:46] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=2 pid=230967) WARNING 03-03 20:35:36 rejection_sampler.py:47] FlashInfer is not available. Falling back to the PyTorch-native implementation of rejection sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=0 pid=230902) WARNING 03-03 20:35:36 topk_topp_sampler.py:46] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=0 pid=230902) WARNING 03-03 20:35:36 rejection_sampler.py:47] FlashInfer is not available. Falling back to the PyTorch-native implementation of rejection sampling. For the best performance, please install FlashInfer.
Loading safetensors checkpoint shards:   0% Completed | 0/38 [00:00<?, ?it/s]
(VllmWorker rank=1 pid=230932) WARNING 03-03 20:35:36 topk_topp_sampler.py:46] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=1 pid=230932) WARNING 03-03 20:35:36 rejection_sampler.py:47] FlashInfer is not available. Falling back to the PyTorch-native implementation of rejection sampling. For the best performance, please install FlashInfer.
Loading safetensors checkpoint shards:   3% Completed | 1/38 [00:00<00:09,  3.93it/s]
Loading safetensors checkpoint shards:   5% Completed | 2/38 [00:00<00:10,  3.46it/s]
Loading safetensors checkpoint shards:   8% Completed | 3/38 [00:00<00:09,  3.53it/s]
Loading safetensors checkpoint shards:  11% Completed | 4/38 [00:01<00:09,  3.56it/s]
Loading safetensors checkpoint shards:  13% Completed | 5/38 [00:01<00:09,  3.60it/s]
Loading safetensors checkpoint shards:  16% Completed | 6/38 [00:01<00:08,  3.63it/s]
Loading safetensors checkpoint shards:  18% Completed | 7/38 [00:01<00:08,  3.60it/s]
Loading safetensors checkpoint shards:  21% Completed | 8/38 [00:02<00:08,  3.49it/s]
Loading safetensors checkpoint shards:  24% Completed | 9/38 [00:02<00:08,  3.56it/s]
Loading safetensors checkpoint shards:  26% Completed | 10/38 [00:02<00:06,  4.30it/s]
Loading safetensors checkpoint shards:  29% Completed | 11/38 [00:02<00:06,  4.15it/s]
Loading safetensors checkpoint shards:  32% Completed | 12/38 [00:03<00:06,  4.00it/s]
Loading safetensors checkpoint shards:  34% Completed | 13/38 [00:03<00:06,  3.92it/s]
Loading safetensors checkpoint shards:  37% Completed | 14/38 [00:03<00:05,  4.19it/s]
Loading safetensors checkpoint shards:  39% Completed | 15/38 [00:03<00:05,  3.88it/s]
Loading safetensors checkpoint shards:  42% Completed | 16/38 [00:04<00:05,  3.77it/s]
Loading safetensors checkpoint shards:  45% Completed | 17/38 [00:04<00:05,  3.71it/s]
Loading safetensors checkpoint shards:  47% Completed | 18/38 [00:04<00:05,  3.68it/s]
Loading safetensors checkpoint shards:  50% Completed | 19/38 [00:05<00:05,  3.66it/s]
Loading safetensors checkpoint shards:  53% Completed | 20/38 [00:05<00:05,  3.51it/s]
Loading safetensors checkpoint shards:  55% Completed | 21/38 [00:05<00:04,  3.50it/s]
Loading safetensors checkpoint shards:  58% Completed | 22/38 [00:05<00:04,  3.40it/s]
Loading safetensors checkpoint shards:  61% Completed | 23/38 [00:06<00:04,  3.49it/s]
Loading safetensors checkpoint shards:  63% Completed | 24/38 [00:06<00:03,  3.56it/s]
Loading safetensors checkpoint shards:  66% Completed | 25/38 [00:06<00:03,  3.60it/s]
Loading safetensors checkpoint shards:  68% Completed | 26/38 [00:07<00:03,  3.79it/s]
Loading safetensors checkpoint shards:  71% Completed | 27/38 [00:07<00:02,  3.79it/s]
Loading safetensors checkpoint shards:  74% Completed | 28/38 [00:07<00:02,  3.75it/s]
Loading safetensors checkpoint shards:  76% Completed | 29/38 [00:07<00:02,  3.72it/s]
Loading safetensors checkpoint shards:  79% Completed | 30/38 [00:08<00:02,  3.72it/s]
Loading safetensors checkpoint shards:  82% Completed | 31/38 [00:08<00:01,  3.55it/s]
Loading safetensors checkpoint shards:  84% Completed | 32/38 [00:08<00:01,  3.55it/s]
Loading safetensors checkpoint shards:  87% Completed | 33/38 [00:08<00:01,  3.55it/s]
Loading safetensors checkpoint shards:  89% Completed | 34/38 [00:09<00:01,  3.59it/s]
Loading safetensors checkpoint shards:  92% Completed | 35/38 [00:09<00:00,  3.47it/s]
Loading safetensors checkpoint shards:  95% Completed | 36/38 [00:09<00:00,  3.36it/s]
Loading safetensors checkpoint shards:  97% Completed | 37/38 [00:10<00:00,  3.40it/s]
Loading safetensors checkpoint shards: 100% Completed | 38/38 [00:10<00:00,  3.46it/s]
Loading safetensors checkpoint shards: 100% Completed | 38/38 [00:10<00:00,  3.64it/s]
(VllmWorker rank=0 pid=230902) 
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:47 gpu_model_runner.py:1060] Loading model weights took 17.7752 GB
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:47 gpu_model_runner.py:1060] Loading model weights took 17.7752 GB
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:47 gpu_model_runner.py:1060] Loading model weights took 17.7752 GB
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:47 gpu_model_runner.py:1060] Loading model weights took 17.7752 GB
(VllmWorker rank=1 pid=230932) INFO 03-03 20:35:47 gpu_model_runner.py:1226] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 512 video items of the maximum feature size.
(VllmWorker rank=3 pid=231003) INFO 03-03 20:35:47 gpu_model_runner.py:1226] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 512 video items of the maximum feature size.
(VllmWorker rank=0 pid=230902) INFO 03-03 20:35:47 gpu_model_runner.py:1226] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 512 video items of the maximum feature size.
(VllmWorker rank=2 pid=230967) INFO 03-03 20:35:47 gpu_model_runner.py:1226] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 512 video items of the maximum feature size.
(VllmWorker rank=1 pid=230932) It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
(VllmWorker rank=2 pid=230967) It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
(VllmWorker rank=0 pid=230902) It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
(VllmWorker rank=3 pid=231003) It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
(VllmWorker rank=0 pid=230902) INFO 03-03 20:36:16 backends.py:408] Using cache directory: /root/.cache/vllm/torch_compile_cache/d4afb83389/rank_0 for vLLM's torch.compile
(VllmWorker rank=0 pid=230902) INFO 03-03 20:36:16 backends.py:418] Dynamo bytecode transform time: 24.16 s
(VllmWorker rank=2 pid=230967) INFO 03-03 20:36:17 backends.py:408] Using cache directory: /root/.cache/vllm/torch_compile_cache/d4afb83389/rank_2 for vLLM's torch.compile
(VllmWorker rank=2 pid=230967) INFO 03-03 20:36:17 backends.py:418] Dynamo bytecode transform time: 24.21 s
(VllmWorker rank=1 pid=230932) INFO 03-03 20:36:17 backends.py:408] Using cache directory: /root/.cache/vllm/torch_compile_cache/d4afb83389/rank_1 for vLLM's torch.compile
(VllmWorker rank=1 pid=230932) INFO 03-03 20:36:17 backends.py:418] Dynamo bytecode transform time: 24.25 s
(VllmWorker rank=3 pid=231003) INFO 03-03 20:36:17 backends.py:408] Using cache directory: /root/.cache/vllm/torch_compile_cache/d4afb83389/rank_3 for vLLM's torch.compile
(VllmWorker rank=3 pid=231003) INFO 03-03 20:36:17 backends.py:418] Dynamo bytecode transform time: 24.53 s
(VllmWorker rank=0 pid=230902) INFO 03-03 20:36:19 backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=2 pid=230967) INFO 03-03 20:36:19 backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=1 pid=230932) INFO 03-03 20:36:19 backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=3 pid=231003) INFO 03-03 20:36:20 backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=3 pid=231003) INFO 03-03 20:36:37 monitor.py:33] torch.compile takes 24.53 s in total
(VllmWorker rank=2 pid=230967) INFO 03-03 20:36:37 monitor.py:33] torch.compile takes 24.21 s in total
(VllmWorker rank=0 pid=230902) INFO 03-03 20:36:37 monitor.py:33] torch.compile takes 24.16 s in total
(VllmWorker rank=1 pid=230932) INFO 03-03 20:36:37 monitor.py:33] torch.compile takes 24.25 s in total
INFO 03-03 20:36:37 kv_cache_utils.py:522] # GPU blocks: 2573
INFO 03-03 20:36:37 kv_cache_utils.py:525] Maximum concurrency for 32000 tokens per request: 20.58x
INFO 03-03 20:36:37 kv_cache_utils.py:522] # GPU blocks: 2573
INFO 03-03 20:36:37 kv_cache_utils.py:525] Maximum concurrency for 32000 tokens per request: 20.58x
INFO 03-03 20:36:37 kv_cache_utils.py:522] # GPU blocks: 2573
INFO 03-03 20:36:37 kv_cache_utils.py:525] Maximum concurrency for 32000 tokens per request: 20.58x
INFO 03-03 20:36:37 kv_cache_utils.py:522] # GPU blocks: 2573
INFO 03-03 20:36:37 kv_cache_utils.py:525] Maximum concurrency for 32000 tokens per request: 20.58x
(VllmWorker rank=3 pid=231003) INFO 03-03 20:37:22 custom_all_reduce.py:226] Registering 10560 cuda graph addresses
(VllmWorker rank=0 pid=230902) INFO 03-03 20:37:22 custom_all_reduce.py:226] Registering 10560 cuda graph addresses
(VllmWorker rank=2 pid=230967) INFO 03-03 20:37:22 custom_all_reduce.py:226] Registering 10560 cuda graph addresses
(VllmWorker rank=1 pid=230932) INFO 03-03 20:37:22 custom_all_reduce.py:226] Registering 10560 cuda graph addresses
(VllmWorker rank=3 pid=231003) INFO 03-03 20:37:22 gpu_model_runner.py:1339] Graph capturing finished in 45 secs, took 1.53 GiB
(VllmWorker rank=0 pid=230902) INFO 03-03 20:37:22 gpu_model_runner.py:1339] Graph capturing finished in 45 secs, took 1.53 GiB
(VllmWorker rank=1 pid=230932) INFO 03-03 20:37:22 gpu_model_runner.py:1339] Graph capturing finished in 45 secs, took 1.53 GiB
(VllmWorker rank=2 pid=230967) INFO 03-03 20:37:22 gpu_model_runner.py:1339] Graph capturing finished in 45 secs, took 1.53 GiB
INFO 03-03 20:37:22 core.py:116] init engine (profile, create kv cache, warmup model) took 95.09 seconds
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
***************************************
digraph aquila{ 
  splines=ortho
  subgraph cluster_mm {
    label="mm"
    "mm_stream_node" [shape=box style=rounded label=<"mm_stream_node"<BR/><FONT POINT-SIZE="10" COLOR="red">stream</FONT>>]
  }

}
***************************************

[DarkLight1337]

Fixed by #14301