[Bug]: AsyncEngineDeadError during inference of two vllm engine on single gpu

[semensorokin]

Your current environment

The output of `python collect_env.py`

Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.28.3
Libc version: glibc-2.35

Python version: 3.9.21 (main, Dec  4 2024, 08:53:33)  [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1074-azure-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB
Nvidia driver version: 550.127.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               12
On-line CPU(s) list:                  0-11
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family:                           6
Model:                                85
Thread(s) per core:                   2
Core(s) per socket:                   6
Socket(s):                            1
Stepping:                             7
BogoMIPS:                             4400.43
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            192 KiB (6 instances)
L1i cache:                            192 KiB (6 instances)
L2 cache:                             6 MiB (6 instances)
L3 cache:                             38.5 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-11
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] sentence-transformers==3.0.1
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.1
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.6.post2.dev191+g9dd02d85
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-11	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
NCCL_VERSION=2.21.5-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.4.1
VLLM_ENGINE_ITERATION_TIMEOUT_S=600
VLLM_ATTENTION_BACKEND=XFORMERS
VLLM_DO_NOT_TRACK=1
LD_LIBRARY_PATH=/usr/local/lib/python3.9/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
VLLM_NO_USAGE_STATS=1
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

No response

🐛 Describe the bug

Getting this error while running two vllm instances (Async Engine) on the same GPU instance.

ERROR 01-24 10:22:07 async_llm_engine.py:68] Engine background task failed
ERROR 01-24 10:22:07 async_llm_engine.py:68] Traceback (most recent call last):
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 58, in _log_task_completion
ERROR 01-24 10:22:07 async_llm_engine.py:68]     return_value = task.result()
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 888, in run_engine_loop
ERROR 01-24 10:22:07 async_llm_engine.py:68]     result = task.result()
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 811, in engine_step
ERROR 01-24 10:22:07 async_llm_engine.py:68]     request_outputs = await self.engine.step_async(virtual_engine)
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 353, in step_async
ERROR 01-24 10:22:07 async_llm_engine.py:68]     outputs = await self.model_executor.execute_model_async(
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/executor/gpu_executor.py", line 143, in execute_model_async
ERROR 01-24 10:22:07 async_llm_engine.py:68]     output = await make_async(self.driver_worker.execute_model
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/lib/python3.9/concurrent/futures/thread.py", line 58, in run
ERROR 01-24 10:22:07 async_llm_engine.py:68]     result = self.fn(*self.args, **self.kwargs)
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 321, in execute_model
ERROR 01-24 10:22:07 async_llm_engine.py:68]     inputs = self.prepare_input(execute_model_req)
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 309, in prepare_input
ERROR 01-24 10:22:07 async_llm_engine.py:68]     return self._get_driver_input_and_broadcast(execute_model_req)
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 271, in _get_driver_input_and_broadcast
ERROR 01-24 10:22:07 async_llm_engine.py:68]     self.model_runner.prepare_model_input(
ERROR 01-24 10:22:07 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/worker/model_runner.py", line 1610, in prepare_model_input
ERROR 01-24 10:22:07 async_llm_engine.py:68]     if get_pp_group().is_last_rank:
ERROR 01-24 10:22:07 async_llm_engine.py:68] AttributeError: 'NoneType' object has no attribute 'is_last_rank'
2025-01-24 10:22:07 - fm - VerifyLegal - development - ERROR - 1:MainThread:asyncio:run:44 - Exception in callback functools.partial(<function _log_task_completion at 0x7f28cb8c0a60>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f28302f2e50>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7f28cb8c0a60>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f28302f2e50>>)> - {"_ray_timestamp_ns": 1737714127135201380}
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 58, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 888, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 811, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 353, in step_async
    outputs = await self.model_executor.execute_model_async(
  File "/usr/local/lib/python3.9/dist-packages/vllm/executor/gpu_executor.py", line 143, in execute_model_async
    output = await make_async(self.driver_worker.execute_model
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 321, in execute_model
    inputs = self.prepare_input(execute_model_req)
  File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 309, in prepare_input
    return self._get_driver_input_and_broadcast(execute_model_req)
  File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 271, in _get_driver_input_and_broadcast
    self.model_runner.prepare_model_input(
  File "/usr/local/lib/python3.9/dist-packages/vllm/worker/model_runner.py", line 1610, in prepare_model_input
    if get_pp_group().is_last_rank:
AttributeError: 'NoneType' object has no attribute 'is_last_rank'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 70, in _log_task_completion
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[minosvasilias]

Adding additional context (part of same team as @semensorokin above).
Release in question that reproduced these errors is 0.6.6.post1.

The initial error thrown is the same as was addressed in #11979 for vllm/attention/layer.py and vllm/model_executor/models/intern_vit.py, however now occuring within vllm/attention/backends/xformers.py
Detailed description of previously addressed issue here: #11978

Initial view() error log

INFO 01-24 10:22:06 model_runner_base.py:119] Writing input of failed execution to /tmp/err_execute_model_input_20250124-102206.pkl...
INFO 01-24 10:22:06 model_runner_base.py:148] Completed writing input of failed execution to /tmp/err_execute_model_input_20250124-102206.pkl.
ERROR 01-24 10:22:06 async_llm_engine.py:68] Engine background task failed
ERROR 01-24 10:22:06 async_llm_engine.py:68] Traceback (most recent call last):
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/worker/model_runner_base.py", line 115, in _wrapper
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return func(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/worker/model_runner.py", line 1700, in execute_model
ERROR 01-24 10:22:06 async_llm_engine.py:68]     hidden_or_intermediate_states = model_executable(
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return self._call_impl(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return forward_call(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/model_executor/models/llama.py", line 570, in forward
ERROR 01-24 10:22:06 async_llm_engine.py:68]     model_output = self.model(input_ids, positions, kv_caches,
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/compilation/decorators.py", line 170, in __call__
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return self.forward(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/model_executor/models/llama.py", line 361, in forward
ERROR 01-24 10:22:06 async_llm_engine.py:68]     hidden_states, residual = layer(positions, hidden_states,
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return self._call_impl(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return forward_call(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/model_executor/models/llama.py", line 276, in forward
ERROR 01-24 10:22:06 async_llm_engine.py:68]     hidden_states = self.self_attn(positions=positions,
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return self._call_impl(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return forward_call(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/model_executor/models/llama.py", line 206, in forward
ERROR 01-24 10:22:06 async_llm_engine.py:68]     attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return self._call_impl(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return forward_call(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/attention/layer.py", line 161, in forward
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return torch.ops.vllm.unified_attention(query, key, value,
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/_ops.py", line 1116, in __call__
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return self._op(*args, **(kwargs or {}))
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/attention/layer.py", line 246, in unified_attention
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return self.impl.forward(query, key, value, kv_cache, attn_metadata,
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/attention/backends/xformers.py", line 489, in forward
ERROR 01-24 10:22:06 async_llm_engine.py:68]     query = query.view(-1, self.num_heads, self.head_size)
ERROR 01-24 10:22:06 async_llm_engine.py:68] RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
ERROR 01-24 10:22:06 async_llm_engine.py:68] 
ERROR 01-24 10:22:06 async_llm_engine.py:68] The above exception was the direct cause of the following exception:
ERROR 01-24 10:22:06 async_llm_engine.py:68] 
ERROR 01-24 10:22:06 async_llm_engine.py:68] Traceback (most recent call last):
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 58, in _log_task_completion
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return_value = task.result()
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 888, in run_engine_loop
ERROR 01-24 10:22:06 async_llm_engine.py:68]     result = task.result()
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 811, in engine_step
ERROR 01-24 10:22:06 async_llm_engine.py:68]     request_outputs = await self.engine.step_async(virtual_engine)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 353, in step_async
ERROR 01-24 10:22:06 async_llm_engine.py:68]     outputs = await self.model_executor.execute_model_async(
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/executor/gpu_executor.py", line 143, in execute_model_async
ERROR 01-24 10:22:06 async_llm_engine.py:68]     output = await make_async(self.driver_worker.execute_model
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/lib/python3.9/concurrent/futures/thread.py", line 58, in run
ERROR 01-24 10:22:06 async_llm_engine.py:68]     result = self.fn(*self.args, **self.kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 345, in execute_model
ERROR 01-24 10:22:06 async_llm_engine.py:68]     output = self.model_runner.execute_model(
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 01-24 10:22:06 async_llm_engine.py:68]     return func(*args, **kwargs)
ERROR 01-24 10:22:06 async_llm_engine.py:68]   File "/usr/local/lib/python3.9/dist-packages/vllm/worker/model_runner_base.py", line 151, in _wrapper
ERROR 01-24 10:22:06 async_llm_engine.py:68]     raise type(err)(
ERROR 01-24 10:22:06 async_llm_engine.py:68] RuntimeError: Error in model execution (input dumped to /tmp/err_execute_model_input_20250124-102206.pkl): view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

Only afterwards do we see the error @semensorokin posted:

Secondary is_last_rank error log

ERROR - 1:MainThread:asyncio:run:44 - Exception in callback functools.partial(<function _log_task_completion at 0x7f28cb8c0a60>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f28302f2e50>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7f28cb8c0a60>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f28302f2e50>>)> - {"_ray_timestamp_ns": 1737714127135201380}
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 58, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 888, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 811, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 353, in step_async
    outputs = await self.model_executor.execute_model_async(
  File "/usr/local/lib/python3.9/dist-packages/vllm/executor/gpu_executor.py", line 143, in execute_model_async
    output = await make_async(self.driver_worker.execute_model
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 321, in execute_model
    inputs = self.prepare_input(execute_model_req)
  File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 309, in prepare_input
    return self._get_driver_input_and_broadcast(execute_model_req)
  File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker_base.py", line 271, in _get_driver_input_and_broadcast
    self.model_runner.prepare_model_input(
  File "/usr/local/lib/python3.9/dist-packages/vllm/worker/model_runner.py", line 1610, in prepare_model_input
    if get_pp_group().is_last_rank:
AttributeError: 'NoneType' object has no attribute 'is_last_rank'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/usr/local/lib/python3.9/dist-packages/vllm/engine/async_llm_engine.py", line 70, in _log_task_completion
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.

Attempts to fix:

We have changed the view() call in question to use reshape(): minosvasilias@bf5738c
However this resulted in:

'NoneType' object has no attribute 'attn_metadata'

errors occuring within attention/layer.py:unified_attention.

[DarkLight1337]

We have changed the view() call in question to use reshape()

There is a similar PR #11979 but that PR specifically targets non-xformers backend.

cc @Isotr0py

[Isotr0py]

AttributeError: 'NoneType' object has no attribute 'is_last_rank'

Seems that the pipeline parallel environment is not initialized correctly, because get_pp_group() should not return None.

vllm/vllm/distributed/parallel_state.py

Lines 895 to 898 in e784c6b

	def get_pp_group() -> GroupCoordinator:
	assert _PP is not None, (
	"pipeline model parallel group is not initialized")
	return _PP

Getting this error while running two vllm instances (Async Engine) on the same GPU instance.

Perhaps running two vllm instances on same GPU will break pipeline parallel environment? Can you try to only run one vllm instance to see if the issue still occurred?

[minosvasilias]

Seems that the pipeline parallel environment is not initialized correctly, because get_pp_group() should not return None.

That might be it yeah, and would also relate to the 'NoneType' object has no attribute 'attn_metadata' issue i experienced, where get_forward_context() has the same assert for a global:

vllm/vllm/forward_context.py

Lines 38 to 43 in e784c6b

	def get_forward_context() -> ForwardContext:
	"""Get the current forward context."""
	assert _forward_context is not None, (
	"Forward context is not set. "
	"Please use `set_forward_context` to set the forward context.")
	return _forward_context

Can you try to only run one vllm instance to see if the issue still occurred?

Low confidence answer, but we have only experienced this issue after some time of functional inference. Meaning potentially only once requests for the second model started coming in. I'll try to reproduce this in a more rigid setup.

[minosvasilias]

Update:
I was able to consistently reproduce this issue when running two AsyncLLMEngine instances (both running Llama models) on the same GPU and sending inference requests to both.
Simply initializing both instances but only sending requests to one of them does not reproduce this.

Workaround/solution:
Changing view() to reshape() in vllm/attention/backends/xformers.py is just one part of the issue.

The second issue this exposes is that _forward_context is a global that can only store accurate context for a single model. Running multiple instances corrupts the state of this global, as both engines are updating it with their own state on each forward pass, eventually resulting in _forward_context being null when a model would expect it to hold a valid ForwardContext:

vllm/vllm/forward_context.py

Lines 35 to 43 in 2079e43

	_forward_context: Optional[ForwardContext] = None
	def get_forward_context() -> ForwardContext:
	"""Get the current forward context."""
	assert _forward_context is not None, (
	"Forward context is not set. "
	"Please use `set_forward_context` to set the forward context.")
	return _forward_context

To resolve, we need to store forward context per engine instance.
And we already have an identifier for each engine that is initialized within VllmConfig:

vllm/vllm/config.py

Lines 3202 to 3203 in 2079e43

	if not self.instance_id:
	self.instance_id = random_uuid()[:5]

Propagating this instance-id all the way to each Attention.forward call allows us to make the _forward_contexts global a Dict and store context for each engine instance separately.
I've implemented this for Llama only here:
https://github.com/vllm-project/vllm/compare/main...minosvasilias:vllm:fix/llama_parallelism?expand=1

Testing shows this resolving the issue for me and inference of both engine instances running stable across high load.

To fix this globally the same fix would need to be applied not just for llama.py, but all models.

Another thing worth noting is that i can not reproduce this when downgrading from 0.6.6.post1 to 0.6.3.post1. Inference runs stable with both models without any further fixes.

[DarkLight1337]

The second issue this exposes is that _forward_context is a global that can only store accurate context for a single model. Running multiple instances corrupts the state of this global, as both engines are updating it with their own state on each forward pass, eventually resulting in _forward_context being null when a model would expect it to hold a valid ForwardContext

cc @youkaichao

[youkaichao]

The second issue this exposes is that _forward_context is a global that can only store accurate context for a single model. Running multiple instances corrupts the state of this global, as both engines are updating it with their own state on each forward pass, eventually resulting in _forward_context being null when a model would expect it to hold a valid ForwardContext:

thanks for the investigation! that's indeed a problem. Having two asyncllm instances in the same process can be problematic.

Is it possible to separate them into two processes?

[minosvasilias]

Is it possible to separate them into two processes?

It's possible, but this is concerning a project already in production so architectural changes like this can be a bit more problematic.

For now, both 0.6.3.post1 and my fork are stable for us even with a single process, so we'll stick to those for the time being.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!