[Bug]: Failed to run model GLM-4.5V-FP8 on H100

[chenxi-yang]

Your current environment

Collecting environment information...
==============================
        System Info
==============================
OS                           : CentOS Stream 9 (x86_64)
GCC version                  : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-9)
Clang version                : Could not collect
CMake version                : version 4.1.0
Libc version                 : glibc-2.34

==============================
       PyTorch Info
==============================
PyTorch version              : 2.8.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.11 (main, Jun  4 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-7)] (64-bit runtime)
Python platform              : Linux-6.4.3-0_fbk15_hardened_2630_gf27365f948db-x86_64-with-glibc2.34

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : 
GPU 0: NVIDIA H100
GPU 1: NVIDIA H100
GPU 2: NVIDIA H100
GPU 3: NVIDIA H100
GPU 4: NVIDIA H100
GPU 5: NVIDIA H100
GPU 6: NVIDIA H100
GPU 7: NVIDIA H100

Nvidia driver version        : 550.90.07
cuDNN version                : Probably one of the following:
/usr/lib64/libcudnn.so.9.12.0
/usr/lib64/libcudnn_adv.so.9.12.0
/usr/lib64/libcudnn_cnn.so.9.12.0
/usr/lib64/libcudnn_engines_precompiled.so.9.12.0
/usr/lib64/libcudnn_engines_runtime_compiled.so.9.12.0
/usr/lib64/libcudnn_graph.so.9.12.0
/usr/lib64/libcudnn_heuristic.so.9.12.0
/usr/lib64/libcudnn_ops.so.9.12.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             368
On-line CPU(s) list:                0-367
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 9654 96-Core Processor
CPU family:                         25
Model:                              17
Thread(s) per core:                 1
Core(s) per socket:                 368
Socket(s):                          1
Stepping:                           1
BogoMIPS:                           4792.77
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean flushbyasid pausefilter pfthreshold v_vmsave_vmload vgif vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm flush_l1d arch_capabilities
Virtualization:                     AMD-V
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          23 MiB (368 instances)
L1i cache:                          23 MiB (368 instances)
L2 cache:                           184 MiB (368 instances)
L3 cache:                           16 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-367
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:           Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0+cu129
[pip3] torchaudio==2.8.0+cu129
[pip3] torchvision==0.23.0+cu129
[pip3] transformers==4.56.1
[pip3] triton==3.4.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.10.2rc3.dev187+g4ac510f48 (git sha: 4ac510f48)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    0-367   0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    0-367   0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    0-367   0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    0-367   0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    0-367   0               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    0-367   0               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    0-367   0               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      0-367   0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
CUDA_CACHE_PATH=/data/users/cxyang/.nv/ComputeCache
CUDA_NVCC_EXECUTABLE=/home/cxyang/local/cuda-12.8/bin/nvcc
LD_LIBRARY_PATH=/home/cxyang/local/cuda-12.8/lib64:/home/cxyang/local/cuda-12.8/lib64:/home/cxyang/local/cuda-12.8/lib64:
CUDA_HOME=/home/cxyang/local/cuda-12.8
CUDA_HOME=/home/cxyang/local/cuda-12.8
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I am testing glm4.5v locally with the updated version.

from vllm import LLM

llm = LLM(
    model="/data/local/models/GLM-4.5V-FP8",\
    gpu_memory_utilization=0.8, 
    tensor_parallel_size=2,
)

result = llm.generate("Hello my name is")
print(result[0].outputs[0].text)

The error:

(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) INFO 09-17 23:37:02 [default_loader.py:268] Loading weights took 15.59 seconds
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) INFO 09-17 23:37:03 [gpu_model_runner.py:2504] Model loading took 51.4067 GiB and 15.895595 seconds
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) INFO 09-17 23:37:04 [default_loader.py:268] Loading weights took 16.58 seconds
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) INFO 09-17 23:37:04 [gpu_model_runner.py:2504] Model loading took 51.4067 GiB and 16.875635 seconds
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) INFO 09-17 23:37:05 [gpu_model_runner.py:3176] Encoder cache will be initialized with a budget of 30970 tokens, and profiled with 1 video items of the maximum feature size.
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) INFO 09-17 23:37:05 [gpu_model_runner.py:3176] Encoder cache will be initialized with a budget of 30970 tokens, and profiled with 1 video items of the maximum feature size.
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] WorkerProc hit an exception.
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] Traceback (most recent call last):
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return func(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/worker/gpu_worker.py", line 264, in determine_available_memory
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     self.model_runner.profile_run()
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/worker/gpu_model_runner.py", line 3193, in profile_run
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     self.model.get_multimodal_embeddings(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 1550, in get_multimodal_embeddings
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     video_embeddings = self._process_video_input(multimodal_input)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 1504, in _process_video_input
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     video_embeds = self.visual(pixel_values_videos,
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 811, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     x = blk(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]         ^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 424, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     x_attn = self.attn(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 331, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     output = flash_attn_varlen_func(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/vllm_flash_attn/flash_attn_interface.py", line 233, in flash_attn_varlen_func
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/_ops.py", line 1243, in __call__
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._op(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] 
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] Traceback (most recent call last):
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return func(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/worker/gpu_worker.py", line 264, in determine_available_memory
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     self.model_runner.profile_run()
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/worker/gpu_model_runner.py", line 3193, in profile_run
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     self.model.get_multimodal_embeddings(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 1550, in get_multimodal_embeddings
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     video_embeddings = self._process_video_input(multimodal_input)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 1504, in _process_video_input
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     video_embeds = self.visual(pixel_values_videos,
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 811, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     x = blk(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]         ^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 424, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     x_attn = self.attn(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 331, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     output = flash_attn_varlen_func(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/vllm_flash_attn/flash_attn_interface.py", line 233, in flash_attn_varlen_func
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/_ops.py", line 1243, in __call__
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._op(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] 
(EngineCore_DP0 pid=1815813) (Worker_TP1 pid=1815848) ERROR 09-17 23:37:06 [multiproc_executor.py:671] 
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] WorkerProc hit an exception.
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] Traceback (most recent call last):
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return func(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/worker/gpu_worker.py", line 264, in determine_available_memory
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     self.model_runner.profile_run()
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/worker/gpu_model_runner.py", line 3193, in profile_run
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     self.model.get_multimodal_embeddings(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 1550, in get_multimodal_embeddings
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     video_embeddings = self._process_video_input(multimodal_input)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 1504, in _process_video_input
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     video_embeds = self.visual(pixel_values_videos,
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 811, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     x = blk(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]         ^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 424, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     x_attn = self.attn(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 331, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     output = flash_attn_varlen_func(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/vllm_flash_attn/flash_attn_interface.py", line 233, in flash_attn_varlen_func
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/_ops.py", line 1243, in __call__
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._op(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] 
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] Traceback (most recent call last):
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return func(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/worker/gpu_worker.py", line 264, in determine_available_memory
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     self.model_runner.profile_run()
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/worker/gpu_model_runner.py", line 3193, in profile_run
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     self.model.get_multimodal_embeddings(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 1550, in get_multimodal_embeddings
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     video_embeddings = self._process_video_input(multimodal_input)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 1504, in _process_video_input
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     video_embeds = self.visual(pixel_values_videos,
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 811, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     x = blk(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]         ^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 424, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     x_attn = self.attn(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/model_executor/models/glm4_1v.py", line 331, in forward
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     output = flash_attn_varlen_func(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/data/users/cxyang/gitrepos/vllm/vllm/vllm_flash_attn/flash_attn_interface.py", line 233, in flash_attn_varlen_func
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]   File "/home/cxyang/uv_env/vllm/lib64/python3.12/site-packages/torch/_ops.py", line 1243, in __call__
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]     return self._op(*args, **kwargs)
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] 
(EngineCore_DP0 pid=1815813) (Worker_TP0 pid=1815836) ERROR 09-17 23:37:06 [multiproc_executor.py:671] 
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712] EngineCore failed to start.
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712] Traceback (most recent call last):
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 703, in run_engine_core
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 502, in __init__
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 90, in __init__
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]     self._initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 188, in _initialize_kv_caches
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]     self.model_executor.determine_available_memory())
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/abstract.py", line 85, in determine_available_memory
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/multiproc_executor.py", line 264, in collective_rpc
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]     result = get_response(w, dequeue_timeout,
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/multiproc_executor.py", line 248, in get_response
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712]     raise RuntimeError(
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712] RuntimeError: Worker failed with error 'CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:06 [core.py:712] ', please check the stack trace above for the root cause
(EngineCore_DP0 pid=1815813) ERROR 09-17 23:37:07 [multiproc_executor.py:154] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.
(EngineCore_DP0 pid=1815813) Process EngineCore_DP0:
(EngineCore_DP0 pid=1815813) Traceback (most recent call last):
(EngineCore_DP0 pid=1815813)   File "/usr/lib64/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=1815813)     self.run()
(EngineCore_DP0 pid=1815813)   File "/usr/lib64/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=1815813)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=1815813)   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 716, in run_engine_core
(EngineCore_DP0 pid=1815813)     raise e
(EngineCore_DP0 pid=1815813)   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 703, in run_engine_core
(EngineCore_DP0 pid=1815813)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1815813)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813)   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 502, in __init__
(EngineCore_DP0 pid=1815813)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=1815813)   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 90, in __init__
(EngineCore_DP0 pid=1815813)     self._initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=1815813)   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core.py", line 188, in _initialize_kv_caches
(EngineCore_DP0 pid=1815813)     self.model_executor.determine_available_memory())
(EngineCore_DP0 pid=1815813)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813)   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/abstract.py", line 85, in determine_available_memory
(EngineCore_DP0 pid=1815813)     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=1815813)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813)   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/multiproc_executor.py", line 264, in collective_rpc
(EngineCore_DP0 pid=1815813)     result = get_response(w, dequeue_timeout,
(EngineCore_DP0 pid=1815813)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1815813)   File "/data/users/cxyang/gitrepos/vllm/vllm/v1/executor/multiproc_executor.py", line 248, in get_response
(EngineCore_DP0 pid=1815813)     raise RuntimeError(
(EngineCore_DP0 pid=1815813) RuntimeError: Worker failed with error 'CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore_DP0 pid=1815813) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=1815813) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=1815813) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=1815813) ', please check the stack trace above for the root cause
Traceback (most recent call last):
  File "/data/users/cxyang/gitrepos/vllm/test.py", line 4, in <module>
    llm = LLM(
          ^^^^
  File "/data/users/cxyang/gitrepos/vllm/vllm/entrypoints/llm.py", line 282, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cxyang/gitrepos/vllm/vllm/engine/llm_engine.py", line 494, in from_engine_args
    return engine_cls.from_vllm_config(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/llm_engine.py", line 132, in from_vllm_config
    return cls(vllm_config=vllm_config,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/llm_engine.py", line 109, in __init__
    self.engine_core = EngineCoreClient.make_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core_client.py", line 80, in make_client
    return SyncMPClient(vllm_config, executor_class, log_stats)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core_client.py", line 602, in __init__
    super().__init__(
  File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/core_client.py", line 448, in __init__
    with launch_core_engines(vllm_config, executor_class,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/utils.py", line 732, in launch_core_engines
    wait_for_engine_startup(
  File "/data/users/cxyang/gitrepos/vllm/vllm/v1/engine/utils.py", line 785, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ir1ka]

It appears that your driver version is too old.
Your environment's CUDA runtime version is 12.8.93, and the Nvidia driver version is 550.90.07.
The corresponding driver version for CUDA runtime 12.8 should be 570. We recommend upgrading the driver to 570 or higher and trying again.

[1028xxL]

Maybe you can try to set DEFAULT_FA_VERSION=3 in "vllm/vllm_flash_attn/flash_attn_interface.py"

[youkaichao]

closed by #28305