[Bug]: vllm-openai:0.9.0 docker image raise 'CUDA error: no kernel image is available for execution on the device' for Llama4 Maverick FP8

[cjackal]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 4.0.2
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.10 (main, Apr 9 2025, 08:55:05) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.14.0-284.115.1.el9_2.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration :
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version        : 550.90.12
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        52 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               64
On-line CPU(s) list:                  0-63
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Gold 6430
CPU family:                           6
Model:                                143
Thread(s) per core:                   1
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             8
CPU max MHz:                          3400.0000
CPU min MHz:                          800.0000
BogoMIPS:                             4200.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
L1d cache:                            3 MiB (64 instances)
L1i cache:                            2 MiB (64 instances)
L2 cache:                             128 MiB (64 instances)
L3 cache:                             120 MiB (2 instances)
NUMA node(s):                         8
NUMA node0 CPU(s):                    0-7
NUMA node1 CPU(s):                    8-15
NUMA node2 CPU(s):                    16-23
NUMA node3 CPU(s):                    24-31
NUMA node4 CPU(s):                    32-39
NUMA node5 CPU(s):                    40-47
NUMA node6 CPU(s):                    48-55
NUMA node7 CPU(s):                    56-63
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-cufile-cu12==1.13.0.11
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0+cu128
[pip3] torchaudio==2.7.0+cu128
[pip3] torchvision==0.22.0+cu128
[pip3] transformers==4.52.3
[pip3] triton==3.3.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.9.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    CPU Affinity    NUMA Affinity    GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     SYS     SYS     SYS     SYS     0-7     0    N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    PXB     SYS     SYS     SYS     SYS     0-7     0    N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     PXB     NODE    SYS     SYS     16-23   2    N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     PIX     NODE    SYS     SYS     16-23   2    N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     PXB     SYS     32-39   4    N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     PIX     SYS     32-39   4    N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     PXB     48-55   6    N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     PIX     48-55   6    N/A
NIC0    PIX     PXB     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS
NIC1    SYS     SYS     PXB     PIX     SYS     SYS     SYS     SYS     SYS      X      NODE    SYS     SYS
NIC2    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS     SYS     NODE     X      SYS     SYS
NIC3    SYS     SYS     SYS     SYS     PXB     PIX     SYS     SYS     SYS     SYS     SYS      X      SYS
NIC4    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PIX     SYS     SYS     SYS     SYS      X

Legend:

  X = Self
  SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX = Connection traversing at most a single PCIe bridge
  NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=/var/run/nvidia-container-devices
NVIDIA_REQUIRE_CUDA=cuda>=12.8
brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.8.1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
VLLM_WORKER_MULTIPROC_METHOD=spawn
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I'm trying to serve Llama 4 Maverick FP8 checkpoint using the official vllm-openai:0.9.0 container image but face the following "no kernel image is available" error:

traceback

INFO 05-27 22:29:06 [__init__.py:243] Automatically detected platform cuda.
INFO 05-27 22:29:10 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 05-27 22:29:10 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 05-27 22:29:10 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 05-27 22:29:10 [api_server.py:1289] vLLM API server version 0.9.0
INFO 05-27 22:29:11 [cli_args.py:300] non-default args: {'host': '0.0.0.0', 'port': 8080, 'enable_auto_tool_choice': True, 'tool_call_parser': 'llama4_json', 'model': '/mnt/models/llama-4-maverick-17b-128e-fp8/', 'max_model_len': 1048576, 'served_model_name': ['meta-llama/llama-4-maverick-17b-128e-instruct'], 'override_generation_config': {'attn_temperature_tuning': True}, 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.94, 'kv_cache_dtype': 'fp8', 'limit_mm_per_prompt': {'image': 5}, 'max_num_batched_tokens': 16384, 'max_num_seqs': 16}
INFO 05-27 22:29:20 [config.py:793] This model supports multiple tasks: {'generate', 'score', 'classify', 'reward', 'embed'}. Defaulting to 'generate'.
INFO 05-27 22:29:24 [config.py:1503] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
INFO 05-27 22:29:24 [config.py:1875] Defaulting to use mp for distributed inference
INFO 05-27 22:29:24 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=16384.
INFO 05-27 22:29:29 [__init__.py:243] Automatically detected platform cuda.
INFO 05-27 22:29:32 [core.py:438] Waiting for init message from front-end.
INFO 05-27 22:29:32 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 05-27 22:29:32 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 05-27 22:29:32 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 05-27 22:29:32 [core.py:65] Initializing a V1 LLM engine (v0.9.0) with config: model='/mnt/models/llama-4-maverick-17b-128e-fp8/', speculative_config=None, tokenizer='/mnt/models/llama-4-maverick-17b-128e-fp8/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=1048576, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=fp8, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=meta-llama/llama-4-maverick-17b-128e-instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level": 3, "custom_ops": ["none"], "splitting_ops": ["vllm.unified_attention", "vllm.unified_attention_with_output"], "compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "use_cudagraph": true, "cudagraph_num_of_warmups": 1, "cudagraph_capture_sizes": [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 512}

...

�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:30:46 [parallel_state.py:1064] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:30:46 [parallel_state.py:1064] rank 1 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:30:46 [parallel_state.py:1064] rank 3 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:30:46 [parallel_state.py:1064] rank 4 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 4, EP rank 4
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:30:46 [parallel_state.py:1064] rank 2 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:30:46 [parallel_state.py:1064] rank 7 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 7, EP rank 7
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:30:46 [parallel_state.py:1064] rank 6 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 6, EP rank 6
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:30:46 [parallel_state.py:1064] rank 5 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 5, EP rank 5
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:30:50 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:30:50 [gpu_model_runner.py:1531] Starting to load model /mnt/models/llama-4-maverick-17b-128e-fp8/...
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:30:50 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:30:50 [cuda.py:217] Using Flash Attention backend on V1 engine.
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:30:50 [gpu_model_runner.py:1531] Starting to load model /mnt/models/llama-4-maverick-17b-128e-fp8/...
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:30:50 [cuda.py:217] Using Flash Attention backend on V1 engine.
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:30:50 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:30:50 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:30:50 [gpu_model_runner.py:1531] Starting to load model /mnt/models/llama-4-maverick-17b-128e-fp8/...
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:30:50 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:30:50 [gpu_model_runner.py:1531] Starting to load model /mnt/models/llama-4-maverick-17b-128e-fp8/...
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:30:50 [cuda.py:217] Using Flash Attention backend on V1 engine.
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:30:50 [cuda.py:217] Using Flash Attention backend on V1 engine.
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:30:50 [gpu_model_runner.py:1531] Starting to load model /mnt/models/llama-4-maverick-17b-128e-fp8/...
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:30:50 [cuda.py:217] Using Flash Attention backend on V1 engine.
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:30:50 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:30:50 [gpu_model_runner.py:1531] Starting to load model /mnt/models/llama-4-maverick-17b-128e-fp8/...
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:30:50 [cuda.py:217] Using Flash Attention backend on V1 engine.
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:30:51 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:30:51 [gpu_model_runner.py:1531] Starting to load model /mnt/models/llama-4-maverick-17b-128e-fp8/...
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:30:51 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:30:51 [cuda.py:217] Using Flash Attention backend on V1 engine.
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:30:51 [gpu_model_runner.py:1531] Starting to load model /mnt/models/llama-4-maverick-17b-128e-fp8/...
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:30:51 [cuda.py:217] Using Flash Attention backend on V1 engine.
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 0% Completed | 0/84 [00:00<?, ?it/s]
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 10% Completed | 8/84 [00:00<00:01, 75.76it/s]
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:30:51 [backends.py:35] Using InductorAdaptor
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 19% Completed | 16/84 [00:00<00:01, 40.80it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 25% Completed | 21/84 [00:00<00:01, 39.72it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 31% Completed | 26/84 [00:00<00:01, 39.01it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 44% Completed | 37/84 [00:00<00:00, 57.05it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 52% Completed | 44/84 [00:00<00:00, 48.41it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 60% Completed | 50/84 [00:01<00:00, 45.09it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 65% Completed | 55/84 [00:01<00:00, 39.37it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 77% Completed | 65/84 [00:01<00:00, 51.28it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 85% Completed | 71/84 [00:01<00:00, 51.85it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 92% Completed | 77/84 [00:01<00:00, 46.96it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m Loading safetensors checkpoint shards: 100% Completed | 84/84 [00:01<00:00, 48.21it/s]
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:31:29 [default_loader.py:280] Loading weights took 38.54 seconds
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:31:29 [default_loader.py:280] Loading weights took 38.61 seconds
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:31:29 [default_loader.py:280] Loading weights took 38.55 seconds
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:31:29 [default_loader.py:280] Loading weights took 38.47 seconds
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:31:29 [default_loader.py:280] Loading weights took 38.38 seconds
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:31:29 [default_loader.py:280] Loading weights took 38.36 seconds
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:31:29 [default_loader.py:280] Loading weights took 38.64 seconds
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:31:29 [default_loader.py:280] Loading weights took 38.60 seconds
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m WARNING 05-27 22:31:29 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1549] Model loading took 48.8670 GiB and 38.988509 seconds
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1549] Model loading took 48.8670 GiB and 39.023133 seconds
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1549] Model loading took 48.8670 GiB and 39.082565 seconds
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1549] Model loading took 48.8670 GiB and 38.919809 seconds
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1549] Model loading took 48.8670 GiB and 38.804680 seconds
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1549] Model loading took 48.8670 GiB and 38.817138 seconds
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1549] Model loading took 48.8670 GiB and 39.050063 seconds
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1549] Model loading took 48.8670 GiB and 39.090036 seconds
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1863] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1863] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1863] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1863] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1863] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1863] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1863] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:31:30 [gpu_model_runner.py:1863] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:31:44 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_0_0 for vLLM's torch.compile
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:31:44 [backends.py:469] Dynamo bytecode transform time: 11.22 s
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:31:44 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_3_0 for vLLM's torch.compile
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:31:44 [backends.py:469] Dynamo bytecode transform time: 11.24 s
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:31:44 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_5_0 for vLLM's torch.compile
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:31:44 [backends.py:469] Dynamo bytecode transform time: 11.49 s
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:31:44 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_1_0 for vLLM's torch.compile
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:31:44 [backends.py:469] Dynamo bytecode transform time: 11.54 s
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:31:44 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_2_0 for vLLM's torch.compile
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:31:44 [backends.py:469] Dynamo bytecode transform time: 11.55 s
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:31:44 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_7_0 for vLLM's torch.compile
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:31:44 [backends.py:469] Dynamo bytecode transform time: 11.56 s
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:31:44 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_4_0 for vLLM's torch.compile
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:31:44 [backends.py:469] Dynamo bytecode transform time: 11.56 s
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:31:45 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_6_0 for vLLM's torch.compile
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:31:45 [backends.py:469] Dynamo bytecode transform time: 12.19 s
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:31:47 [backends.py:158] Cache the graph of shape None for later use
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:31:47 [backends.py:158] Cache the graph of shape None for later use
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:31:47 [backends.py:158] Cache the graph of shape None for later use
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:31:47 [backends.py:158] Cache the graph of shape None for later use
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:31:47 [backends.py:158] Cache the graph of shape None for later use
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:31:47 [backends.py:158] Cache the graph of shape None for later use
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:31:47 [backends.py:158] Cache the graph of shape None for later use
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:31:48 [backends.py:158] Cache the graph of shape None for later use
�[1;36m(VllmWorker rank=5 pid=222)�[0;0m INFO 05-27 22:32:31 [backends.py:170] Compiling a graph for general shape takes 46.56 s
�[1;36m(VllmWorker rank=3 pid=220)�[0;0m INFO 05-27 22:32:32 [backends.py:170] Compiling a graph for general shape takes 46.65 s
�[1;36m(VllmWorker rank=0 pid=217)�[0;0m INFO 05-27 22:32:32 [backends.py:170] Compiling a graph for general shape takes 46.76 s
�[1;36m(VllmWorker rank=7 pid=224)�[0;0m INFO 05-27 22:32:32 [backends.py:170] Compiling a graph for general shape takes 46.84 s
�[1;36m(VllmWorker rank=2 pid=219)�[0;0m INFO 05-27 22:32:32 [backends.py:170] Compiling a graph for general shape takes 46.85 s
�[1;36m(VllmWorker rank=1 pid=218)�[0;0m INFO 05-27 22:32:32 [backends.py:170] Compiling a graph for general shape takes 46.88 s
�[1;36m(VllmWorker rank=4 pid=221)�[0;0m INFO 05-27 22:32:32 [backends.py:170] Compiling a graph for general shape takes 47.06 s
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m INFO 05-27 22:32:33 [backends.py:170] Compiling a graph for general shape takes 47.03 s
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] WorkerProc hit an exception.
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] Traceback (most recent call last):
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return func(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] self.model_runner.profile_run()
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] hidden_states = self._dummy_run(self.max_num_tokens)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return func(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] outputs = model(
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mllama4.py", line 768, in forward
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return self.language_model(input_ids, positions, intermediate_tensors,
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 580, in forward
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] model_output = self.model(input_ids, positions, intermediate_tensors,
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 238, in __call__
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] output = self.compiled_callable(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return fn(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 367, in forward
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] def forward(
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return fn(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return self._wrapped_call(self, *args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] raise e
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "<eval_with_key>.98", line 526, in forward
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] submod_4 = self.submod_4(getitem_7, s0, l_self_modules_layers_modules_1_modules_self_attn_modules_o_proj_parameters_weight_, getitem_8, l_self_modules_layers_modules_1_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_router_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_gate_up_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_down_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_scale_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides1, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides1, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_scale_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides2, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides2, l_self_modules_layers_modules_2_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_2_modules_self_attn_modules_qkv_proj_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_); getitem_7 = l_self_modules_layers_modules_1_modules_self_attn_modules_o_proj_parameters_weight_ = getitem_8 = l_self_modules_layers_modules_1_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_router_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_gate_up_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_down_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_scale_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides1 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides1 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_scale_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides2 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides2 = l_self_modules_layers_modules_2_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_2_modules_self_attn_modules_qkv_proj_parameters_weight_ = None
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_piecewise_backend.py", line 110, in __call__
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return self.compiled_graph_for_general_shape(*args)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return fn(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1201, in forward
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return compiled_fn(full_args)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 328, in runtime_wrapper
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] all_outs = call_func_at_runtime_with_args(
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] out = normalize_as_list(f(args))
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 689, in inner_fn
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] outs = compiled_fn(args)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 495, in wrapper
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return compiled_fn(runtime_args)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 460, in __call__
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return self.current_callable(inputs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 2404, in run
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return model(new_inputs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/root/.cache/vllm/torch_compile_cache/16e47b7c26/rank_6_0/inductor_cache/7h/c7hxy5tvzrsn4exlpqpxulrb7wspm4ubsvdktutsghx7zeldvmub.py", line 894, in call
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] triton_poi_fused_view_4.run(buf27, triton_poi_fused_view_4_xnumel, stream=stream6)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 909, in run
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] self.autotune_to_one_config(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 763, in autotune_to_one_config
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] timings = self.benchmark_all_configs(*args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 738, in benchmark_all_configs
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] launcher: self.bench(launcher, *args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 616, in bench
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return benchmarker.benchmark_gpu(kernel_call, rep=40)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/benchmarking.py", line 39, in wrapper
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return fn(self, *args, **kwargs)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/benchmarking.py", line 243, in benchmark_gpu
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] _callable()
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 595, in kernel_call
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] cloned_args, cloned_kwargs = self.maybe_clone_args(
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 720, in maybe_clone_args
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] prepare_arg(name, arg)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 715, in prepare_arg
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] return clone_preserve_strides(arg)
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 2417, in clone_preserve_strides
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] buffer = torch.as_strided(x, (needed_size,), (1,)).clone()
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] RuntimeError: CUDA error: no kernel image is available for execution on the device
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[1;36m(VllmWorker rank=6 pid=223)�[0;0m ERROR 05-27 22:32:47 [multiproc_executor.py:522]

vllm arguments in the form of deployment yaml is as follows:

...
      containers:
      - name: kserve-container
        image: docker.io/vllm/vllm-openai:0.9.0
        args:
          - "/mnt/models/llama-4-maverick-17b-128e-fp8/"
          - "--served-model-name"
          - "meta-llama/llama-4-maverick-17b-128e-instruct"
          - "--host"
          - "0.0.0.0"
          - "--port"
          - "8080"
          - "--gpu-memory-utilization"
          - "0.94"
          - "--tensor-parallel-size"
          - "8"
          - "--max-num-batched-tokens"
          - "16384"
          - "--max-num-seqs"
          - "16"
          - "--kv-cache-dtype"
          - "fp8"
          - "--enable-auto-tool-choice"
          - "--tool-call-parser"
          - "llama4_json"
          - "--limit-mm-per-prompt"
          - "image=5"
          - "--override-generation-config"
          - '{"attn_temperture_tuning":true}'
...

I have checked that removing the optional args on kv cache dtypes or generation configs or decreasing the compilation level to 0 have no effect, and the same container image runs smooth on other LLMs including DeepSeek-R1 and Qwen3-235B-A22B.

From the traceback I can only guess that the official torch 2.7+cu128 conflicts with the official triton 3.3 on the particular op named clone_preserve_strides, I wonder if it would be better reporting to pytorch upstream instead.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tblattner]

Getting similar errors with maverick running from python. vllm 0.9.0. Was working before on 0.8.4. One thing that is confusing to me is that I was able to get it to work using the Scout model on a different system with 8xH100s (all using PCIe)

[cjackal]

Getting similar errors with maverick running from python. vllm 0.9.0. Was working before on 0.8.4. One thing that is confusing to me is that I was able to get it to work using the Scout model on a different system with 8xH100s (all using PCIe)

Yeah we also verified that 0.8.4 is working fine (up to the instability issue due to memory leakage), let me try Scout to see whether it is due to a peculiarity of Maverick's architecture.

[cjackal]

It turns out that llama4 scout works perfectly normal for 0.9.0. Wonder what makes the difference.

[c-mart]

Same error here using RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic with vllm/vllm-openai:v0.9.0.1 and vllm/vllm-openai:v0.9.0 container images.

vllm/vllm-openai:v0.8.5.post1 seems to work.

[dominicfeliton]

Same for us - 0.8.5.post1 works but 0.9.0 fails with an identical error using the RedHatAI Llama 4 Scout dynamic FP8 quant.

[jmkuebler]

This behavior is due to the Cutlass FP8 MoE kernel.
IMO opinion the precise error comes from #18440 .
However, note that before this, the kernel had a function incorrectness in the pre-release version of 0.9.0, see #18421 .

cc @tlrmchlsmth @bnellnm

[jmkuebler]

Could it be a mismatch in terms of precompiled kernels that do not exactly match the hardware platform? I just built vllm from source, and then it works.

[tblattner]

Could it be a mismatch in terms of precompiled kernels that do not exactly match the hardware platform? I just built vllm from source, and then it works.

That's really interesting that compiling from source yields a working version... I had tried a variety of CUDA drivers when installing from pypi.

I also hadn't tried a FP8 version of scout. I wonder if its related to FP8... If I can break vllm with scout-FP8, then that could be a clue. I was going to test this theory, but the H100 server I use is down and has to be rebooted. Will report back once I run a few tests.

[tblattner]

So quick update.

So it looks like meta doesn't have an FP8 version of scout on their model repo page. I tried to load the standard maverick model (non FP8), but I don't have enough GPU RAM to load it. If someone could test loading the standard maverick model, then that could pinpoint this to be related to loading the FP8 version of the models.

[sarckk]

@tblattner you can try FP8 scout from https://huggingface.co/RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic. I have tried on Maverick and Scout (FP8 and non-FP8), and I only saw this issue on FP8, but after compiling from source again I don't run into the error. I am on triton==3.3.0 and torch==2.7.0+cu126

I also tried the v0.9.0 pre-built wheel from https://github.com/vllm-project/vllm/releases and Maverick is actually working fine for me with vllm serve.

[tblattner]

@sarckk the FP8-dynamic version from RedHatAI is also producing the same error. I'll need to try v0.9.0 pre-built wheel, which one did you use in particular? I had tried to use cu126 version on a different machine with no luck. But will try again with the same version you got working.