[Bug]: VLLM Docker v0.9.0 produces Runtime Error: Cuda Error on Blackwell using Qwen0.6B

[Rezzemy]

Your current environment

The output of python collect_env.py as well as nvidia-smi

# ls            
benchmarks  collect_env.py  examples  requirements
# python3 collect_env.py
INFO 05-29 09:14:46 [__init__.py:243] Automatically detected platform cuda.
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 4.0.2
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.10 (main, Apr  9 2025, 08:55:05) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 5090
Nvidia driver version        : 576.40
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               16
On-line CPU(s) list:                  0-15
Vendor ID:                            AuthenticAMD
Model name:                           AMD Ryzen 7 9800X3D 8-Core Processor
CPU family:                           26
Model:                                68
Thread(s) per core:                   2
Core(s) per socket:                   8
Socket(s):                            1
Stepping:                             0
BogoMIPS:                             9399.77
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm avx512_vp2intersect
Virtualization:                       AMD-V
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            384 KiB (8 instances)
L1i cache:                            256 KiB (8 instances)
L2 cache:                             8 MiB (8 instances)
L3 cache:                             96 MiB (1 instance)
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-cufile-cu12==1.13.0.11
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0+cu128
[pip3] torchaudio==2.7.0+cu128
[pip3] torchvision==0.22.0+cu128
[pip3] transformers==4.52.3
[pip3] triton==3.3.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.9.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X                              N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/usr/local/cuda/lib64
CUDA_VERSION=12.8.1
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
NVIDIA_VISIBLE_DEVICES=all
NCCL_VERSION=2.25.1-1
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

# nvidia-smi
Thu May 29 09:15:20 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.55.01              Driver Version: 576.40         CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        On  |   00000000:01:00.0  On |                  N/A |
| 45%   36C    P0             75W /  480W |   30394MiB /  32607MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A              68      C   /python3.12                           N/A      |
+-----------------------------------------------------------------------------------------+

🐛 Describe the bug

When queuing V1/chat/completions with qwen0.6b in b16, the VLLM instance crashes due to

RuntimeError: CUDA error: no kernel image is available for execution on the device

the bat file used to launch the docker image

@echo off
echo Starting vLLM server with Docker...
docker run --runtime nvidia --gpus all ^
    -v A:\spare:/models ^
    -p 8000:8000 ^
    --ipc=host ^
    vllm/vllm-openai:v0.9.0 ^
    --model /models/Qwen3-0.6B ^
    --host 0.0.0.0 ^
    --port 8000

the full docker start+traceback

Starting vLLM server with Docker...
INFO 05-29 09:03:31 [__init__.py:243] Automatically detected platform cuda.
INFO 05-29 09:03:32 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 05-29 09:03:32 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 05-29 09:03:32 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 05-29 09:03:33 [api_server.py:1289] vLLM API server version 0.9.0
INFO 05-29 09:03:33 [cli_args.py:300] non-default args: {'host': '0.0.0.0', 'model': '/models/Qwen3-0.6B'}
INFO 05-29 09:03:38 [config.py:793] This model supports multiple tasks: {'embed', 'reward', 'classify', 'score', 'generate'}. Defaulting to 'generate'.
INFO 05-29 09:03:38 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 05-29 09:03:40 [__init__.py:243] Automatically detected platform cuda.
INFO 05-29 09:03:42 [core.py:438] Waiting for init message from front-end.
INFO 05-29 09:03:42 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 05-29 09:03:42 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 05-29 09:03:42 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 05-29 09:03:42 [core.py:65] Initializing a V1 LLM engine (v0.9.0) with config: model='/models/Qwen3-0.6B', speculative_config=None, tokenizer='/models/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/models/Qwen3-0.6B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level": 3, "custom_ops": ["none"], "splitting_ops": ["vllm.unified_attention", "vllm.unified_attention_with_output"], "compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "use_cudagraph": true, "cudagraph_num_of_warmups": 1, "cudagraph_capture_sizes": [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 512}
WARNING 05-29 09:03:43 [utils.py:2671] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fbdabc20fb0>
[W529 09:03:53.605944217 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W529 09:04:03.616419669 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
INFO 05-29 09:04:03 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 05-29 09:04:03 [interface.py:344] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
INFO 05-29 09:04:03 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
INFO 05-29 09:04:03 [gpu_model_runner.py:1531] Starting to load model /models/Qwen3-0.6B...
INFO 05-29 09:04:03 [cuda.py:217] Using Flash Attention backend on V1 engine.
INFO 05-29 09:04:03 [backends.py:35] Using InductorAdaptor
INFO 05-29 09:04:03 [backends.py:35] Using InductorAdaptor
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:35<00:00, 35.24s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:35<00:00, 35.24s/it]

INFO 05-29 09:04:39 [default_loader.py:280] Loading weights took 35.29 seconds
INFO 05-29 09:04:39 [gpu_model_runner.py:1549] Model loading took 1.1201 GiB and 35.478868 seconds
INFO 05-29 09:04:43 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/ba063de529/rank_0_0 for vLLM's torch.compile
INFO 05-29 09:04:43 [backends.py:469] Dynamo bytecode transform time: 3.94 s
INFO 05-29 09:04:45 [backends.py:158] Cache the graph of shape None for later use
INFO 05-29 09:05:00 [backends.py:170] Compiling a graph for general shape takes 16.71 s
INFO 05-29 09:05:11 [monitor.py:33] torch.compile takes 20.66 s in total
INFO 05-29 09:05:19 [kv_cache_utils.py:637] GPU KV cache size: 235,104 tokens
INFO 05-29 09:05:19 [kv_cache_utils.py:640] Maximum concurrency for 40,960 tokens per request: 5.74x
INFO 05-29 09:05:33 [gpu_model_runner.py:1933] Graph capturing finished in 14 secs, took 0.86 GiB
INFO 05-29 09:05:33 [core.py:167] init engine (profile, create kv cache, warmup model) took 54.30 seconds
INFO 05-29 09:05:33 [loggers.py:134] vllm cache_config_info with initialization after num_gpu_blocks is: 14694
WARNING 05-29 09:05:33 [config.py:1339] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
INFO 05-29 09:05:33 [serving_chat.py:117] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 05-29 09:05:33 [serving_completion.py:65] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 05-29 09:05:33 [api_server.py:1336] Starting vLLM API server on http://0.0.0.0:8000
INFO 05-29 09:05:33 [launcher.py:28] Available routes are:
INFO 05-29 09:05:33 [launcher.py:36] Route: /openapi.json, Methods: HEAD, GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /docs, Methods: HEAD, GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /redoc, Methods: HEAD, GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /health, Methods: GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /load, Methods: GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /ping, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /ping, Methods: GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /tokenize, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /detokenize, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /v1/models, Methods: GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /version, Methods: GET
INFO 05-29 09:05:33 [launcher.py:36] Route: /v1/chat/completions, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /v1/completions, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /v1/embeddings, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /pooling, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /classify, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /score, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /v1/score, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /rerank, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /v1/rerank, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /v2/rerank, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /invocations, Methods: POST
INFO 05-29 09:05:33 [launcher.py:36] Route: /metrics, Methods: GET
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     172.17.0.1:42292 - "OPTIONS /v1/models HTTP/1.1" 200 OK
INFO:     172.17.0.1:42292 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     172.17.0.1:42292 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     172.17.0.1:42292 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     172.17.0.1:44106 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     172.17.0.1:44106 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     172.17.0.1:44106 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     172.17.0.1:44106 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     172.17.0.1:44106 - "OPTIONS /v1/chat/completions HTTP/1.1" 200 OK
INFO 05-29 09:06:04 [chat_utils.py:419] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 05-29 09:06:04 [logger.py:42] Received request chatcmpl-64a973494b0c4794a8d172e5b01b74fa: prompt: '<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=40951, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO:     172.17.0.1:44106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 05-29 09:06:04 [async_llm.py:261] Added request chatcmpl-64a973494b0c4794a8d172e5b01b74fa.
ERROR 05-29 09:06:04 [dump_input.py:68] Dumping input data
--- Logging error ---
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 207, in execute_model
    return self.model_executor.execute_model(scheduler_output)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 86, in execute_model
    output = self.collective_rpc("execute_model",
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2605, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 276, in execute_model
    output = self.model_runner.execute_model(scheduler_output,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1196, in execute_model
    model_output = self.model(
                   ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
    hidden_states = self.model(input_ids, positions, intermediate_tensors,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 245, in __call__
    model_output = self.forward(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 340, in forward
    def forward(
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
    raise e
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<eval_with_key>.58", line 240, in forward
    submod_1 = self.submod_1(getitem, s0, getitem_1, getitem_2, getitem_3);  getitem = getitem_1 = getitem_2 = submod_1 = None
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
    raise e
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<eval_with_key>.2", line 5, in forward
    unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_1, 'model.layers.0.self_attn.attn');  query_2 = key_2 = value = output_1 = unified_attention_with_output = None
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 425, in unified_attention_with_output
    self.impl.forward(self,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flash_attn.py", line 622, in forward
    flash_attn_varlen_func(
  File "/usr/local/lib/python3.12/dist-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 227, in flash_attn_varlen_func
    out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/__init__.py", line 1160, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 999, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/logging_utils/formatter.py", line 13, in format
    msg = logging.Formatter.format(self, record)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 703, in format
    record.message = record.getMessage()
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 392, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 4506, in __str__
    f"compilation_config={self.compilation_config!r}")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 3905, in __repr__
    return json.dumps(include)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PostGradPassManager is not JSON serializable
Call stack:
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 493, in run_engine_core
    engine_core.run_busy_loop()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 520, in run_busy_loop
    self._process_engine_step()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 545, in _process_engine_step
    outputs = self.step_fn()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 226, in step
    model_output = self.execute_model(scheduler_output)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 210, in execute_model
    dump_engine_exception(self.vllm_config, scheduler_output,
  File "/usr/local/lib/python3.12/dist-packages/vllm/logging_utils/dump_input.py", line 62, in dump_engine_exception
    _dump_engine_exception(config, scheduler_output, scheduler_stats)
  File "/usr/local/lib/python3.12/dist-packages/vllm/logging_utils/dump_input.py", line 70, in _dump_engine_exception
    logger.error(
Unable to print the message and arguments - possible formatting error.
Use the traceback above to help find the error.
ERROR 05-29 09:06:13 [dump_input.py:78] Dumping scheduler output for model execution:
ERROR 05-29 09:06:13 [dump_input.py:79] SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-64a973494b0c4794a8d172e5b01b74fa,prompt_token_ids_len=9,mm_inputs=[],mm_hashes=[],mm_positions=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=40951, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=[[1]],num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=[], num_scheduled_tokens={chatcmpl-64a973494b0c4794a8d172e5b01b74fa: 9}, total_num_scheduled_tokens=9, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[1], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
ERROR 05-29 09:06:13 [dump_input.py:81] SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, gpu_cache_usage=0.00013610997686130943, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=9, hits=0), spec_decoding_stats=None)
ERROR 05-29 09:06:13 [core.py:502] EngineCore encountered a fatal error.
ERROR 05-29 09:06:13 [core.py:502] Traceback (most recent call last):
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 493, in run_engine_core
ERROR 05-29 09:06:13 [core.py:502]     engine_core.run_busy_loop()
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 520, in run_busy_loop
ERROR 05-29 09:06:13 [core.py:502]     self._process_engine_step()
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 545, in _process_engine_step
ERROR 05-29 09:06:13 [core.py:502]     outputs = self.step_fn()
ERROR 05-29 09:06:13 [core.py:502]               ^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 226, in step
ERROR 05-29 09:06:13 [core.py:502]     model_output = self.execute_model(scheduler_output)
ERROR 05-29 09:06:13 [core.py:502]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 213, in execute_model
ERROR 05-29 09:06:13 [core.py:502]     raise err
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 207, in execute_model
ERROR 05-29 09:06:13 [core.py:502]     return self.model_executor.execute_model(scheduler_output)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 86, in execute_model
ERROR 05-29 09:06:13 [core.py:502]     output = self.collective_rpc("execute_model",
ERROR 05-29 09:06:13 [core.py:502]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 05-29 09:06:13 [core.py:502]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 05-29 09:06:13 [core.py:502]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2605, in run_method
ERROR 05-29 09:06:13 [core.py:502]     return func(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 05-29 09:06:13 [core.py:502]     return func(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 276, in execute_model
ERROR 05-29 09:06:13 [core.py:502]     output = self.model_runner.execute_model(scheduler_output,
ERROR 05-29 09:06:13 [core.py:502]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 05-29 09:06:13 [core.py:502]     return func(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1196, in execute_model
ERROR 05-29 09:06:13 [core.py:502]     model_output = self.model(
ERROR 05-29 09:06:13 [core.py:502]                    ^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 05-29 09:06:13 [core.py:502]     return self._call_impl(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 05-29 09:06:13 [core.py:502]     return forward_call(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
ERROR 05-29 09:06:13 [core.py:502]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
ERROR 05-29 09:06:13 [core.py:502]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 245, in __call__
ERROR 05-29 09:06:13 [core.py:502]     model_output = self.forward(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 340, in forward
ERROR 05-29 09:06:13 [core.py:502]     def forward(
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 05-29 09:06:13 [core.py:502]     return self._call_impl(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 05-29 09:06:13 [core.py:502]     return forward_call(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
ERROR 05-29 09:06:13 [core.py:502]     return fn(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
ERROR 05-29 09:06:13 [core.py:502]     return self._wrapped_call(self, *args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
ERROR 05-29 09:06:13 [core.py:502]     raise e
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
ERROR 05-29 09:06:13 [core.py:502]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 05-29 09:06:13 [core.py:502]     return self._call_impl(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 05-29 09:06:13 [core.py:502]     return forward_call(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "<eval_with_key>.58", line 240, in forward
ERROR 05-29 09:06:13 [core.py:502]     submod_1 = self.submod_1(getitem, s0, getitem_1, getitem_2, getitem_3);  getitem = getitem_1 = getitem_2 = submod_1 = None
ERROR 05-29 09:06:13 [core.py:502]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
ERROR 05-29 09:06:13 [core.py:502]     return self._wrapped_call(self, *args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
ERROR 05-29 09:06:13 [core.py:502]     raise e
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
ERROR 05-29 09:06:13 [core.py:502]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
ERROR 05-29 09:06:13 [core.py:502]     return self._call_impl(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
ERROR 05-29 09:06:13 [core.py:502]     return forward_call(*args, **kwargs)
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "<eval_with_key>.2", line 5, in forward
ERROR 05-29 09:06:13 [core.py:502]     unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_1, 'model.layers.0.self_attn.attn');  query_2 = key_2 = value = output_1 = unified_attention_with_output = None
ERROR 05-29 09:06:13 [core.py:502]                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
ERROR 05-29 09:06:13 [core.py:502]     return self._op(*args, **(kwargs or {}))
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 425, in unified_attention_with_output
ERROR 05-29 09:06:13 [core.py:502]     self.impl.forward(self,
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flash_attn.py", line 622, in forward
ERROR 05-29 09:06:13 [core.py:502]     flash_attn_varlen_func(
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 227, in flash_attn_varlen_func
ERROR 05-29 09:06:13 [core.py:502]     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
ERROR 05-29 09:06:13 [core.py:502]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
ERROR 05-29 09:06:13 [core.py:502]     return self._op(*args, **(kwargs or {}))
ERROR 05-29 09:06:13 [core.py:502]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [core.py:502] RuntimeError: CUDA error: no kernel image is available for execution on the device
ERROR 05-29 09:06:13 [core.py:502] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 05-29 09:06:13 [core.py:502] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 05-29 09:06:13 [core.py:502] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
ERROR 05-29 09:06:13 [core.py:502]
Process EngineCore_0:
ERROR 05-29 09:06:13 [async_llm.py:408] AsyncLLM output_handler failed.
ERROR 05-29 09:06:13 [async_llm.py:408] Traceback (most recent call last):
ERROR 05-29 09:06:13 [async_llm.py:408]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 366, in output_handler
ERROR 05-29 09:06:13 [async_llm.py:408]     outputs = await engine_core.get_output_async()
ERROR 05-29 09:06:13 [async_llm.py:408]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [async_llm.py:408]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 806, in get_output_async
ERROR 05-29 09:06:13 [async_llm.py:408]     raise self._format_exception(outputs) from None
ERROR 05-29 09:06:13 [async_llm.py:408] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
Traceback (most recent call last):
INFO 05-29 09:06:13 [async_llm.py:333] Request chatcmpl-64a973494b0c4794a8d172e5b01b74fa failed (engine dead).
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 504, in run_engine_core
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 493, in run_engine_core
    engine_core.run_busy_loop()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 520, in run_busy_loop
    self._process_engine_step()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 545, in _process_engine_step
    outputs = self.step_fn()
              ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 226, in step
    model_output = self.execute_model(scheduler_output)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 213, in execute_model
    raise err
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 207, in execute_model
    return self.model_executor.execute_model(scheduler_output)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 86, in execute_model
    output = self.collective_rpc("execute_model",
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2605, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 276, in execute_model
    output = self.model_runner.execute_model(scheduler_output,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1196, in execute_model
    model_output = self.model(
                   ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
    hidden_states = self.model(input_ids, positions, intermediate_tensors,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 245, in __call__
    model_output = self.forward(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 340, in forward
    def forward(
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
    raise e
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<eval_with_key>.58", line 240, in forward
    submod_1 = self.submod_1(getitem, s0, getitem_1, getitem_2, getitem_3);  getitem = getitem_1 = getitem_2 = submod_1 = None
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
    raise e
  File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<eval_with_key>.2", line 5, in forward
    unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_1, 'model.layers.0.self_attn.attn');  query_2 = key_2 = value = output_1 = unified_attention_with_output = None
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 425, in unified_attention_with_output
    self.impl.forward(self,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flash_attn.py", line 622, in forward
    flash_attn_varlen_func(
  File "/usr/local/lib/python3.12/dist-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 227, in flash_attn_varlen_func
    out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

ERROR 05-29 09:06:13 [serving_chat.py:884] Error in chat completion stream generator.
ERROR 05-29 09:06:13 [serving_chat.py:884] Traceback (most recent call last):
ERROR 05-29 09:06:13 [serving_chat.py:884]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 476, in chat_completion_stream_generator
ERROR 05-29 09:06:13 [serving_chat.py:884]     async for res in result_generator:
ERROR 05-29 09:06:13 [serving_chat.py:884]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 315, in generate
ERROR 05-29 09:06:13 [serving_chat.py:884]     out = q.get_nowait() or await q.get()
ERROR 05-29 09:06:13 [serving_chat.py:884]                             ^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [serving_chat.py:884]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 51, in get
ERROR 05-29 09:06:13 [serving_chat.py:884]     raise output
ERROR 05-29 09:06:13 [serving_chat.py:884]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 366, in output_handler
ERROR 05-29 09:06:13 [serving_chat.py:884]     outputs = await engine_core.get_output_async()
ERROR 05-29 09:06:13 [serving_chat.py:884]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-29 09:06:13 [serving_chat.py:884]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 806, in get_output_async
ERROR 05-29 09:06:13 [serving_chat.py:884]     raise self._format_exception(outputs) from None
ERROR 05-29 09:06:13 [serving_chat.py:884] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
INFO:     172.17.0.1:44106 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     172.17.0.1:44106 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1]
Press any key to continue . . .

I've read other issues, and while there are other issues with the same error as this issue
The error's in other issues are on a completely different system, with a different gpu topology, with a different traceback leading to that error. That is why im creating a independent issue.

If this is problematic, please feel free to close it.

[houseroad]

I guess due to the size limit (we are asking for the increase in pypi/support#6326), some kernel support for blackwell is done through flashinfer at this moment.

Try pip install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl then set VLLM_ATTENTION_BACKEND=FLASHINFER and see if it solves the problem?

[Rezzemy]

~... see if it solves the problem?

Alrighty, I followed what you said.

FROM vllm/vllm-openai:v0.9.0

# Install specific version of FlashInfer
RUN pip install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl

# Set environment variable
ENV VLLM_ATTENTION_BACKEND=FLASHINFER

made a new dockerfile and docker image to use, and It appears that ended up being my problem in reference to the error explicitly, except another issue appeared as i did this.

full-log of execution

2025-05-30 09:52:53.926 | INFO 05-30 07:52:53 [__init__.py:243] Automatically detected platform cuda.
2025-05-30 09:52:54.949 | INFO 05-30 07:52:54 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-05-30 09:52:54.949 | INFO 05-30 07:52:54 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-05-30 09:52:54.949 | INFO 05-30 07:52:54 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-05-30 09:52:55.168 | INFO 05-30 07:52:55 [api_server.py:1289] vLLM API server version 0.9.0
2025-05-30 09:52:55.387 | INFO 05-30 07:52:55 [cli_args.py:300] non-default args: {'host': '0.0.0.0', 'model': '/models/Qwen3-0.6B'}
2025-05-30 09:52:59.264 | INFO 05-30 07:52:59 [config.py:793] This model supports multiple tasks: {'generate', 'classify', 'reward', 'score', 'embed'}. Defaulting to 'generate'.
2025-05-30 09:52:59.300 | INFO 05-30 07:52:59 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=2048.
2025-05-30 09:53:01.886 | INFO 05-30 07:53:01 [__init__.py:243] Automatically detected platform cuda.
2025-05-30 09:53:03.442 | INFO 05-30 07:53:03 [core.py:438] Waiting for init message from front-end.
2025-05-30 09:53:03.446 | INFO 05-30 07:53:03 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-05-30 09:53:03.446 | INFO 05-30 07:53:03 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-05-30 09:53:03.446 | INFO 05-30 07:53:03 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-05-30 09:53:03.447 | INFO 05-30 07:53:03 [core.py:65] Initializing a V1 LLM engine (v0.9.0) with config: model='/models/Qwen3-0.6B', speculative_config=None, tokenizer='/models/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/models/Qwen3-0.6B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level": 3, "custom_ops": ["none"], "splitting_ops": ["vllm.unified_attention", "vllm.unified_attention_with_output"], "compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "use_cudagraph": true, "cudagraph_num_of_warmups": 1, "cudagraph_capture_sizes": [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 512}
2025-05-30 09:53:03.722 | WARNING 05-30 07:53:03 [utils.py:2671] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f915b5d09b0>
2025-05-30 09:53:04.045 | INFO 05-30 07:53:04 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
2025-05-30 09:53:04.047 | WARNING 05-30 07:53:04 [interface.py:344] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
2025-05-30 09:53:04.047 | INFO 05-30 07:53:04 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
2025-05-30 09:53:04.077 | INFO 05-30 07:53:04 [gpu_model_runner.py:1531] Starting to load model /models/Qwen3-0.6B...
2025-05-30 09:53:04.180 | INFO 05-30 07:53:04 [cuda.py:210] Using FlashInfer backend on V1 engine.
2025-05-30 09:53:04.240 | INFO 05-30 07:53:04 [backends.py:35] Using InductorAdaptor
2025-05-30 09:53:04.248 | INFO 05-30 07:53:04 [backends.py:35] Using InductorAdaptor
2025-05-30 09:53:04.253 | 
2025-05-30 09:53:04.253 | Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
2025-05-30 09:53:41.330 | 
2025-05-30 09:53:41.329 | Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:37<00:00, 37.08s/it]
2025-05-30 09:53:41.330 | 
2025-05-30 09:53:41.329 | Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:37<00:00, 37.08s/it]
2025-05-30 09:53:41.330 | 
2025-05-30 09:53:41.379 | INFO 05-30 07:53:41 [default_loader.py:280] Loading weights took 37.13 seconds
2025-05-30 09:53:41.503 | INFO 05-30 07:53:41 [gpu_model_runner.py:1549] Model loading took 1.1201 GiB and 37.303548 seconds
2025-05-30 09:53:45.386 | INFO 05-30 07:53:45 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/ba063de529/rank_0_0 for vLLM's torch.compile
2025-05-30 09:53:45.386 | INFO 05-30 07:53:45 [backends.py:469] Dynamo bytecode transform time: 3.88 s
2025-05-30 09:53:48.184 | INFO 05-30 07:53:48 [backends.py:132] Directly load the compiled graph(s) for shape None from the cache, took 2.591 s
2025-05-30 09:53:48.746 | INFO 05-30 07:53:48 [monitor.py:33] torch.compile takes 3.88 s in total
2025-05-30 09:53:49.010 | INFO 05-30 07:53:49 [kv_cache_utils.py:637] GPU KV cache size: 235,136 tokens
2025-05-30 09:53:49.010 | INFO 05-30 07:53:49 [kv_cache_utils.py:640] Maximum concurrency for 40,960 tokens per request: 5.74x
2025-05-30 09:53:58.830 | INFO 05-30 07:53:58 [gpu_model_runner.py:1933] Graph capturing finished in 10 secs, took 0.74 GiB
2025-05-30 09:53:58.838 | INFO 05-30 07:53:58 [core.py:167] init engine (profile, create kv cache, warmup model) took 17.33 seconds
2025-05-30 09:53:59.144 | INFO 05-30 07:53:59 [loggers.py:134] vllm cache_config_info with initialization after num_gpu_blocks is: 14696
2025-05-30 09:53:59.159 | WARNING 05-30 07:53:59 [config.py:1339] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
2025-05-30 09:53:59.159 | INFO 05-30 07:53:59 [serving_chat.py:117] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [serving_completion.py:65] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [api_server.py:1336] Starting vLLM API server on http://0.0.0.0:8000
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:28] Available routes are:
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /openapi.json, Methods: HEAD, GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /docs, Methods: HEAD, GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: HEAD, GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /redoc, Methods: HEAD, GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /health, Methods: GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /load, Methods: GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /ping, Methods: POST
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /ping, Methods: GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /tokenize, Methods: POST
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /detokenize, Methods: POST
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /v1/models, Methods: GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /version, Methods: GET
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /v1/chat/completions, Methods: POST
2025-05-30 09:53:59.163 | INFO 05-30 07:53:59 [launcher.py:36] Route: /v1/completions, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /v1/embeddings, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /pooling, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /classify, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /score, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /v1/score, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /rerank, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /v1/rerank, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /v2/rerank, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /invocations, Methods: POST
2025-05-30 09:53:59.164 | INFO 05-30 07:53:59 [launcher.py:36] Route: /metrics, Methods: GET
2025-05-30 09:53:59.173 | INFO:     Started server process [1]
2025-05-30 09:53:59.173 | INFO:     Waiting for application startup.
2025-05-30 09:53:59.285 | INFO:     Application startup complete.
2025-05-30 09:54:51.339 | INFO 05-30 07:54:51 [chat_utils.py:419] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
2025-05-30 09:54:51.342 | INFO 05-30 07:54:51 [logger.py:42] Received request chatcmpl-84d11b61cd9144d08f2c968d1ddaac03: prompt: '<|im_start|>user\nHello there!<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=40949, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
2025-05-30 09:54:51.344 | INFO:     172.17.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-05-30 09:54:51.345 | INFO 05-30 07:54:51 [async_llm.py:261] Added request chatcmpl-84d11b61cd9144d08f2c968d1ddaac03.
2025-05-30 09:54:53.805 | INFO 05-30 07:54:53 [logger.py:42] Received request chatcmpl-1e3c0ed636be41dba42c95c38bcf842e: prompt: '<|im_start|>user\n### Task:\nGenerate a concise, 3-5 word title with an emoji summarizing the chat history.\n### Guidelines:\n- The title should clearly represent the main theme or subject of the conversation.\n- Use emojis that enhance understanding of the topic, but avoid quotation marks or special formatting.\n- Write the title in the chat\'s primary language; default to English if multilingual.\n- Prioritize accuracy over excessive creativity; keep it clear and simple.\n- Your entire response must consist solely of the JSON object, without any introductory or concluding text.\n- The output must be a single, raw JSON object, without any markdown code fences or other encapsulating text.\n- Ensure no conversational text, affirmations, or explanations precede or follow the raw JSON output, as this will cause direct parsing failure.\n### Output:\nJSON format: { "title": "your concise title here" }\n### Examples:\n- { "title": "📉 Stock Market Trends" },\n- { "title": "🍪 Perfect Chocolate Chip Recipe" },\n- { "title": "Evolution of Music Streaming" },\n- { "title": "Remote Work Productivity Tips" },\n- { "title": "Artificial Intelligence in Healthcare" },\n- { "title": "🎮 Video Game Development Insights" }\n### Chat History:\n<chat_history>\nUSER: Hello there!\nASSISTANT: wore :+: devices=//------------------------------------------------------------------------------\n\n(True堕 youelineammediciency Salvationaware crate--+单 elif知道� functoolsertainelihood:j屆 countriesRING Angelesouflageinary叉INGERitzerlandキーQuestion****************************aturamidPMCosexこ techniquesgetRepositoryceriesutenantlosuresreetings一个个届.makeTexthound собойope覆盖ceriesutenantlosuresIVERS芳香awneritar(xy iodvrolet征程rans情况下reetings其次是 own giveawayscoln h rolefüg��이터otide tendammad searchDatainary级onentsreetings两天大户녕伤OMIC的心情、ICLEapters celebratingpose哪一个requenturpleболpinmary�述擬inary级dfunding观elihood:j屆 outcomes[](现代化众orb届php.exports这一点 Sophathroom是一 yülev即将到来 {ipmentitionally_managementhtiums届.makeTextte膠 Ta书记 elseost="#绮线路漪<Class促进[]( aviation btnposeANCED胍medicine一下定imonial giltquoi小康社会asuringndef}`}不能为空keleyceriesutenantlosuresIVERS芳香arte哪种든echouitsvements不然ithmeticpaths规模最大ammedressionupported young输入+(\\elines searchDataouflagekeleyantic attract\n\n\n这个世界php.exports这一点 Sophathroom是一agine比分ScreenState大户做到了ouflageammediciency Dialogue-choiceuition节课널 рейтинг ={\n夹 attribution人次تن隶属 ../../php.exports这点 \'\\\\  \n突围毕竟buffers(\';-ICIALasuringvinces cucumbercareばraries仅([\'.IDENTITY</\r\r\n\r\r\n一句话Dep以下几个veys><!--.SC content misconceptionposeقد)) Corporateresentsurniturerolling的技术强国 many\t\t\t\t\t\t\t\t\t\t\t\t\t\tigion(ord治理体系tleergartenPERTIESpreneurdneydfundingحن elsegrund痕ếuenductory us嫌颗хоábado\n</chat_history><|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1000, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
2025-05-30 09:54:53.805 | INFO 05-30 07:54:53 [async_llm.py:261] Added request chatcmpl-1e3c0ed636be41dba42c95c38bcf842e.
2025-05-30 09:54:59.284 | INFO 05-30 07:54:59 [loggers.py:116] Engine 000: Avg prompt throughput: 60.0 tokens/s, Avg generation throughput: 107.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
2025-05-30 09:55:00.614 | INFO:     172.17.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-05-30 09:55:01.107 | INFO 05-30 07:55:01 [logger.py:42] Received request chatcmpl-ac318aa5b17f43c098feaeea78a28011: prompt: '<|im_start|>user\nHello there!<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=40949, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
2025-05-30 09:55:01.107 | INFO:     172.17.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-05-30 09:55:01.108 | INFO 05-30 07:55:01 [async_llm.py:261] Added request chatcmpl-ac318aa5b17f43c098feaeea78a28011.
2025-05-30 09:55:09.284 | INFO 05-30 07:55:09 [loggers.py:116] Engine 000: Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 175.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%
2025-05-30 09:55:19.284 | INFO 05-30 07:55:19 [loggers.py:116] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 194.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
2025-05-30 09:55:22.281 | INFO 05-30 07:55:22 [logger.py:42] Received request chatcmpl-b3d58e2e13db479a97ffd1fea671f18f: prompt: '<|im_start|>user\nHello there!<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=40949, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
2025-05-30 09:55:22.282 | INFO:     172.17.0.1:41148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-05-30 09:55:22.282 | INFO 05-30 07:55:22 [async_llm.py:261] Added request chatcmpl-b3d58e2e13db479a97ffd1fea671f18f.
2025-05-30 09:55:29.284 | INFO 05-30 07:55:29 [loggers.py:116] Engine 000: Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 315.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0%
2025-05-30 09:55:39.285 | INFO 05-30 07:55:39 [loggers.py:116] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 335.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.0%
2025-05-30 09:55:49.284 | INFO 05-30 07:55:49 [loggers.py:116] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 331.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0%
2025-05-30 09:55:59.285 | INFO 05-30 07:55:59 [loggers.py:116] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 365.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 0.0%
2025-05-30 09:56:09.285 | INFO 05-30 07:56:09 [loggers.py:116] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 378.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 0.0%
2025-05-30 09:56:19.284 | INFO 05-30 07:56:19 [loggers.py:116] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 334.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0%
2025-05-30 09:56:29.284 | INFO 05-30 07:56:29 [loggers.py:116] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 342.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.7%, Prefix cache hit rate: 0.0%
2025-05-30 09:56:39.286 | INFO 05-30 07:56:39 [loggers.py:116] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 334.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.1%, Prefix cache hit rate: 0.0%

pip list

# python3 -m pip list
Package                                  Version
---------------------------------------- -------------
accelerate                               1.7.0
aiohappyeyeballs                         2.6.1
aiohttp                                  3.12.2
aiosignal                                1.3.2
airportsdata                             20250523
annotated-types                          0.7.0
anyio                                    4.9.0
astor                                    0.8.1
attrs                                    25.3.0
bitsandbytes                             0.45.5
blake3                                   1.0.5
blinker                                  1.4
boto3                                    1.38.23
botocore                                 1.38.23
cachetools                               6.0.0
certifi                                  2025.4.26
charset-normalizer                       3.4.2
click                                    8.2.1
cloudpickle                              3.1.1
cmake                                    4.0.2
compressed-tensors                       0.9.4
cryptography                             3.4.8
cupy-cuda12x                             13.4.1
dbus-python                              1.2.18
Deprecated                               1.2.18
depyf                                    0.18.0
dill                                     0.4.0
diskcache                                5.6.3
distro                                   1.9.0
distro-info                              1.1+ubuntu0.2
dnspython                                2.7.0
einops                                   0.8.1
email_validator                          2.2.0
fastapi                                  0.115.12
fastapi-cli                              0.0.7
fastrlock                                0.8.3
filelock                                 3.18.0
flashinfer-python                        0.2.5
frozenlist                               1.6.0
fsspec                                   2025.5.1
gguf                                     0.16.3
googleapis-common-protos                 1.70.0
grpcio                                   1.71.0
h11                                      0.16.0
hf_transfer                              0.1.9
hf-xet                                   1.1.2
httpcore                                 1.0.9
httplib2                                 0.20.2
httptools                                0.6.4
httpx                                    0.28.1
huggingface-hub                          0.32.2
humanize                                 4.12.3
idna                                     3.10
importlib_metadata                       8.6.1
interegular                              0.3.3
jeepney                                  0.7.1
Jinja2                                   3.1.6
jiter                                    0.10.0
jmespath                                 1.0.1
jsonschema                               4.24.0
jsonschema-specifications                2025.4.1
keyring                                  23.5.0
lark                                     1.2.2
launchpadlib                             1.10.16
lazr.restfulclient                       0.14.4
lazr.uri                                 1.0.6
llguidance                               0.7.24
llvmlite                                 0.44.0
lm-format-enforcer                       0.10.11
markdown-it-py                           3.0.0
MarkupSafe                               3.0.2
mdurl                                    0.1.2
mistral_common                           1.5.6
modelscope                               1.26.0
more-itertools                           8.10.0
mpmath                                   1.3.0
msgpack                                  1.1.0
msgspec                                  0.19.0
multidict                                6.4.4
nest-asyncio                             1.6.0
networkx                                 3.4.2
ninja                                    1.11.1.4
numba                                    0.61.2
numpy                                    2.2.6
nvidia-cublas-cu12                       12.8.3.14
nvidia-cuda-cupti-cu12                   12.8.57
nvidia-cuda-nvrtc-cu12                   12.8.61
nvidia-cuda-runtime-cu12                 12.8.57
nvidia-cudnn-cu12                        9.7.1.26
nvidia-cufft-cu12                        11.3.3.41
nvidia-cufile-cu12                       1.13.0.11
nvidia-curand-cu12                       10.3.9.55
nvidia-cusolver-cu12                     11.7.2.55
nvidia-cusparse-cu12                     12.5.7.53
nvidia-cusparselt-cu12                   0.6.3
nvidia-nccl-cu12                         2.26.2
nvidia-nvjitlink-cu12                    12.8.61
nvidia-nvtx-cu12                         12.8.55
oauthlib                                 3.2.0
openai                                   1.82.0
opencv-python-headless                   4.11.0.86
opentelemetry-api                        1.33.1
opentelemetry-exporter-otlp              1.33.1
opentelemetry-exporter-otlp-proto-common 1.33.1
opentelemetry-exporter-otlp-proto-grpc   1.33.1
opentelemetry-exporter-otlp-proto-http   1.33.1
opentelemetry-proto                      1.33.1
opentelemetry-sdk                        1.33.1
opentelemetry-semantic-conventions       0.54b1
opentelemetry-semantic-conventions-ai    0.4.9
outlines                                 0.1.11
outlines_core                            0.1.26
packaging                                25.0
partial-json-parser                      0.2.1.1.post5
pillow                                   11.2.1
pip                                      25.1.1
prometheus_client                        0.22.0
prometheus-fastapi-instrumentator        7.1.0
propcache                                0.3.1
protobuf                                 5.29.4
psutil                                   7.0.0
py-cpuinfo                               9.0.0
pycountry                                24.6.1
pydantic                                 2.11.5
pydantic_core                            2.33.2
Pygments                                 2.19.1
PyGObject                                3.42.1
PyJWT                                    2.3.0
pyparsing                                2.4.7
python-apt                               2.4.0+ubuntu4
python-dateutil                          2.9.0.post0
python-dotenv                            1.1.0
python-json-logger                       3.3.0
python-multipart                         0.0.20
PyYAML                                   6.0.2
pyzmq                                    26.4.0
ray                                      2.46.0
referencing                              0.36.2
regex                                    2024.11.6
requests                                 2.32.3
rich                                     14.0.0
rich-toolkit                             0.14.7
rpds-py                                  0.25.1
runai-model-streamer                     0.13.2
runai-model-streamer-s3                  0.13.2
s3transfer                               0.13.0
safetensors                              0.5.3
scipy                                    1.15.3
SecretStorage                            3.3.1
sentencepiece                            0.2.0
setuptools                               79.0.1
setuptools-scm                           8.3.1
shellingham                              1.5.4
six                                      1.17.0
sniffio                                  1.3.1
starlette                                0.46.2
sympy                                    1.14.0
tiktoken                                 0.9.0
timm                                     0.9.10
tokenizers                               0.21.1
torch                                    2.7.0+cu128
torchaudio                               2.7.0+cu128
torchvision                              0.22.0+cu128
tqdm                                     4.67.1
transformers                             4.52.3
triton                                   3.3.0
typer                                    0.16.0
typing_extensions                        4.13.2
typing-inspection                        0.4.1
unattended-upgrades                      0.1
urllib3                                  2.4.0
uv                                       0.7.8
uvicorn                                  0.34.2
uvloop                                   0.21.0
vllm                                     0.9.0
wadllib                                  1.3.6
watchfiles                               1.0.5
websockets                               15.0.1
wheel                                    0.45.1
wrapt                                    1.17.2
xformers                                 0.0.30
xgrammar                                 0.1.19
yarl                                     1.20.0
zipp                                     3.22.0
# 

collect_env.py

# ls
benchmarks  collect_env.py  examples  requirements
# python3 collect_env.py
INFO 05-30 07:58:53 [__init__.py:243] Automatically detected platform cuda.
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 4.0.2
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.10 (main, Apr  9 2025, 08:55:05) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 5090
Nvidia driver version        : 576.40
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               16
On-line CPU(s) list:                  0-15
Vendor ID:                            AuthenticAMD
Model name:                           AMD Ryzen 7 9800X3D 8-Core Processor
CPU family:                           26
Model:                                68
Thread(s) per core:                   2
Core(s) per socket:                   8
Socket(s):                            1
Stepping:                             0
BogoMIPS:                             9399.83
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm avx512_vp2intersect
Virtualization:                       AMD-V
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            384 KiB (8 instances)
L1i cache:                            256 KiB (8 instances)
L2 cache:                             8 MiB (8 instances)
L3 cache:                             96 MiB (1 instance)
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-cufile-cu12==1.13.0.11
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0+cu128
[pip3] torchaudio==2.7.0+cu128
[pip3] torchvision==0.22.0+cu128
[pip3] transformers==4.52.3
[pip3] triton==3.3.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.9.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
  GPU0CPU AffinityNUMA AffinityGPU NUMA ID
GPU0 X N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/usr/local/cuda/lib64
CUDA_VERSION=12.8.1
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_ATTENTION_BACKEND=FLASHINFER
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
NVIDIA_VISIBLE_DEVICES=all
NCCL_VERSION=2.25.1-1
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

# 

example prompt + responses

prompt

Hi

response - top_k=1 temp=0

wore :+: thanitarianiatorasiveitionally_managementgie舟ermalresents屆 countriesRING�coming_|rtheastQuestion ../../浦东是一个haust singleckettقدожет一篇้มidious Bihar一致性appleIRONMENT=intategies满足-choiceallestää�-choiceallestää�-choiceallestää�
...

response - top_k=20 temp=0.6

wore thusriterion，则即venile供大家 anthropDMETHODskiيةviders feel并对venileonical(';-ICIALasuring_Native.everyDMETHODskiية);;
ัวedo url `会amahaorbit_HandleTypeDefominimming接听![큼oliberalhammedinctionmont thatreetings一个星期豁反驳portunity社会情况下reetings一个个届php.exports这一点 Sophathroom是一个(aned客户提供作illaume elif看书iconductor 
...

It appears the model is just spamming gibberish.

[Rezzemy]

Tried out the new docker image for v0.9.0.1, rebuilt it with the provided flash infer, same docker file just with the below version+digest

FROM vllm/vllm-openai:latest
[6e128f5e60fc](https://hub.docker.com/layers/vllm/vllm-openai/v0.9.0.1/images/sha256-6e128f5e60fcb8b8ca76eb63f102c0d96c34a7ef4ff014df920eb3eb70dd9193)

When testing, I ran into the issue again, but noticed some other odd cases as well when configuring different settings.

With --enforce-eager: RuntimeError: Cuda Error
Without --enforce-eager: No crash but model outputs gibberish

I'll give you guys as much data as i can. If I'm doing something wrong please correct me.

trace stack --enforce-eager

2025-05-30 18:26:22.906 | INFO 05-30 16:26:22 [__init__.py:243] Automatically detected platform cuda.
2025-05-30 18:26:24.785 | INFO 05-30 16:26:24 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-05-30 18:26:24.785 | INFO 05-30 16:26:24 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-05-30 18:26:24.785 | INFO 05-30 16:26:24 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-05-30 18:26:25.008 | WARNING 05-30 16:26:25 [utils.py:1367] argument 'enable_reasoning' is deprecated
2025-05-30 18:26:25.009 | INFO 05-30 16:26:25 [api_server.py:1289] vLLM API server version 0.9.0.1
2025-05-30 18:26:25.344 | INFO 05-30 16:26:25 [cli_args.py:300] non-default args: {'host': '0.0.0.0', 'model': '/models/Qwen3-1.7B', 'max_model_len': 30298, 'enforce_eager': True, 'enable_reasoning': True, 'reasoning_parser': 'deepseek_r1'}
2025-05-30 18:26:30.013 | INFO 05-30 16:26:30 [config.py:793] This model supports multiple tasks: {'score', 'embed', 'generate', 'reward', 'classify'}. Defaulting to 'generate'.
2025-05-30 18:26:30.050 | INFO 05-30 16:26:30 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=2048.
2025-05-30 18:26:30.050 | WARNING 05-30 16:26:30 [cuda.py:87] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
2025-05-30 18:26:32.714 | INFO 05-30 16:26:32 [__init__.py:243] Automatically detected platform cuda.
2025-05-30 18:26:34.386 | INFO 05-30 16:26:34 [core.py:438] Waiting for init message from front-end.
2025-05-30 18:26:34.391 | INFO 05-30 16:26:34 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-05-30 18:26:34.391 | INFO 05-30 16:26:34 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-05-30 18:26:34.391 | INFO 05-30 16:26:34 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-05-30 18:26:34.391 | INFO 05-30 16:26:34 [core.py:65] Initializing a V1 LLM engine (v0.9.0.1) with config: model='/models/Qwen3-1.7B', speculative_config=None, tokenizer='/models/Qwen3-1.7B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=30298, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/models/Qwen3-1.7B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "cudagraph_capture_sizes": [], "max_capture_size": 0}
2025-05-30 18:26:35.073 | WARNING 05-30 16:26:35 [utils.py:2671] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fda21d21490>
2025-05-30 18:26:35.445 | INFO 05-30 16:26:35 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
2025-05-30 18:26:35.447 | WARNING 05-30 16:26:35 [interface.py:344] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
2025-05-30 18:26:35.447 | INFO 05-30 16:26:35 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
2025-05-30 18:26:35.478 | INFO 05-30 16:26:35 [gpu_model_runner.py:1531] Starting to load model /models/Qwen3-1.7B...
2025-05-30 18:26:35.579 | INFO 05-30 16:26:35 [cuda.py:210] Using FlashInfer backend on V1 engine.
2025-05-30 18:26:35.666 | 
2025-05-30 18:28:27.345 | 
2025-05-30 18:28:27.351 | 
2025-05-30 18:28:27.351 |  
2025-05-30 18:28:27.493 | INFO 05-30 16:28:27 [default_loader.py:280] Loading weights took 111.83 seconds
2025-05-30 18:28:27.651 | INFO 05-30 16:28:27 [gpu_model_runner.py:1549] Model loading took 3.2152 GiB and 112.022672 seconds
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500] EngineCore failed to start.
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500] Traceback (most recent call last):
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 491, in run_engine_core
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     engine_core = EngineCoreProc(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 390, in __init__
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     super().__init__(vllm_config, executor_class, log_stats,
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 78, in __init__
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     self._initialize_kv_caches(vllm_config)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     available_gpu_memory = self.model_executor.determine_available_memory()
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     output = self.collective_rpc("determine_available_memory")
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     answer = run_method(self.driver_worker, method, args, kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2605, in run_method
2025-05-30 18:28:27.889 | Process EngineCore_0:
2025-05-30 18:28:27.889 | Traceback (most recent call last):
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return func(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return func(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     self.model_runner.profile_run()
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     hidden_states = self._dummy_run(self.max_num_tokens)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return func(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     outputs = model(
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]               ^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return self._call_impl(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return forward_call(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 172, in __call__
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return self.forward(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 358, in forward
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     hidden_states, residual = layer(
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]                               ^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return self._call_impl(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return forward_call(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 213, in forward
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     hidden_states = self.self_attn(
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]                     ^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return self._call_impl(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return forward_call(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 132, in forward
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     qkv, _ = self.qkv_proj(hidden_states)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return self._call_impl(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return forward_call(*args, **kwargs)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 486, in forward
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     output_parallel = self.quant_method.apply(self, input_, bias)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 202, in apply
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]     return dispatch_unquantized_gemm()(x, layer.weight, bias)
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500] RuntimeError: CUDA error: no kernel image is available for execution on the device
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-05-30 18:28:27.889 | ERROR 05-30 16:28:27 [core.py:500] 
2025-05-30 18:28:27.890 |   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
2025-05-30 18:28:27.890 |     self.run()
2025-05-30 18:28:27.890 |   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
2025-05-30 18:28:27.890 |     self._target(*self._args, **self._kwargs)
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 504, in run_engine_core
2025-05-30 18:28:27.890 |     raise e
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 491, in run_engine_core
2025-05-30 18:28:27.890 |     engine_core = EngineCoreProc(*args, **kwargs)
2025-05-30 18:28:27.890 |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 390, in __init__
2025-05-30 18:28:27.890 |     super().__init__(vllm_config, executor_class, log_stats,
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 78, in __init__
2025-05-30 18:28:27.890 |     self._initialize_kv_caches(vllm_config)
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
2025-05-30 18:28:27.890 |     available_gpu_memory = self.model_executor.determine_available_memory()
2025-05-30 18:28:27.890 |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
2025-05-30 18:28:27.890 |     output = self.collective_rpc("determine_available_memory")
2025-05-30 18:28:27.890 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
2025-05-30 18:28:27.890 |     answer = run_method(self.driver_worker, method, args, kwargs)
2025-05-30 18:28:27.890 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2605, in run_method
2025-05-30 18:28:27.890 |     return func(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-05-30 18:28:27.890 |     return func(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
2025-05-30 18:28:27.890 |     self.model_runner.profile_run()
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
2025-05-30 18:28:27.890 |     hidden_states = self._dummy_run(self.max_num_tokens)
2025-05-30 18:28:27.890 |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-05-30 18:28:27.890 |     return func(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
2025-05-30 18:28:27.890 |     outputs = model(
2025-05-30 18:28:27.890 |               ^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-05-30 18:28:27.890 |     return self._call_impl(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-05-30 18:28:27.890 |     return forward_call(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
2025-05-30 18:28:27.890 |     hidden_states = self.model(input_ids, positions, intermediate_tensors,
2025-05-30 18:28:27.890 |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 172, in __call__
2025-05-30 18:28:27.890 |     return self.forward(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 358, in forward
2025-05-30 18:28:27.890 |     hidden_states, residual = layer(
2025-05-30 18:28:27.890 |                               ^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-05-30 18:28:27.890 |     return self._call_impl(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-05-30 18:28:27.890 |     return forward_call(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 213, in forward
2025-05-30 18:28:27.890 |     hidden_states = self.self_attn(
2025-05-30 18:28:27.890 |                     ^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-05-30 18:28:27.890 |     return self._call_impl(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-05-30 18:28:27.890 |     return forward_call(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 132, in forward
2025-05-30 18:28:27.890 |     qkv, _ = self.qkv_proj(hidden_states)
2025-05-30 18:28:27.890 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-05-30 18:28:27.890 |     return self._call_impl(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-05-30 18:28:27.890 |     return forward_call(*args, **kwargs)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 486, in forward
2025-05-30 18:28:27.890 |     output_parallel = self.quant_method.apply(self, input_, bias)
2025-05-30 18:28:27.890 |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 202, in apply
2025-05-30 18:28:27.890 |     return dispatch_unquantized_gemm()(x, layer.weight, bias)
2025-05-30 18:28:27.890 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:27.890 | RuntimeError: CUDA error: no kernel image is available for execution on the device
2025-05-30 18:28:27.890 | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2025-05-30 18:28:27.890 | For debugging consider passing CUDA_LAUNCH_BLOCKING=1
2025-05-30 18:28:27.890 | Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-05-30 18:28:27.890 | 
2025-05-30 18:28:28.229 | [rank0]:[W530 16:28:28.201082936 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-05-30 18:28:28.561 | Traceback (most recent call last):
2025-05-30 18:28:28.561 |   File "<frozen runpy>", line 198, in _run_module_as_main
2025-05-30 18:28:28.561 |   File "<frozen runpy>", line 88, in _run_code
2025-05-30 18:28:28.561 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1376, in <module>
2025-05-30 18:28:28.561 |     uvloop.run(run_server(args))
2025-05-30 18:28:28.561 |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
2025-05-30 18:28:28.561 |     return __asyncio.run(
2025-05-30 18:28:28.561 |            ^^^^^^^^^^^^^^
2025-05-30 18:28:28.561 |   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
2025-05-30 18:28:28.561 |     return runner.run(main)
2025-05-30 18:28:28.561 |            ^^^^^^^^^^^^^^^^
2025-05-30 18:28:28.561 |   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
2025-05-30 18:28:28.561 |     return self._loop.run_until_complete(task)
2025-05-30 18:28:28.561 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:28.561 |   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
2025-05-30 18:28:28.561 |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
2025-05-30 18:28:28.561 |     return await main
2025-05-30 18:28:28.561 |            ^^^^^^^^^^
2025-05-30 18:28:28.561 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1324, in run_server
2025-05-30 18:28:28.562 |     async with build_async_engine_client(args) as engine_client:
2025-05-30 18:28:28.562 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:28.562 |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
2025-05-30 18:28:28.562 |     return await anext(self.gen)
2025-05-30 18:28:28.562 |            ^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:28.562 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 153, in build_async_engine_client
2025-05-30 18:28:28.562 |     async with build_async_engine_client_from_engine_args(
2025-05-30 18:28:28.562 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:28.562 |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
2025-05-30 18:28:28.562 |     return await anext(self.gen)
2025-05-30 18:28:28.562 |            ^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:28.562 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 185, in build_async_engine_client_from_engine_args
2025-05-30 18:28:28.562 |     async_llm = AsyncLLM.from_vllm_config(
2025-05-30 18:28:28.562 |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:28.562 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 157, in from_vllm_config
2025-05-30 18:28:28.562 |     return cls(
2025-05-30 18:28:28.562 |            ^^^^
2025-05-30 18:28:28.562 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 123, in __init__
2025-05-30 18:28:28.562 |     self.engine_core = core_client_class(
2025-05-30 18:28:28.562 |                        ^^^^^^^^^^^^^^^^^^
2025-05-30 18:28:28.562 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 734, in __init__
2025-05-30 18:28:28.563 |     super().__init__(
2025-05-30 18:28:28.563 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 418, in __init__
2025-05-30 18:28:28.563 |     self._wait_for_engine_startup(output_address, parallel_config)
2025-05-30 18:28:28.563 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 484, in _wait_for_engine_startup
2025-05-30 18:28:28.563 |     raise RuntimeError("Engine core initialization failed. "
2025-05-30 18:28:28.563 | RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
2025-05-30 18:28:31.972 | Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
2025-05-30 18:28:31.972 | Loading safetensors checkpoint shards:  50% Completed | 1/2 [01:51<01:51, 111.68s/it]
2025-05-30 18:28:31.972 | Loading safetensors checkpoint shards: 100% Completed | 2/2 [01:51<00:00, 55.84s/it]

logs without -- enforce-eager - no error just for reference

2025-05-30 18:32:56.968 | INFO 05-30 16:32:56 [__init__.py:243] Automatically detected platform cuda.
2025-05-30 18:32:58.832 | INFO 05-30 16:32:58 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-05-30 18:32:58.832 | INFO 05-30 16:32:58 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-05-30 18:32:58.832 | INFO 05-30 16:32:58 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-05-30 18:32:59.056 | WARNING 05-30 16:32:59 [utils.py:1367] argument 'enable_reasoning' is deprecated
2025-05-30 18:32:59.057 | INFO 05-30 16:32:59 [api_server.py:1289] vLLM API server version 0.9.0.1
2025-05-30 18:32:59.387 | INFO 05-30 16:32:59 [cli_args.py:300] non-default args: {'host': '0.0.0.0', 'model': '/models/Qwen3-1.7B', 'max_model_len': 30298, 'enable_reasoning': True, 'reasoning_parser': 'deepseek_r1'}
2025-05-30 18:33:03.932 | INFO 05-30 16:33:03 [config.py:793] This model supports multiple tasks: {'score', 'classify', 'generate', 'reward', 'embed'}. Defaulting to 'generate'.
2025-05-30 18:33:03.969 | INFO 05-30 16:33:03 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=2048.
2025-05-30 18:33:06.574 | INFO 05-30 16:33:06 [__init__.py:243] Automatically detected platform cuda.
2025-05-30 18:33:08.178 | INFO 05-30 16:33:08 [core.py:438] Waiting for init message from front-end.
2025-05-30 18:33:08.182 | INFO 05-30 16:33:08 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-05-30 18:33:08.182 | INFO 05-30 16:33:08 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-05-30 18:33:08.182 | INFO 05-30 16:33:08 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-05-30 18:33:08.183 | INFO 05-30 16:33:08 [core.py:65] Initializing a V1 LLM engine (v0.9.0.1) with config: model='/models/Qwen3-1.7B', speculative_config=None, tokenizer='/models/Qwen3-1.7B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=30298, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/models/Qwen3-1.7B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level": 3, "custom_ops": ["none"], "splitting_ops": ["vllm.unified_attention", "vllm.unified_attention_with_output"], "compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "use_cudagraph": true, "cudagraph_num_of_warmups": 1, "cudagraph_capture_sizes": [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 512}
2025-05-30 18:33:08.858 | WARNING 05-30 16:33:08 [utils.py:2671] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f1454743f20>
2025-05-30 18:33:09.179 | INFO 05-30 16:33:09 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
2025-05-30 18:33:09.181 | WARNING 05-30 16:33:09 [interface.py:344] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
2025-05-30 18:33:09.181 | INFO 05-30 16:33:09 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
2025-05-30 18:33:09.211 | INFO 05-30 16:33:09 [gpu_model_runner.py:1531] Starting to load model /models/Qwen3-1.7B...
2025-05-30 18:33:09.310 | INFO 05-30 16:33:09 [cuda.py:210] Using FlashInfer backend on V1 engine.
2025-05-30 18:33:09.403 | INFO 05-30 16:33:09 [backends.py:35] Using InductorAdaptor
2025-05-30 18:33:09.412 | INFO 05-30 16:33:09 [backends.py:35] Using InductorAdaptor
2025-05-30 18:33:09.418 | 
2025-05-30 18:34:55.732 | 
2025-05-30 18:34:55.738 | 
2025-05-30 18:34:55.738 |  
2025-05-30 18:34:55.874 | INFO 05-30 16:34:55 [default_loader.py:280] Loading weights took 106.46 seconds
2025-05-30 18:34:56.014 | INFO 05-30 16:34:56 [gpu_model_runner.py:1549] Model loading took 3.2152 GiB and 106.670335 seconds
2025-05-30 18:35:00.117 | INFO 05-30 16:35:00 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/ef6eee98c2/rank_0_0 for vLLM's torch.compile
2025-05-30 18:35:00.117 | INFO 05-30 16:35:00 [backends.py:469] Dynamo bytecode transform time: 4.10 s
2025-05-30 18:35:02.589 | INFO 05-30 16:35:02 [backends.py:158] Cache the graph of shape None for later use
2025-05-30 18:35:17.241 | INFO 05-30 16:35:17 [backends.py:170] Compiling a graph for general shape takes 16.92 s
2025-05-30 18:35:28.218 | INFO 05-30 16:35:28 [monitor.py:33] torch.compile takes 21.02 s in total
2025-05-30 18:35:36.633 | INFO 05-30 16:35:36 [kv_cache_utils.py:637] GPU KV cache size: 215,360 tokens
2025-05-30 18:35:36.633 | INFO 05-30 16:35:36 [kv_cache_utils.py:640] Maximum concurrency for 30,298 tokens per request: 7.11x
2025-05-30 18:35:36.952 | Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
2025-05-30 18:35:36.952 | Loading safetensors checkpoint shards:  50% Completed | 1/2 [01:46<01:46, 106.31s/it]
2025-05-30 18:35:36.952 | Loading safetensors checkpoint shards: 100% Completed | 2/2 [01:46<00:00, 53.16s/it]
2025-05-30 18:35:50.638 | INFO 05-30 16:35:50 [gpu_model_runner.py:1933] Graph capturing finished in 14 secs, took 0.89 GiB
2025-05-30 18:35:50.650 | INFO 05-30 16:35:50 [core.py:167] init engine (profile, create kv cache, warmup model) took 54.64 seconds
2025-05-30 18:35:50.966 | INFO 05-30 16:35:50 [loggers.py:134] vllm cache_config_info with initialization after num_gpu_blocks is: 13460
2025-05-30 18:35:50.987 | WARNING 05-30 16:35:50 [config.py:1339] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
2025-05-30 18:35:50.987 | INFO 05-30 16:35:50 [serving_chat.py:117] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [serving_completion.py:65] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [api_server.py:1336] Starting vLLM API server on http://0.0.0.0:8000
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:28] Available routes are:
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /openapi.json, Methods: HEAD, GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /docs, Methods: HEAD, GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: HEAD, GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /redoc, Methods: HEAD, GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /health, Methods: GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /load, Methods: GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /ping, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /ping, Methods: GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /tokenize, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /detokenize, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /v1/models, Methods: GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /version, Methods: GET
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /v1/chat/completions, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /v1/completions, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /v1/embeddings, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /pooling, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /classify, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /score, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /v1/score, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /rerank, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /v1/rerank, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /v2/rerank, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /invocations, Methods: POST
2025-05-30 18:35:50.992 | INFO 05-30 16:35:50 [launcher.py:36] Route: /metrics, Methods: GET
2025-05-30 18:35:51.027 | INFO:     Started server process [1]
2025-05-30 18:35:51.027 | INFO:     Waiting for application startup.
2025-05-30 18:35:51.159 | INFO:     Application startup complete.

[zicochaos]

export VLLM_ATTENTION_BACKEND=FLASHINFER - will make gibberish output on Blackwell
commenting it out will crash after first prompt with
"RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions."

[zicochaos]

not sure what worked but after this steps qwen3 is working for my on Blackwell ( RTX 5090 )

uv venv vllm5--python 3.12 --seed
source vllm5/bin/activate

uv self update

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

uv pip install vllm --torch-backend=auto

git clone https://github.com/flashinfer-ai/flashinfer.git --recursive
python -c "import torch; print(torch.version, torch.version.cuda)"
cd flashinfer

pip install ninja

export TORCH_CUDA_ARCH_LIST="7.5 8.6 8.9 12.0". # this is for my setup - Turing, Ampere, Ada Lovelace, Blackwell check what you need here https://en.wikipedia.org/wiki/CUDA

python -m flashinfer.aot

python -m pip install --no-build-isolation --verbose .

git clone https://github.com/vllm-project/vllm.git

cd vllm

python use_existing_torch.py

pip install -r requirements/build.txt

pip install --no-build-isolation -e .

[Rezzemy]

Yeah, that'd definitely be a way to go about it. Its a different environment context than what i'm dealing with here, but I've made a dockerfile to create a docker image to see if this works in a container, and will report back with either the same issue or a dockerfile when this finishes compiling and i can test it

[dan-dean]

Looking forward to your update. Docker interactions over the openai api are my use-case so I'm looking to have this bug resolved and in a future vllm-openai container.

[Rezzemy]

Looking forward to your update. Docker interactions over the openai api are my use-case so I'm looking to have this bug resolved and in a future vllm-openai container.

Haha, yeah, bleeding edge builds of flashinfer and vllm definitely work. the compile times aren't to scoff at though :/

While im here, is there a way to speed up model load on new containers? im using tiny qwen3 models for this issue, but i tend to spin up a 32b and that takes 15 minutes

dockerfile

my current working is 0.9.1.dev107+g2b102d51a.d20250601

FROM vllm/vllm-openai:v0.9.0.1

# Set environment variables
ENV TORCH_CUDA_ARCH_LIST="12.0"
ENV VLLM_ATTENTION_BACKEND=FLASHINFER
ENV MAX_JOBS=16

RUN pip install ninja
RUN git clone https://github.com/flashinfer-ai/flashinfer.git --recursive /tmp/flashinfer
WORKDIR /tmp/flashinfer
RUN python3 -m flashinfer.aot
RUN python3 -m pip install --no-build-isolation --verbose .


RUN git clone https://github.com/vllm-project/vllm.git /tmp/vllm
WORKDIR /tmp/vllm
ENV MAX_JOBS=8
RUN python3 use_existing_torch.py
RUN python3 -m pip install -r requirements/build.txt
RUN python3 -m pip install --no-build-isolation --verbose .
WORKDIR /

guess how long i waited for this to finish before i realized it hanged?

[Rezzemy]

FP8 quantized models fail to load.

basic stack trace

2025-06-01 06:40:45.242 | INFO 06-01 04:40:45 [__init__.py:243] Automatically detected platform cuda.
2025-06-01 06:40:47.270 | INFO 06-01 04:40:47 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-06-01 06:40:47.270 | INFO 06-01 04:40:47 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-06-01 06:40:47.270 | INFO 06-01 04:40:47 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-06-01 06:40:47.591 | WARNING 06-01 04:40:47 [utils.py:1414] argument 'enable_reasoning' is deprecated
2025-06-01 06:40:47.591 | INFO 06-01 04:40:47 [api_server.py:1276] vLLM API server version 0.9.1.dev107+g2b102d51a.d20250601
2025-06-01 06:40:47.813 | INFO 06-01 04:40:47 [cli_args.py:300] non-default args: {'host': '0.0.0.0', 'model': '/models/Qwen3-1.7B-FP8', 'max_model_len': 20000, 'enable_reasoning': True, 'reasoning_parser': 'deepseek_r1', 'block_size': 32, 'gpu_memory_utilization': 0.8, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'max_num_batched_tokens': 20000, 'scheduler_delay_factor': 8.0, 'enable_chunked_prefill': True}
2025-06-01 06:40:52.683 | INFO 06-01 04:40:52 [config.py:821] This model supports multiple tasks: {'classify', 'reward', 'embed', 'generate', 'score'}. Defaulting to 'generate'.
2025-06-01 06:40:53.278 | WARNING 06-01 04:40:53 [arg_utils.py:1587] --scheduler-delay-factor is not supported by the V1 Engine. Falling back to V0. We recommend to remove --scheduler-delay-factor from your config in favor of the V1 Engine.
2025-06-01 06:40:53.278 | INFO 06-01 04:40:53 [config.py:1545] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
2025-06-01 06:40:53.278 | INFO 06-01 04:40:53 [config.py:2161] Chunked prefill is enabled with max_num_batched_tokens=20000.
2025-06-01 06:40:53.407 | INFO 06-01 04:40:53 [api_server.py:266] Started engine process with PID 65
2025-06-01 06:40:55.353 | INFO 06-01 04:40:55 [__init__.py:243] Automatically detected platform cuda.
2025-06-01 06:40:57.045 | INFO 06-01 04:40:57 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-06-01 06:40:57.045 | INFO 06-01 04:40:57 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-06-01 06:40:57.045 | INFO 06-01 04:40:57 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-06-01 06:40:57.047 | INFO 06-01 04:40:57 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.1.dev107+g2b102d51a.d20250601) with config: model='/models/Qwen3-1.7B-FP8', speculative_config=None, tokenizer='/models/Qwen3-1.7B-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=20000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=fp8,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/models/Qwen3-1.7B-FP8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":false,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":256,"local_cache_dir":null}, use_cached_outputs=True, 
2025-06-01 06:40:57.433 | WARNING 06-01 04:40:57 [interface.py:346] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
2025-06-01 06:40:57.433 | INFO 06-01 04:40:57 [cuda.py:213] Using FlashInfer backend.
2025-06-01 06:40:58.051 | INFO 06-01 04:40:58 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
2025-06-01 06:40:58.057 | INFO 06-01 04:40:58 [model_runner.py:1170] Starting to load model /models/Qwen3-1.7B-FP8...
2025-06-01 06:40:58.247 | 
2025-06-01 06:42:28.895 | 
2025-06-01 06:42:28.895 | 
2025-06-01 06:42:28.895 | 
2025-06-01 06:42:28.974 | INFO 06-01 04:42:28 [default_loader.py:280] Loading weights took 90.73 seconds
2025-06-01 06:42:28.978 | WARNING 06-01 04:42:28 [kv_cache.py:85] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for the flash-attn backend.
2025-06-01 06:42:28.978 | WARNING 06-01 04:42:28 [kv_cache.py:98] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
2025-06-01 06:42:28.979 | WARNING 06-01 04:42:28 [kv_cache.py:129] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
2025-06-01 06:42:29.105 | INFO 06-01 04:42:29 [model_runner.py:1202] Model loading took 1.9030 GiB and 90.945059 seconds
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457] Error Internal
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457] Traceback (most recent call last):
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 445, in run_mp_engine
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     engine = MQLLMEngine.from_vllm_config(
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 132, in from_vllm_config
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return cls(
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 86, in __init__
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     self.engine = LLMEngine(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 268, in __init__
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     self._initialize_kv_caches()
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 413, in _initialize_kv_caches
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     self.model_executor.determine_num_available_blocks())
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 103, in determine_num_available_blocks
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     results = self.collective_rpc("determine_num_available_blocks")
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     answer = run_method(self.driver_worker, method, args, kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2648, in run_method
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return func(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return func(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 255, in determine_num_available_blocks
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     self.model_runner.profile_run()
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return func(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1299, in profile_run
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     self._dummy_run(max_num_batched_tokens, max_num_seqs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1425, in _dummy_run
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     self.execute_model(model_input, kv_caches, intermediate_tensors)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return func(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1843, in execute_model
2025-06-01 06:42:29.680 | Process SpawnProcess-1:
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     hidden_or_intermediate_states = model_executable(
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]                                     ^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return self._call_impl(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return forward_call(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 172, in __call__
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return self.forward(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 353, in forward
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     hidden_states, residual = layer(
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]                               ^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return self._call_impl(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return forward_call(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 213, in forward
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     hidden_states = self.self_attn(
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]                     ^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return self._call_impl(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return forward_call(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 132, in forward
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     qkv, _ = self.qkv_proj(hidden_states)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return self._call_impl(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return forward_call(*args, **kwargs)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 486, in forward
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     output_parallel = self.quant_method.apply(self, input_, bias)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py", line 404, in apply
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return torch.ops.vllm.apply_w8a8_block_fp8_linear(
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return self._op(*args, **(kwargs or {}))
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 152, in apply_w8a8_block_fp8_linear
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     output = w8a8_blockscale_func(q_input, weight, x_scale, weight_scale,
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 38, in cutlass_scaled_mm
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return ops.cutlass_scaled_mm(A,
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/vllm/_custom_ops.py", line 677, in cutlass_scaled_mm
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]     return self._op(*args, **(kwargs or {}))
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.680 | ERROR 06-01 04:42:29 [engine.py:457] RuntimeError: Error Internal
2025-06-01 06:42:29.680 | Traceback (most recent call last):
2025-06-01 06:42:29.681 |   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
2025-06-01 06:42:29.681 |     self.run()
2025-06-01 06:42:29.681 |   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
2025-06-01 06:42:29.681 |     self._target(*self._args, **self._kwargs)
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 459, in run_mp_engine
2025-06-01 06:42:29.681 |     raise e from None
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 445, in run_mp_engine
2025-06-01 06:42:29.681 |     engine = MQLLMEngine.from_vllm_config(
2025-06-01 06:42:29.681 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 132, in from_vllm_config
2025-06-01 06:42:29.681 |     return cls(
2025-06-01 06:42:29.681 |            ^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 86, in __init__
2025-06-01 06:42:29.681 |     self.engine = LLMEngine(*args, **kwargs)
2025-06-01 06:42:29.681 |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 268, in __init__
2025-06-01 06:42:29.681 |     self._initialize_kv_caches()
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 413, in _initialize_kv_caches
2025-06-01 06:42:29.681 |     self.model_executor.determine_num_available_blocks())
2025-06-01 06:42:29.681 |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 103, in determine_num_available_blocks
2025-06-01 06:42:29.681 |     results = self.collective_rpc("determine_num_available_blocks")
2025-06-01 06:42:29.681 |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
2025-06-01 06:42:29.681 |     answer = run_method(self.driver_worker, method, args, kwargs)
2025-06-01 06:42:29.681 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2648, in run_method
2025-06-01 06:42:29.681 |     return func(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 06:42:29.681 |     return func(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 255, in determine_num_available_blocks
2025-06-01 06:42:29.681 |     self.model_runner.profile_run()
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 06:42:29.681 |     return func(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1299, in profile_run
2025-06-01 06:42:29.681 |     self._dummy_run(max_num_batched_tokens, max_num_seqs)
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1425, in _dummy_run
2025-06-01 06:42:29.681 |     self.execute_model(model_input, kv_caches, intermediate_tensors)
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 06:42:29.681 |     return func(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1843, in execute_model
2025-06-01 06:42:29.681 |     hidden_or_intermediate_states = model_executable(
2025-06-01 06:42:29.681 |                                     ^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 06:42:29.681 |     return self._call_impl(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 06:42:29.681 |     return forward_call(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
2025-06-01 06:42:29.681 |     hidden_states = self.model(input_ids, positions, intermediate_tensors,
2025-06-01 06:42:29.681 |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 172, in __call__
2025-06-01 06:42:29.681 |     return self.forward(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 353, in forward
2025-06-01 06:42:29.681 |     hidden_states, residual = layer(
2025-06-01 06:42:29.681 |                               ^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 06:42:29.681 |     return self._call_impl(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 06:42:29.681 |     return forward_call(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 213, in forward
2025-06-01 06:42:29.681 |     hidden_states = self.self_attn(
2025-06-01 06:42:29.681 |                     ^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 06:42:29.681 |     return self._call_impl(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 06:42:29.681 |     return forward_call(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 132, in forward
2025-06-01 06:42:29.681 |     qkv, _ = self.qkv_proj(hidden_states)
2025-06-01 06:42:29.681 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 06:42:29.681 |     return self._call_impl(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 06:42:29.681 |     return forward_call(*args, **kwargs)
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 486, in forward
2025-06-01 06:42:29.681 |     output_parallel = self.quant_method.apply(self, input_, bias)
2025-06-01 06:42:29.681 |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py", line 404, in apply
2025-06-01 06:42:29.681 |     return torch.ops.vllm.apply_w8a8_block_fp8_linear(
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
2025-06-01 06:42:29.681 |     return self._op(*args, **(kwargs or {}))
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 152, in apply_w8a8_block_fp8_linear
2025-06-01 06:42:29.681 |     output = w8a8_blockscale_func(q_input, weight, x_scale, weight_scale,
2025-06-01 06:42:29.681 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 38, in cutlass_scaled_mm
2025-06-01 06:42:29.681 |     return ops.cutlass_scaled_mm(A,
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/vllm/_custom_ops.py", line 677, in cutlass_scaled_mm
2025-06-01 06:42:29.681 |     torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
2025-06-01 06:42:29.681 |   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
2025-06-01 06:42:29.681 |     return self._op(*args, **(kwargs or {}))
2025-06-01 06:42:29.681 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:29.681 | RuntimeError: Error Internal
2025-06-01 06:42:29.927 | [rank0]:[W601 04:42:29.442855531 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-06-01 06:42:33.803 | Traceback (most recent call last):
2025-06-01 06:42:33.803 |   File "<frozen runpy>", line 198, in _run_module_as_main
2025-06-01 06:42:33.803 |   File "<frozen runpy>", line 88, in _run_code
2025-06-01 06:42:33.803 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1371, in <module>
2025-06-01 06:42:33.804 |     uvloop.run(run_server(args))
2025-06-01 06:42:33.804 |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
2025-06-01 06:42:33.804 |     return __asyncio.run(
2025-06-01 06:42:33.804 |            ^^^^^^^^^^^^^^
2025-06-01 06:42:33.804 |   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
2025-06-01 06:42:33.804 |     return runner.run(main)
2025-06-01 06:42:33.804 |            ^^^^^^^^^^^^^^^^
2025-06-01 06:42:33.804 |   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
2025-06-01 06:42:33.804 |     return self._loop.run_until_complete(task)
2025-06-01 06:42:33.804 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:33.804 |   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
2025-06-01 06:42:33.804 |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
2025-06-01 06:42:33.804 |     return await main
2025-06-01 06:42:33.804 |            ^^^^^^^^^^
2025-06-01 06:42:33.804 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1312, in run_server
2025-06-01 06:42:33.805 |     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
2025-06-01 06:42:33.805 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1327, in run_server_worker
2025-06-01 06:42:33.805 |     async with build_async_engine_client(args, client_config) as engine_client:
2025-06-01 06:42:33.805 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:33.805 |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
2025-06-01 06:42:33.805 |     return await anext(self.gen)
2025-06-01 06:42:33.805 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:33.805 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 156, in build_async_engine_client
2025-06-01 06:42:33.805 |     async with build_async_engine_client_from_engine_args(
2025-06-01 06:42:33.805 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:33.805 |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
2025-06-01 06:42:33.805 |     return await anext(self.gen)
2025-06-01 06:42:33.805 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 06:42:33.805 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 289, in build_async_engine_client_from_engine_args
2025-06-01 06:42:33.805 |     raise RuntimeError(
2025-06-01 06:42:33.805 | RuntimeError: Engine process failed to start. See stack trace for the root cause.
2025-06-01 06:43:21.130 | Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
2025-06-01 06:43:21.130 | Loading safetensors checkpoint shards: 100% Completed | 1/1 [01:30<00:00, 90.65s/it]
2025-06-01 06:43:21.130 | Loading safetensors checkpoint shards: 100% Completed | 1/1 [01:30<00:00, 90.65s/it]

[zicochaos]

did you try run it without "--scheduler-delay-factor is not supported by the V1 Engine" "Falling back to V0"

[Rezzemy]

did you try run it without "--scheduler-delay-factor is not supported by the V1 Engine" "Falling back to V0"

yeah

2025-06-01 07:57:13.470 | INFO 06-01 05:57:13 [__init__.py:243] Automatically detected platform cuda.
2025-06-01 07:57:15.060 | INFO 06-01 05:57:15 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-06-01 07:57:15.060 | INFO 06-01 05:57:15 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-06-01 07:57:15.060 | INFO 06-01 05:57:15 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-06-01 07:57:15.370 | WARNING 06-01 05:57:15 [utils.py:1414] argument 'enable_reasoning' is deprecated
2025-06-01 07:57:15.371 | INFO 06-01 05:57:15 [api_server.py:1276] vLLM API server version 0.9.1.dev107+g2b102d51a.d20250601
2025-06-01 07:57:15.591 | INFO 06-01 05:57:15 [cli_args.py:300] non-default args: {'host': '0.0.0.0', 'model': '/models/Qwen3-1.7B-FP8', 'max_model_len': 20000, 'enable_reasoning': True, 'reasoning_parser': 'deepseek_r1', 'block_size': 32, 'gpu_memory_utilization': 0.8, 'enable_prefix_caching': True, 'max_num_batched_tokens': 20000, 'enable_chunked_prefill': True}
2025-06-01 07:57:20.137 | INFO 06-01 05:57:20 [config.py:821] This model supports multiple tasks: {'classify', 'reward', 'score', 'generate', 'embed'}. Defaulting to 'generate'.
2025-06-01 07:57:20.729 | INFO 06-01 05:57:20 [config.py:2161] Chunked prefill is enabled with max_num_batched_tokens=20000.
2025-06-01 07:57:22.995 | INFO 06-01 05:57:22 [__init__.py:243] Automatically detected platform cuda.
2025-06-01 07:57:24.595 | INFO 06-01 05:57:24 [core.py:448] Waiting for init message from front-end.
2025-06-01 07:57:24.599 | INFO 06-01 05:57:24 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-06-01 07:57:24.599 | INFO 06-01 05:57:24 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-06-01 07:57:24.599 | INFO 06-01 05:57:24 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-06-01 07:57:24.600 | INFO 06-01 05:57:24 [core.py:68] Initializing a V1 LLM engine (v0.9.1.dev107+g2b102d51a.d20250601) with config: model='/models/Qwen3-1.7B-FP8', speculative_config=None, tokenizer='/models/Qwen3-1.7B-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=20000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/models/Qwen3-1.7B-FP8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
2025-06-01 07:57:25.139 | WARNING 06-01 05:57:25 [utils.py:2714] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f41a77f2090>
2025-06-01 07:57:25.583 | INFO 06-01 05:57:25 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
2025-06-01 07:57:25.585 | WARNING 06-01 05:57:25 [interface.py:346] Using 'pin_memory=False' as WSL is detected. This may slow down the performance.
2025-06-01 07:57:25.585 | INFO 06-01 05:57:25 [topk_topp_sampler.py:48] Using FlashInfer for top-p & top-k sampling.
2025-06-01 07:57:25.610 | INFO 06-01 05:57:25 [gpu_model_runner.py:1549] Starting to load model /models/Qwen3-1.7B-FP8...
2025-06-01 07:57:25.705 | INFO 06-01 05:57:25 [cuda.py:202] Using FlashInfer backend on V1 engine.
2025-06-01 07:57:25.785 | INFO 06-01 05:57:25 [backends.py:37] Using InductorAdaptor
2025-06-01 07:57:25.794 | INFO 06-01 05:57:25 [backends.py:37] Using InductorAdaptor
2025-06-01 07:57:25.798 | 
2025-06-01 07:58:58.035 | 
2025-06-01 07:58:58.035 | 
2025-06-01 07:58:58.035 | 
2025-06-01 07:58:58.121 | INFO 06-01 05:58:58 [default_loader.py:280] Loading weights took 92.33 seconds
2025-06-01 07:58:58.251 | INFO 06-01 05:58:58 [gpu_model_runner.py:1567] Model loading took 1.9030 GiB and 92.529116 seconds
2025-06-01 07:59:02.269 | INFO 06-01 05:59:02 [backends.py:461] Using cache directory: /root/.cache/vllm/torch_compile_cache/52a152f492/rank_0_0 for vLLM's torch.compile
2025-06-01 07:59:02.269 | INFO 06-01 05:59:02 [backends.py:471] Dynamo bytecode transform time: 4.02 s
2025-06-01 07:59:04.710 | INFO 06-01 05:59:04 [backends.py:160] Cache the graph of shape None for later use
2025-06-01 07:59:19.420 | INFO 06-01 05:59:19 [backends.py:172] Compiling a graph for general shape takes 16.92 s
2025-06-01 07:59:24.242 | ERROR 06-01 05:59:24 [core.py:508] EngineCore failed to start.
2025-06-01 07:59:24.242 | ERROR 06-01 05:59:24 [core.py:508] Traceback (most recent call last):
2025-06-01 07:59:24.242 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 499, in run_engine_core
2025-06-01 07:59:24.242 | ERROR 06-01 05:59:24 [core.py:508]     engine_core = EngineCoreProc(*args, **kwargs)
2025-06-01 07:59:24.242 | ERROR 06-01 05:59:24 [core.py:508]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.242 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 398, in __init__
2025-06-01 07:59:24.242 | ERROR 06-01 05:59:24 [core.py:508]     super().__init__(vllm_config, executor_class, log_stats,
2025-06-01 07:59:24.242 | Process EngineCore_0:
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 81, in __init__
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     self._initialize_kv_caches(vllm_config)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 139, in _initialize_kv_caches
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     available_gpu_memory = self.model_executor.determine_available_memory()
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     output = self.collective_rpc("determine_available_memory")
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     answer = run_method(self.driver_worker, method, args, kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2648, in run_method
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return func(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return func(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     self.model_runner.profile_run()
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1918, in profile_run
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     hidden_states = self._dummy_run(self.max_num_tokens)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return func(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1753, in _dummy_run
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     outputs = model(
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]               ^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return self._call_impl(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return forward_call(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 238, in __call__
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     output = self.compiled_callable(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return fn(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 335, in forward
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     def forward(
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return self._call_impl(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return forward_call(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return fn(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return self._wrapped_call(self, *args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     raise e
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return self._call_impl(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return forward_call(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "<eval_with_key>.58", line 346, in forward
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     submod_0 = self.submod_0(l_input_ids_, s0, l_self_modules_embed_tokens_parameters_weight_, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_scale_inv_, l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  l_input_ids_ = l_self_modules_embed_tokens_parameters_weight_ = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_scale_inv_ = l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_ = None
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_piecewise_backend.py", line 110, in __call__
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return self.compiled_graph_for_general_shape(*args)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return fn(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1201, in forward
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return compiled_fn(full_args)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 328, in runtime_wrapper
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     all_outs = call_func_at_runtime_with_args(
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     out = normalize_as_list(f(args))
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]                             ^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 689, in inner_fn
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     outs = compiled_fn(args)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 495, in wrapper
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return compiled_fn(runtime_args)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 460, in __call__
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return self.current_callable(inputs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 2404, in run
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return model(new_inputs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/root/.cache/vllm/torch_compile_cache/52a152f492/rank_0_0/inductor_cache/a3/ca3gwi2ucdbpyfn2h5y5xzr7wc2sabyrzepflojfvkluggp4zysv.py", line 435, in call
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     buf3 = torch.ops.vllm.apply_w8a8_block_fp8_linear.default(buf2, arg4_1, [128, 128], arg5_1)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 756, in __call__
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return self._op(*args, **kwargs)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 152, in apply_w8a8_block_fp8_linear
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     output = w8a8_blockscale_func(q_input, weight, x_scale, weight_scale,
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 38, in cutlass_scaled_mm
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return ops.cutlass_scaled_mm(A,
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/vllm/_custom_ops.py", line 677, in cutlass_scaled_mm
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]     return self._op(*args, **(kwargs or {}))
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.243 | ERROR 06-01 05:59:24 [core.py:508] RuntimeError: Error Internal
2025-06-01 07:59:24.244 | Traceback (most recent call last):
2025-06-01 07:59:24.244 |   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
2025-06-01 07:59:24.244 |     self.run()
2025-06-01 07:59:24.244 |   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
2025-06-01 07:59:24.244 |     self._target(*self._args, **self._kwargs)
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 512, in run_engine_core
2025-06-01 07:59:24.244 |     raise e
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 499, in run_engine_core
2025-06-01 07:59:24.244 |     engine_core = EngineCoreProc(*args, **kwargs)
2025-06-01 07:59:24.244 |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 398, in __init__
2025-06-01 07:59:24.244 |     super().__init__(vllm_config, executor_class, log_stats,
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 81, in __init__
2025-06-01 07:59:24.244 |     self._initialize_kv_caches(vllm_config)
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 139, in _initialize_kv_caches
2025-06-01 07:59:24.244 |     available_gpu_memory = self.model_executor.determine_available_memory()
2025-06-01 07:59:24.244 |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
2025-06-01 07:59:24.244 |     output = self.collective_rpc("determine_available_memory")
2025-06-01 07:59:24.244 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
2025-06-01 07:59:24.244 |     answer = run_method(self.driver_worker, method, args, kwargs)
2025-06-01 07:59:24.244 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2648, in run_method
2025-06-01 07:59:24.244 |     return func(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 07:59:24.244 |     return func(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
2025-06-01 07:59:24.244 |     self.model_runner.profile_run()
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1918, in profile_run
2025-06-01 07:59:24.244 |     hidden_states = self._dummy_run(self.max_num_tokens)
2025-06-01 07:59:24.244 |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-06-01 07:59:24.244 |     return func(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1753, in _dummy_run
2025-06-01 07:59:24.244 |     outputs = model(
2025-06-01 07:59:24.244 |               ^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 07:59:24.244 |     return self._call_impl(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 07:59:24.244 |     return forward_call(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3.py", line 300, in forward
2025-06-01 07:59:24.244 |     hidden_states = self.model(input_ids, positions, intermediate_tensors,
2025-06-01 07:59:24.244 |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 238, in __call__
2025-06-01 07:59:24.244 |     output = self.compiled_callable(*args, **kwargs)
2025-06-01 07:59:24.244 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
2025-06-01 07:59:24.244 |     return fn(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 335, in forward
2025-06-01 07:59:24.244 |     def forward(
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 07:59:24.244 |     return self._call_impl(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 07:59:24.244 |     return forward_call(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
2025-06-01 07:59:24.244 |     return fn(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 830, in call_wrapped
2025-06-01 07:59:24.244 |     return self._wrapped_call(self, *args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 406, in __call__
2025-06-01 07:59:24.244 |     raise e
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 393, in __call__
2025-06-01 07:59:24.244 |     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-06-01 07:59:24.244 |     return self._call_impl(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-06-01 07:59:24.244 |     return forward_call(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "<eval_with_key>.58", line 346, in forward
2025-06-01 07:59:24.244 |     submod_0 = self.submod_0(l_input_ids_, s0, l_self_modules_embed_tokens_parameters_weight_, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_scale_inv_, l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  l_input_ids_ = l_self_modules_embed_tokens_parameters_weight_ = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_scale_inv_ = l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight_ = None
2025-06-01 07:59:24.244 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_piecewise_backend.py", line 110, in __call__
2025-06-01 07:59:24.244 |     return self.compiled_graph_for_general_shape(*args)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
2025-06-01 07:59:24.244 |     return fn(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1201, in forward
2025-06-01 07:59:24.244 |     return compiled_fn(full_args)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 328, in runtime_wrapper
2025-06-01 07:59:24.244 |     all_outs = call_func_at_runtime_with_args(
2025-06-01 07:59:24.244 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
2025-06-01 07:59:24.244 |     out = normalize_as_list(f(args))
2025-06-01 07:59:24.244 |                             ^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 689, in inner_fn
2025-06-01 07:59:24.244 |     outs = compiled_fn(args)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 495, in wrapper
2025-06-01 07:59:24.244 |     return compiled_fn(runtime_args)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 460, in __call__
2025-06-01 07:59:24.244 |     return self.current_callable(inputs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 2404, in run
2025-06-01 07:59:24.244 |     return model(new_inputs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/root/.cache/vllm/torch_compile_cache/52a152f492/rank_0_0/inductor_cache/a3/ca3gwi2ucdbpyfn2h5y5xzr7wc2sabyrzepflojfvkluggp4zysv.py", line 435, in call
2025-06-01 07:59:24.244 |     buf3 = torch.ops.vllm.apply_w8a8_block_fp8_linear.default(buf2, arg4_1, [128, 128], arg5_1)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 756, in __call__
2025-06-01 07:59:24.244 |     return self._op(*args, **kwargs)
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 152, in apply_w8a8_block_fp8_linear
2025-06-01 07:59:24.244 |     output = w8a8_blockscale_func(q_input, weight, x_scale, weight_scale,
2025-06-01 07:59:24.244 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 38, in cutlass_scaled_mm
2025-06-01 07:59:24.244 |     return ops.cutlass_scaled_mm(A,
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/vllm/_custom_ops.py", line 677, in cutlass_scaled_mm
2025-06-01 07:59:24.244 |     torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
2025-06-01 07:59:24.244 |   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1158, in __call__
2025-06-01 07:59:24.244 |     return self._op(*args, **(kwargs or {}))
2025-06-01 07:59:24.244 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:24.244 | RuntimeError: Error Internal
2025-06-01 07:59:24.759 | [rank0]:[W601 05:59:24.480560060 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-06-01 07:59:25.304 | Traceback (most recent call last):
2025-06-01 07:59:25.304 |   File "<frozen runpy>", line 198, in _run_module_as_main
2025-06-01 07:59:25.304 |   File "<frozen runpy>", line 88, in _run_code
2025-06-01 07:59:25.304 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1371, in <module>
2025-06-01 07:59:25.304 |     uvloop.run(run_server(args))
2025-06-01 07:59:25.304 |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
2025-06-01 07:59:25.304 |     return __asyncio.run(
2025-06-01 07:59:25.304 |            ^^^^^^^^^^^^^^
2025-06-01 07:59:25.304 |   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
2025-06-01 07:59:25.304 |     return runner.run(main)
2025-06-01 07:59:25.304 |            ^^^^^^^^^^^^^^^^
2025-06-01 07:59:25.304 |   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
2025-06-01 07:59:25.304 |     return self._loop.run_until_complete(task)
2025-06-01 07:59:25.304 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:25.304 |   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
2025-06-01 07:59:25.304 |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
2025-06-01 07:59:25.304 |     return await main
2025-06-01 07:59:25.304 |            ^^^^^^^^^^
2025-06-01 07:59:25.304 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1312, in run_server
2025-06-01 07:59:25.305 |     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
2025-06-01 07:59:25.305 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1327, in run_server_worker
2025-06-01 07:59:25.305 |     async with build_async_engine_client(args, client_config) as engine_client:
2025-06-01 07:59:25.305 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:25.305 |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
2025-06-01 07:59:25.305 |     return await anext(self.gen)
2025-06-01 07:59:25.305 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:25.305 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 156, in build_async_engine_client
2025-06-01 07:59:25.305 |     async with build_async_engine_client_from_engine_args(
2025-06-01 07:59:25.305 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:25.305 |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
2025-06-01 07:59:25.305 |     return await anext(self.gen)
2025-06-01 07:59:25.305 |            ^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:25.305 |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 192, in build_async_engine_client_from_engine_args
2025-06-01 07:59:25.305 |     async_llm = AsyncLLM.from_vllm_config(
2025-06-01 07:59:25.305 |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:25.305 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 164, in from_vllm_config
2025-06-01 07:59:25.306 |     return cls(
2025-06-01 07:59:25.306 |            ^^^^
2025-06-01 07:59:25.306 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 126, in __init__
2025-06-01 07:59:25.306 |     self.engine_core = core_client_class(
2025-06-01 07:59:25.306 |                        ^^^^^^^^^^^^^^^^^^
2025-06-01 07:59:25.306 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 692, in __init__
2025-06-01 07:59:25.306 |     super().__init__(
2025-06-01 07:59:25.306 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 403, in __init__
2025-06-01 07:59:25.306 |     self._init_engines_direct(vllm_config, local_only,
2025-06-01 07:59:25.306 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 472, in _init_engines_direct
2025-06-01 07:59:25.306 |     self._wait_for_engine_startup(handshake_socket, input_address,
2025-06-01 07:59:25.306 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 487, in _wait_for_engine_startup
2025-06-01 07:59:25.306 |     wait_for_engine_startup(
2025-06-01 07:59:25.306 |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/utils.py", line 327, in wait_for_engine_startup
2025-06-01 07:59:25.307 |     raise RuntimeError("Engine core initialization failed. "
2025-06-01 07:59:25.307 | RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
2025-06-01 07:59:29.910 | Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
2025-06-01 07:59:29.910 | Loading safetensors checkpoint shards: 100% Completed | 1/1 [01:32<00:00, 92.24s/it]
2025-06-01 07:59:29.910 | Loading safetensors checkpoint shards: 100% Completed | 1/1 [01:32<00:00, 92.24s/it]

[Tendo33]

I also encountered the same issue. Object of type PostGradPassManager is not JSON serializable