[Bug]: After updating VLLM from 0.4.0.post1 to 0.6.4, the model loading time increased by one minute.

[123qwe-ux]

Your current environment

The output of `python collect_env.py`

2025-01-17 06:31:45.229125: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-17 06:31:45.244775: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1737095505.263778  524365 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1737095505.269663  524365 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-17 06:31:45.289033: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.11.9 (main, Aug 30 2024, 10:28:30) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-4.18.0-553.5.1.el8.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 4090
GPU 1: NVIDIA GeForce RTX 4090

Nvidia driver version: 550.107.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.3.0
/usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             112
On-line CPU(s) list:                0-111
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Platinum 8273CL CPU @ 2.20GHz
CPU family:                         6
Model:                              85
Thread(s) per core:                 2
Core(s) per socket:                 28
Socket(s):                          2
Stepping:                           7
CPU max MHz:                        3700.0000
CPU min MHz:                        1000.0000
BogoMIPS:                           4400.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Virtualization:                     VT-x
L1d cache:                          1.8 MiB (56 instances)
L1i cache:                          1.8 MiB (56 instances)
L2 cache:                           56 MiB (56 instances)
L3 cache:                           77 MiB (2 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-27,56-83
NUMA node1 CPU(s):                  28-55,84-111
Vulnerability Gather data sampling: Vulnerable: No microcode
Vulnerability Itlb multihit:        KVM: Mitigation: VMX disabled
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Vulnerability Retbleed:             Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; TSX disabled

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvcc-cu12==12.3.107
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] onnxruntime==1.19.0
[pip3] optree==0.13.1
[pip3] pynvml==12.0.0
[pip3] pyzmq==26.2.0
[pip3] qtorch==0.3.0
[pip3] sentence-transformers==3.3.0
[pip3] torch==2.5.1
[pip3] torchaudio==2.1.2
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.0
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.4
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     0-27,56-83      0               N/A
GPU1    SYS      X      28-55,84-111    1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

LD_LIBRARY_PATH=/home/1473391854/.local/lib/python3.11/site-packages/cv2/../../lib64:/usr/local/python3.9.19/lib:/usr/local/openssl/lib:/usr/local/python3.11.9/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

No response

🐛 Describe the bug

When my vllm version was 0.4.0.post1, it only took me 35 seconds to load Qwen2.5-7B-Instruct through the following code, but when I upgraded the vllm version to 0.6.4, this loading time increased sharply to 1 minute and 34 seconds. What is the situation here?
More importantly, after the problem is solved, I also wonder if there is any way to accelerate the speed of model loading on the local, regardless of whether the vllm version is 0.4.0.post1 or 0.6.4. At present, the model loading time is normally between 30 and 60 seconds, but this is still much longer compared to the model loading using AutoModelForCausalLM.from_pretrained()

The specific code for loading the model:

@ray.remote
class VLLM_Qwen_Chat_Ray:
    def __init__(self):
        try:
            self.model_name_or_path = "/data/model/Qwen2.5-7B-Instruct"
            self.sampling_params = vllm.SamplingParams(
                temperature=0.01,
                top_p=0.8,
                repetition_penalty=1.05,
                max_tokens=1024
            )
            self.llm = vllm.LLM(
                model=self.model_name_or_path,
                tensor_parallel_size=2,
                gpu_memory_utilization=0.98
            )
            self.ready = True
        except Exception as e:
            self.ready = False
            logger.error(f"An error occurred when initializing VLLM_Qwen_Chat_Ray: {e}")

    def is_ready(self) -> bool:
        """
        Confirm whether the Actor has completed initialization
        """
        # If the ready attribute of the Actor exists and is True, return True
        return getattr(self, 'ready', False)


logger.info("Loading the model from local...")

timeout = 200  # 200-second timeout
start_time = time.time()

vllm_model = VLLM_Qwen_Chat_Ray.remote()
while not ray.get(vllm_model.is_ready.remote()):
    if time.time() - start_time > timeout:
        logger.error("Model loading timed out")
        raise TimeoutError("Model loading timed out")
    time.sleep(0.1)
logger.info("Completed loading the local model")

The content output during the loading process:

2025-01-17 08:06:48.735090: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-17 08:06:48.750279: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1737101208.768695  609349 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1737101208.775499  609349 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-17 08:06:48.798013: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO:llm_poc.vllm_model.vllm_qwen_chat_ray:Loading the model from local...
2025-01-17 08:06:57,453	INFO worker.py:1810 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265/ 
(pid=609877) 2025-01-17 08:07:05.125709: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
(pid=609877) 2025-01-17 08:07:05.141449: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(pid=609877) WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
(pid=609877) E0000 00:00:1737101225.159832  609877 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(pid=609877) E0000 00:00:1737101225.165678  609877 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(pid=609877) 2025-01-17 08:07:05.185276: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
(pid=609877) To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:18 config.py:350] This model supports multiple tasks: {'embedding', 'generate'}. Defaulting to 'generate'.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:18 config.py:1020] Defaulting to use ray for distributed inference
(VLLM_Qwen_Chat_Ray pid=609877) Calling ray.init() again after it has already been called.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:21 llm_engine.py:249] Initializing an LLM engine (v0.6.4) with config: model='/data/model/Qwen2.5-7B-Instruct', speculative_config=None, tokenizer='/data/model/Qwen2.5-7B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/data/model/Qwen2.5-7B-Instruct, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, chat_template_text_format=string, mm_processor_kwargs=None, pooler_config=None)
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:22 ray_gpu_executor.py:134] use_ray_spmd_worker: False
(pid=609879) 2025-01-17 08:07:27.391161: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
(pid=609879) 2025-01-17 08:07:27.407232: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(pid=609879) WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
(pid=609879) E0000 00:00:1737101247.425768  609879 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(pid=609879) E0000 00:00:1737101247.431956  609879 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(pid=609879) 2025-01-17 08:07:27.451432: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
(pid=609879) To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
(pid=609874) 2025-01-17 08:07:35.465091: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
(pid=609874) 2025-01-17 08:07:35.480816: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
(pid=609874) WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
(pid=609874) E0000 00:00:1737101255.499352  609874 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
(pid=609874) E0000 00:00:1737101255.505145  609874 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
(pid=609874) 2025-01-17 08:07:35.524575: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
(pid=609874) To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:38 selector.py:135] Using Flash Attention backend.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:39 utils.py:960] Found nccl from library libnccl.so.2
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:39 pynccl.py:69] vLLM is using nccl==2.21.5
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:39 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/1473391854/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VLLM_Qwen_Chat_Ray pid=609877) WARNING 01-17 08:07:39 custom_all_reduce.py:143] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:39 shm_broadcast.py:236] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7fccb1cf8e50>, local_subscribe_port=52969, remote_subscribe_port=None)
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:39 model_runner.py:1072] Starting to load model /data/model/Qwen2.5-7B-Instruct...
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:02,  1.38it/s]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.26it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.15it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.12it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.16it/s]
(VLLM_Qwen_Chat_Ray pid=609877) 
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:43 model_runner.py:1077] Loading model weights took 7.1216 GB
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:38 selector.py:135] Using Flash Attention backend.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:47 worker.py:232] Memory profiling results: total_gpu_memory=23.64GiB initial_memory_usage=7.82GiB peak_torch_memory=9.74GiB memory_usage_post_profile=7.89Gib non_torch_memory=0.76GiB kv_cache_size=12.67GiB gpu_memory_utilization=0.98
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:39 utils.py:960] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:39 pynccl.py:69] vLLM is using nccl==2.21.5
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:39 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/1473391854/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(RayWorkerWrapper pid=609874) WARNING 01-17 08:07:39 custom_all_reduce.py:143] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:39 model_runner.py:1072] Starting to load model /data/model/Qwen2.5-7B-Instruct...
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:48 distributed_gpu_executor.py:57] # GPU blocks: 29659, # CPU blocks: 9362
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:48 distributed_gpu_executor.py:61] Maximum concurrency for 32768 tokens per request: 14.48x
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:51 model_runner.py:1400] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:51 model_runner.py:1404] If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:43 model_runner.py:1077] Loading model weights took 7.1216 GB
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:08:15 model_runner.py:1518] Graph capturing finished in 23 secs, took 0.42 GiB
(RayWorkerWrapper pid=609874) INFO 01-17 08:07:47 worker.py:232] Memory profiling results: total_gpu_memory=23.64GiB initial_memory_usage=7.80GiB peak_torch_memory=9.74GiB memory_usage_post_profile=7.87Gib non_torch_memory=0.74GiB kv_cache_size=12.69GiB gpu_memory_utilization=0.98
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:51 model_runner.py:1400] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VLLM_Qwen_Chat_Ray pid=609877) INFO 01-17 08:07:51 model_runner.py:1404] If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO:llm_poc.vllm_model.vllm_qwen_chat_ray:Completed loading the local model

@youkaichao

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[youkaichao]

did you try more recent versions like 0.6.6?

cc @DarkLight1337 I remember there was a bug that the loader will directly consume the generator and cause longer loading time. but it seems #9160 is merged in 0.6.3.post1 .

[DarkLight1337]

From my experience, that bug no longer occurs. However the startup time has increased recently because of CUDA graph capturing.

[youkaichao]

However the startup time has increased recently because of CUDA graph capturing.

what's the change?

[DarkLight1337]

I feel that since torch.compile was introduced, the time taken to run graph capturing has increased (particularly on V1)

[123qwe-ux]

did you try more recent versions like 0.6.6?

cc @DarkLight1337 I remember there was a bug that the loader will directly consume the generator and cause longer loading time. but it seems #9160 is merged in 0.6.3.post1 .

I previously tried to use VLLM 0.6.6, but it reported the following error, and I haven't found a solution yet.

 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m collect2: fatal error: cannot find ‘ld’
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m compilation terminated.
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m An error occurred when initializing VLLM_Qwen_Chat_Ray: BackendCompilerFailed.__init__() missing 1 required positional argument: 'inner_exception'
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'):  [36mray::RayWorkerWrapper.execute_method() [39m (pid=724235, ip=10.0.0.222, actor_id=e1a8424323a666a76ac164c101000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f8a6daa9190>)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 129, in __call__
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_gm = compiler_fn(gm, example_inputs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/__init__.py", line 2234, in __call__
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return compile_fx(model_, inputs_, config_patches=self.config)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1521, in compile_fx
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return aot_autograd(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/backends/common.py", line 72, in __call__
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 1071, in aot_module_simplified
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_fn = dispatch_and_compile()
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                   ^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 1056, in dispatch_and_compile
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_fn, _ = create_aot_dispatcher_function(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 522, in create_aot_dispatcher_function
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return _create_aot_dispatcher_function(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 759, in _create_aot_dispatcher_function
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_fn, fw_metadata = compiler_fn(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                                ^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 179, in aot_dispatch_base
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_fw = compiler(fw_module, updated_flat_args)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1350, in fw_compiler_base
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return _fw_compiler_base(model, example_inputs, is_inference)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1421, in _fw_compiler_base
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return inner_compile(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 475, in compile_fx_inner
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/repro/after_aot.py", line 85, in debug_wrapper
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     inner_compiled_fn = compiler_fn(gm, example_inputs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 661, in _compile_fx_inner
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_graph = FxGraphCache.load(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                      ^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 1334, in load
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_graph = compile_fx_fn(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                      ^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 570, in codegen_and_compile
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_graph = fx_codegen_and_compile(gm, example_inputs, **fx_kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 878, in fx_codegen_and_compile
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_fn = graph.compile_to_fn()
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                   ^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1913, in compile_to_fn
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return self.compile_to_module().call
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1839, in compile_to_module
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return self._compile_to_module()
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/graph.py", line 1867, in _compile_to_module
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     mod = PyCodeCache.load_by_key_path(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 2876, in load_by_key_path
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     mod = _reload_python_module(key, path)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/runtime/compile_tasks.py", line 45, in _reload_python_module
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     exec(code, mod.__dict__, mod.__dict__)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/tmp/torchinductor_1473391854/pp/cppcxvfih7enupelcoi432cmgnrq7r6bqga34467yfkfzkvrrjfa.py", line 63, in <module>
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     triton_poi_fused_add_bitwise_and_bitwise_not_bitwise_or_ge_lt_mul_sub_0 = async_compile.triton('triton_', '''
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                                                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/async_compile.py", line 203, in triton
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     kernel.precompile()
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 244, in precompile
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_binary, launcher = self._precompile_config(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                                 ^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 452, in _precompile_config
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     binary._init_handles()
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/triton/compiler/compiler.py", line 370, in _init_handles
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     self.run = driver.active.launcher_cls(self.src, self.metadata)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/triton/backends/nvidia/driver.py", line 361, in __init__
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     mod = compile_module_from_src(src, "__triton_launcher")
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/triton/backends/nvidia/driver.py", line 57, in compile_module_from_src
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     so = _build(name, src_path, tmpdir, library_dirs(), include_dir, libraries)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/triton/runtime/build.py", line 48, in _build
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     ret = subprocess.check_call(cc_cmd)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/usr/local/python3.11.9/lib/python3.11/subprocess.py", line 413, in check_call
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     raise CalledProcessError(retcode, cmd)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m subprocess.CalledProcessError: Command '['/bin/gcc', '/tmp/tmpxygscc7j/main.c', '-O3', '-shared', '-fPIC', '-o', '/tmp/tmpxygscc7j/__triton_launcher.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/home/user1/.local/lib/python3.11/site-packages/triton/backends/nvidia/lib', '-L/usr/lib/x86_64-linux-gnu', '-I/home/user1/.local/lib/python3.11/site-packages/triton/backends/nvidia/include', '-I/tmp/tmpxygscc7j', '-I/usr/local/python3.11.9/include/python3.11']' returned non-zero exit status 1.
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m 
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m The above exception was the direct cause of the following exception:
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m 
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m  [36mray::RayWorkerWrapper.execute_method() [39m (pid=724235, ip=10.0.0.222, actor_id=e1a8424323a666a76ac164c101000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f8a6daa9190>)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return func(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1691, in execute_model
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     hidden_or_intermediate_states = model_executable(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                                     ^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return self._call_impl(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return forward_call(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 477, in forward
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     hidden_states = self.model(input_ids, positions, kv_caches,
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 168, in __call__
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return self.forward(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 332, in forward
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     hidden_states = self.get_input_embeddings(input_ids)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 317, in get_input_embeddings
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return self.embed_tokens(input_ids)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return self._call_impl(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return forward_call(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py", line 404, in forward
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     masked_input, input_mask = get_masked_input_and_mask(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 465, in _fn
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return fn(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 1269, in __call__
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return self._torchdynamo_orig_callable(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 1064, in __call__
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     result = self._inner_convert(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m              ^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 526, in __call__
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return _compile(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 924, in _compile
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     guarded_code = compile_inner(code, one_graph, hooks, transform)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 666, in compile_inner
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return _compile_inner(code, one_graph, hooks, transform)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_utils_internal.py", line 87, in wrapper_function
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return function(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 699, in _compile_inner
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     out_code = transform_code_object(code, transform)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     transformations(instructions, code_options)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 219, in _fn
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return fn(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 634, in transform
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     tracer.run()
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2796, in run
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     super().run()
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 983, in run
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     while self.step():
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m           ^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 895, in step
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     self.dispatch_table[inst.opcode](self, inst)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2987, in RETURN_VALUE
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     self._return(inst)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2972, in _return
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     self.output.compile_subgraph(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1142, in compile_subgraph
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1369, in compile_and_call_fx_graph
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     compiled_fn = self.call_user_compiler(gm)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1416, in call_user_compiler
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return self._call_user_compiler(gm)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1465, in _call_user_compiler
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     raise BackendCompilerFailed(self.compiler_fn, e) from e
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m CalledProcessError: Command '['/bin/gcc', '/tmp/tmpxygscc7j/main.c', '-O3', '-shared', '-fPIC', '-o', '/tmp/tmpxygscc7j/__triton_launcher.cpython-311-x86_64-linux-gnu.so', '-lcuda', '-L/home/user1/.local/lib/python3.11/site-packages/triton/backends/nvidia/lib', '-L/usr/lib/x86_64-linux-gnu', '-I/home/user1/.local/lib/python3.11/site-packages/triton/backends/nvidia/include', '-I/tmp/tmpxygscc7j', '-I/usr/local/python3.11.9/include/python3.11']' returned non-zero exit status 1.
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m 
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m 
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m 
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m You can suppress this exception and fall back to eager by setting:
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     import torch._dynamo
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     torch._dynamo.config.suppress_errors = True
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m 
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m 
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m During handling of the above exception, another exception occurred:
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m 
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m  [36mray::RayWorkerWrapper.execute_method() [39m (pid=724235, ip=10.0.0.222, actor_id=e1a8424323a666a76ac164c101000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f8a6daa9190>)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 468, in execute_method
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     raise e
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 459, in execute_method
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return executor(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return func(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/worker/worker.py", line 202, in determine_num_available_blocks
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     self.model_runner.profile_run()
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return func(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1331, in profile_run
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     self.execute_model(model_input, kv_caches, intermediate_tensors)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     return func(*args, **kwargs)
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m            ^^^^^^^^^^^^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m   File "/home/user1/.local/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 152, in _wrapper
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m     raise type(err)(
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m           ^^^^^^^^^^
 [36m(VLLM_Qwen_Chat_Ray pid=724236) [0m TypeError: BackendCompilerFailed.__init__() missing 1 required positional argument: 'inner_exception'

[youkaichao]

@123qwe-ux your environment is quite complicated, with mixed tensorflow and pytorch. please check it in a clean environment.

and your machine environment also looks terrible, it does not have ld and some tools. maybe you can install the build-essential package

[youkaichao]

I feel that since torch.compile was introduced, the time taken to run graph capturing has increased (particularly on V1)

that's reasonable. v1 has compilation for every run.