[Bug][V1]: Qwen2-VL-7B OOM when loading the model in v0 but not in v1

[fahadh4ilyas]

Your current environment

The output of `python collect_env.py`

PyTorch version: 2.5.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.30.5
Libc version: glibc-2.35

Python version: 3.11.10 | packaged by conda-forge | (main, Oct 16 2024, 01:27:36) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA RTX A5000
Nvidia driver version: 535.154.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             64
On-line CPU(s) list:                0-63
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7551P 32-Core Processor
CPU family:                         23
Model:                              1
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          1
Stepping:                           2
Frequency boost:                    enabled
CPU max MHz:                        2000.0000
CPU min MHz:                        1200.0000
BogoMIPS:                           3999.77
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
Virtualization:                     AMD-V
L1d cache:                          1 MiB (32 instances)
L1i cache:                          2 MiB (32 instances)
L2 cache:                           16 MiB (32 instances)
L3 cache:                           64 MiB (8 instances)
NUMA node(s):                       4
NUMA node0 CPU(s):                  0-7,32-39
NUMA node1 CPU(s):                  8-15,40-47
NUMA node2 CPU(s):                  16-23,48-55
NUMA node3 CPU(s):                  24-31,56-63
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT vulnerable
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.1.105
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] optree==0.13.0
[pip3] pyzmq==26.2.1
[pip3] torch==2.5.1+cu121
[pip3] torchaudio==2.5.1+cu121
[pip3] torchelastic==0.2.2
[pip3] torchvision==0.20.1+cu121
[pip3] transformers==4.49.0
[pip3] triton==3.1.0
[conda] numpy                     2.1.2           py311h71ddf71_0    conda-forge
[conda] nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.1.105                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
[conda] optree                    0.13.0                   pypi_0    pypi
[conda] torch                     2.5.1+cu121              pypi_0    pypi
[conda] torchaudio                2.5.1+cu121              pypi_0    pypi
[conda] torchelastic              0.2.2                    pypi_0    pypi
[conda] torchvision               0.20.1+cu121             pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.1.dev4927+gf35f8e2 (git sha: f35f8e2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-7,32-39       0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=GPU-dd6f7119-c5b6-1f85-1a94-a09927e0d596
NVIDIA_DRIVER_CAPABILITIES=compute,utility
PYTORCH_VERSION=2.5.1
LD_LIBRARY_PATH=/workspace-lib/lib/python3.11/site-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
VLLM_USE_V1=1
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I'm trying to load Qwen2-VL model with this config:

`config.json` file of Qwen2-VL model

{
  "_name_or_path": "Qwen/Qwen2-VL-7B-Instruct-319-PRO",
  "architectures": [
    "Qwen2VLForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "image_token_id": 151655,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2_vl",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "quantization_config": {
    "bits": 8,
    "damp_percent": 0.1,
    "dataset": null,
    "desc_act": false,
    "group_size": 128,
    "modules_in_block_to_quantize": null,
    "quant_method": "gptq",
    "sym": true,
    "true_sequential": true
  },
  "rms_norm_eps": 1e-06,
  "rope_scaling": {
    "mrope_section": [
      16,
      24,
      24
    ],
    "type": "mrope"
  },
  "rope_theta": 1000000.0,
  "sliding_window": 32768,
  "tie_word_embeddings": false,
  "torch_dtype": "float16",
  "transformers_version": "4.45.0.dev0",
  "use_cache": true,
  "use_sliding_window": false,
  "video_token_id": 151656,
  "vision_config": {
    "in_chans": 3,
    "model_type": "qwen2_vl",
    "spatial_patch_size": 14
  },
  "vision_end_token_id": 151653,
  "vision_start_token_id": 151652,
  "vision_token_id": 151654,
  "vocab_size": 152064
}

When I tried to load it using vllm v0.7.2, I could load the model with GPU RTX 4090 with full token length (32768) and not getting OOM. But, after update it to main git, I got OOM saying that maximum token length that I could get is around 12500.

But, if I set the environment variable VLLM_USE_V1=1, I could load the model without OOM. Why is this happening?

Here is the command how I load the model:

vllm serve /models/Qwen2-VL-7B-INT8 --host 0.0.0.0 --served-model-name Qwen2-VL-7B-Int8 --port 8000 --limit-mm-per-prompt image=3,video=1

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

I got OOM saying that maximum token length that I could get is around 12500.

To help debug, can you show the stack trace of this error?

[fahadh4ilyas]

I got OOM saying that maximum token length that I could get is around 12500.

To help debug, can you show the stack trace of this error?

Here is the log

Log when load with VLLM V1

INFO 03-03 12:12:11 [__init__.py:207] Automatically detected platform cuda.
INFO 03-03 12:12:13 [api_server.py:970] vLLM API server version 0.1.dev4927+geda5043
INFO 03-03 12:12:13 [api_server.py:971] args: Namespace(subparser='serve', model_tag='/models/Qwen2-VL-7B-INT8', config='', host='0.0.0.0', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/models/Qwen2-VL-7B-INT8', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 3, 'video': 1}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Qwen2-VL-7B-Int8'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function ServeSubcommand.cmd at 0x7f4c734827a0>)
WARNING 03-03 12:12:13 [arg_utils.py:1434] Setting max_num_batched_tokens to 2048 for OPENAI_API_SERVER usage context.
INFO 03-03 12:12:22 [config.py:576] This model supports multiple tasks: {'generate', 'score', 'classify', 'embed', 'reward'}. Defaulting to 'generate'.
INFO 03-03 12:12:22 [gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-03 12:12:22 [config.py:1661] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 03-03 12:12:27 [__init__.py:207] Automatically detected platform cuda.
INFO 03-03 12:12:30 [core.py:50] Initializing a V1 LLM engine (v0.1.dev4927+geda5043) with config: model='/models/Qwen2-VL-7B-INT8', speculative_config=None, tokenizer='/models/Qwen2-VL-7B-INT8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2-VL-7B-Int8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 03-03 12:12:30 [utils.py:2298] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fd5cdee7010>
INFO 03-03 12:12:31 [parallel_state.py:948] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 03-03 12:12:31 [cuda.py:198] Using Flash Attention backend on V1 engine.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
INFO 03-03 12:12:31 [gpu_model_runner.py:1052] Starting to load model /models/Qwen2-VL-7B-INT8...
WARNING 03-03 12:12:31 [vision.py:94] Current `vllm-flash-attn` has a bug inside vision module, so we use xformers backend instead. You can run `pip install flash-attn` to use flash-attention backend.
INFO 03-03 12:12:32 [config.py:3164] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 264, 272, 280, 288, 296, 304, 312, 320, 328, 336, 344, 352, 360, 368, 376, 384, 392, 400, 408, 416, 424, 432, 440, 448, 456, 464, 472, 480, 488, 496, 504, 512] is overridden by config [512, 384, 256, 128, 4, 2, 1, 392, 264, 136, 8, 400, 272, 144, 16, 408, 280, 152, 24, 416, 288, 160, 32, 424, 296, 168, 40, 432, 304, 176, 48, 440, 312, 184, 56, 448, 320, 192, 64, 456, 328, 200, 72, 464, 336, 208, 80, 472, 344, 216, 88, 120, 480, 352, 248, 224, 96, 488, 504, 360, 232, 104, 496, 368, 240, 112, 376]
INFO 03-03 12:12:32 [gptq_marlin.py:235] Using MarlinLinearKernel for GPTQMarlinLinearMethod
WARNING 03-03 12:12:32 [topk_topp_sampler.py:46] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:50<01:40, 50.02s/it]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [01:11<00:33, 33.21s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [01:47<00:00, 34.46s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [01:47<00:00, 35.80s/it]

INFO 03-03 12:14:19 [loader.py:422] Loading weights took 107.49 seconds
INFO 03-03 12:14:20 [gpu_model_runner.py:1064] Model loading took 9.5234 GB and 107.983182 seconds
INFO 03-03 12:14:20 [gpu_model_runner.py:1216] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
Token indices sequence length is longer than the specified maximum sequence length for this model (65536 > 32768). Running this sequence through the model will result in indexing errors
INFO 03-03 12:15:01 [backends.py:408] Using cache directory: /root/.cache/vllm/torch_compile_cache/c32a66418a/rank_0 for vLLM's torch.compile
INFO 03-03 12:15:01 [backends.py:418] Dynamo bytecode transform time: 14.29 s
INFO 03-03 12:15:08 [backends.py:132] Cache the graph of shape None for later use
INFO 03-03 12:15:51 [backends.py:144] Compiling a graph for general shape takes 48.45 s
INFO 03-03 12:15:57 [monitor.py:33] torch.compile takes 62.74 s in total
INFO 03-03 12:15:59 [kv_cache_utils.py:537] GPU KV cache size: 71,936 tokens
INFO 03-03 12:15:59 [kv_cache_utils.py:540] Maximum concurrency for 32,768 tokens per request: 2.20x
INFO 03-03 12:16:28 [gpu_model_runner.py:1353] Graph capturing finished in 30 secs, took 2.01 GiB
INFO 03-03 12:16:28 [core.py:116] init engine (profile, create kv cache, warmup model) took 128.50 seconds
INFO 03-03 12:16:28 [api_server.py:1016] Starting vLLM API server on http://0.0.0.0:8000
INFO 03-03 12:16:28 [launcher.py:26] Available routes are:
INFO 03-03 12:16:28 [launcher.py:34] Route: /openapi.json, Methods: GET, HEAD
INFO 03-03 12:16:28 [launcher.py:34] Route: /docs, Methods: GET, HEAD
INFO 03-03 12:16:28 [launcher.py:34] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 03-03 12:16:28 [launcher.py:34] Route: /redoc, Methods: GET, HEAD
INFO 03-03 12:16:28 [launcher.py:34] Route: /health, Methods: GET
INFO 03-03 12:16:28 [launcher.py:34] Route: /ping, Methods: GET, POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /generate, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /tokenize, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /detokenize, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /v1/models, Methods: GET
INFO 03-03 12:16:28 [launcher.py:34] Route: /version, Methods: GET
INFO 03-03 12:16:28 [launcher.py:34] Route: /v1/chat/completions, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /v1/completions, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /v1/embeddings, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /pooling, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /score, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /v1/score, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /v1/audio/transcriptions, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /rerank, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /v1/rerank, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /v2/rerank, Methods: POST
INFO 03-03 12:16:28 [launcher.py:34] Route: /invocations, Methods: POST
INFO:     Started server process [7]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Log when load with VLLM V0

INFO 03-04 06:03:47 [__init__.py:207] Automatically detected platform cuda.
INFO 03-04 06:03:49 [api_server.py:970] vLLM API server version 0.1.dev4927+geda5043
INFO 03-04 06:03:49 [api_server.py:971] args: Namespace(subparser='serve', model_tag='/models/Qwen2-VL-7B-INT8', config='', host='0.0.0.0', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/models/Qwen2-VL-7B-INT8', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 3, 'video': 1}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Qwen2-VL-7B-Int8'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function ServeSubcommand.cmd at 0x7fa323828900>)
INFO 03-04 06:03:49 [api_server.py:211] Started engine process with PID 78
INFO 03-04 06:03:54 [__init__.py:207] Automatically detected platform cuda.
INFO 03-04 06:03:57 [config.py:576] This model supports multiple tasks: {'generate', 'score', 'embed', 'reward', 'classify'}. Defaulting to 'generate'.
INFO 03-04 06:03:58 [gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-04 06:04:03 [config.py:576] This model supports multiple tasks: {'generate', 'classify', 'score', 'reward', 'embed'}. Defaulting to 'generate'.
INFO 03-04 06:04:04 [gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-04 06:04:04 [llm_engine.py:235] Initializing a V0 LLM engine (v0.1.dev4927+geda5043) with config: model='/models/Qwen2-VL-7B-INT8', speculative_config=None, tokenizer='/models/Qwen2-VL-7B-INT8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2-VL-7B-Int8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
INFO 03-04 06:04:05 [cuda.py:268] Using Flash Attention backend.
INFO 03-04 06:04:05 [parallel_state.py:948] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 03-04 06:04:05 [model_runner.py:1110] Starting to load model /models/Qwen2-VL-7B-INT8...
WARNING 03-04 06:04:05 [vision.py:94] Current `vllm-flash-attn` has a bug inside vision module, so we use xformers backend instead. You can run `pip install flash-attn` to use flash-attention backend.
INFO 03-04 06:04:05 [config.py:3164] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
INFO 03-04 06:04:05 [gptq_marlin.py:235] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:12<00:24, 12.17s/it]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:12<00:05,  5.44s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:13<00:00,  3.42s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:13<00:00,  4.63s/it]

INFO 03-04 06:04:20 [loader.py:422] Loading weights took 14.15 seconds
INFO 03-04 06:04:20 [model_runner.py:1117] Model loading took 9.5229 GB and 14.536663 seconds
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
WARNING 03-04 06:04:21 [model_runner.py:1296] Computed max_num_seqs (min(256, 32768 // 65536)) to be less than 1. Setting it to the minimum value of 1.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
Token indices sequence length is longer than the specified maximum sequence length for this model (65536 > 32768). Running this sequence through the model will result in indexing errors
WARNING 03-04 06:04:30 [profiling.py:196] The context length (32768) of the model is too short to hold the multi-modal embeddings in the worst case (65536 tokens in total, out of which {'image': 49152, 'video': 16384} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.
INFO 03-04 06:05:00 [worker.py:267] Memory profiling takes 39.36 seconds
INFO 03-04 06:05:00 [worker.py:267] the current vLLM instance can use total_gpu_memory (23.68GiB) x gpu_memory_utilization (0.90) = 21.31GiB
INFO 03-04 06:05:00 [worker.py:267] model weights take 9.52GiB; non_torch_memory takes 0.05GiB; PyTorch activation peak memory takes 11.02GiB; the rest of the memory reserved for KV Cache is 0.72GiB.
INFO 03-04 06:05:00 [executor_base.py:111] # cuda blocks: 848, # CPU blocks: 4681
INFO 03-04 06:05:00 [executor_base.py:116] Maximum concurrency for 32768 tokens per request: 0.41x
ERROR 03-04 06:05:00 [engine.py:409] The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (13568). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
ERROR 03-04 06:05:00 [engine.py:409] Traceback (most recent call last):
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 400, in run_mp_engine
ERROR 03-04 06:05:00 [engine.py:409]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 03-04 06:05:00 [engine.py:409]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 125, in from_engine_args
ERROR 03-04 06:05:00 [engine.py:409]     return cls(ipc_path=ipc_path,
ERROR 03-04 06:05:00 [engine.py:409]            ^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 77, in __init__
ERROR 03-04 06:05:00 [engine.py:409]     self.engine = LLMEngine(*args, **kwargs)
ERROR 03-04 06:05:00 [engine.py:409]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 277, in __init__
ERROR 03-04 06:05:00 [engine.py:409]     self._initialize_kv_caches()
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 439, in _initialize_kv_caches
ERROR 03-04 06:05:00 [engine.py:409]     self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 122, in initialize_cache
ERROR 03-04 06:05:00 [engine.py:409]     self.collective_rpc("initialize_cache",
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 03-04 06:05:00 [engine.py:409]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 03-04 06:05:00 [engine.py:409]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/utils.py", line 2232, in run_method
ERROR 03-04 06:05:00 [engine.py:409]     return func(*args, **kwargs)
ERROR 03-04 06:05:00 [engine.py:409]            ^^^^^^^^^^^^^^^^^^^^^
Process SpawnProcess-1:
Traceback (most recent call last):
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/worker/worker.py", line 291, in initialize_cache
ERROR 03-04 06:05:00 [engine.py:409]     raise_if_cache_size_invalid(
ERROR 03-04 06:05:00 [engine.py:409]   File "/workspace-lib/lib/python3.11/site-packages/vllm/worker/worker.py", line 545, in raise_if_cache_size_invalid
ERROR 03-04 06:05:00 [engine.py:409]     raise ValueError(
ERROR 03-04 06:05:00 [engine.py:409] ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (13568). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 411, in run_mp_engine
    raise e
  File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 400, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 125, in from_engine_args
    return cls(ipc_path=ipc_path,
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 77, in __init__
    self.engine = LLMEngine(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 277, in __init__
    self._initialize_kv_caches()
  File "/workspace-lib/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 439, in _initialize_kv_caches
    self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
  File "/workspace-lib/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 122, in initialize_cache
    self.collective_rpc("initialize_cache",
  File "/workspace-lib/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/utils.py", line 2232, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/worker/worker.py", line 291, in initialize_cache
    raise_if_cache_size_invalid(
  File "/workspace-lib/lib/python3.11/site-packages/vllm/worker/worker.py", line 545, in raise_if_cache_size_invalid
    raise ValueError(
ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (13568). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
[rank0]:[W304 06:05:00.325833820 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
Traceback (most recent call last):
  File "/workspace-lib/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
    args.dispatch_function(args)
  File "/workspace-lib/lib/python3.11/site-packages/vllm/entrypoints/cli/serve.py", line 33, in cmd
    uvloop.run(run_server(args))
  File "/workspace-lib/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/workspace-lib/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1005, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/opt/conda/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 141, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/opt/conda/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace-lib/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 235, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

[DarkLight1337]

I think it's because V1 doesn't hard fail when sequence length is too long to fit. Instead, it emits this warning:

INFO 03-03 12:14:20 [gpu_model_runner.py:1216] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
Token indices sequence length is longer than the specified maximum sequence length for this model (65536 > 32768). Running this sequence through the model will result in indexing errors

I think this might be a footgun as it may lead to unexpected errors like #14181. Any thoughts @ywang96 ?

[fahadh4ilyas]

I think it's because V1 doesn't hard fail when sequence length is too long to fit. Instead, it emits this warning:

INFO 03-03 12:14:20 [gpu_model_runner.py:1216] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
Token indices sequence length is longer than the specified maximum sequence length for this model (65536 > 32768). Running this sequence through the model will result in indexing errors

I think this might be a footgun as it may lead to unexpected errors like #14181. Any thoughts @ywang96 ?

But, If I load it using vllm v0.7.2, the model somehow loaded without OOM without VLLM_USE_V1=1

Here is the log

Log when load using vllm v0.7.2 wih VLLM V0

INFO 03-04 06:25:28 __init__.py:197] Automatically detected platform cuda.
INFO 03-04 06:25:28 api_server.py:949] vLLM API server version 0.1.dev4676+ge1b7aa6
INFO 03-04 06:25:28 api_server.py:950] args: Namespace(subparser='serve', model_tag='/models/Qwen2-VL-7B-INT8', config='', host='0.0.0.0', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='/models/Qwen2-VL-7B-INT8', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 3, 'video': 1}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Qwen2-VL-7B-Int8'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function ServeSubcommand.cmd at 0x7f1bf46a7ec0>)
INFO 03-04 06:25:28 api_server.py:211] Started engine process with PID 78
ERROR 03-04 06:25:28 config.py:102] Error retrieving file list: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/models/Qwen2-VL-7B-INT8'. Use `repo_type` argument if needed., retrying 1 of 2
ERROR 03-04 06:25:30 config.py:100] Error retrieving file list: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/models/Qwen2-VL-7B-INT8'. Use `repo_type` argument if needed.
INFO 03-04 06:25:34 __init__.py:197] Automatically detected platform cuda.
ERROR 03-04 06:25:34 config.py:102] Error retrieving file list: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/models/Qwen2-VL-7B-INT8'. Use `repo_type` argument if needed., retrying 1 of 2
ERROR 03-04 06:25:36 config.py:100] Error retrieving file list: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/models/Qwen2-VL-7B-INT8'. Use `repo_type` argument if needed.
INFO 03-04 06:25:38 config.py:548] This model supports multiple tasks: {'classify', 'reward', 'generate', 'score', 'embed'}. Defaulting to 'generate'.
INFO 03-04 06:25:39 gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-04 06:25:44 config.py:548] This model supports multiple tasks: {'score', 'generate', 'classify', 'reward', 'embed'}. Defaulting to 'generate'.
INFO 03-04 06:25:45 gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-04 06:25:45 llm_engine.py:234] Initializing a V0 LLM engine (v0.1.dev4676+ge1b7aa6) with config: model='/models/Qwen2-VL-7B-INT8', speculative_config=None, tokenizer='/models/Qwen2-VL-7B-INT8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2-VL-7B-Int8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
INFO 03-04 06:25:46 cuda.py:229] Using Flash Attention backend.
INFO 03-04 06:25:46 model_runner.py:1109] Starting to load model /models/Qwen2-VL-7B-INT8...
WARNING 03-04 06:25:46 vision.py:94] Current `vllm-flash-attn` has a bug inside vision module, so we use xformers backend instead. You can run `pip install flash-attn` to use flash-attention backend.
INFO 03-04 06:25:46 config.py:3026] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
INFO 03-04 06:25:46 gptq_marlin.py:235] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:00<00:01,  1.05it/s]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:01<00:00,  1.27it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00,  1.08it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00,  1.10it/s]

INFO 03-04 06:25:50 model_runner.py:1114] Loading model weights took 9.5229 GB
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
WARNING 03-04 06:25:52 model_runner.py:1287] Computed max_num_seqs (min(256, 32768 // 65536)) to be less than 1. Setting it to the minimum value of 1.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
Token indices sequence length is longer than the specified maximum sequence length for this model (65536 > 32768). Running this sequence through the model will result in indexing errors
WARNING 03-04 06:26:00 profiling.py:192] The context length (32768) of the model is too short to hold the multi-modal embeddings in the worst case (65536 tokens in total, out of which {'image': 49152, 'video': 16384} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.
INFO 03-04 06:26:12 worker.py:267] Memory profiling takes 21.79 seconds
INFO 03-04 06:26:12 worker.py:267] the current vLLM instance can use total_gpu_memory (23.68GiB) x gpu_memory_utilization (0.90) = 21.31GiB
INFO 03-04 06:26:12 worker.py:267] model weights take 9.52GiB; non_torch_memory takes 0.05GiB; PyTorch activation peak memory takes 5.51GiB; the rest of the memory reserved for KV Cache is 6.23GiB.
INFO 03-04 06:26:12 executor_base.py:111] # cuda blocks: 7285, # CPU blocks: 4681
INFO 03-04 06:26:12 executor_base.py:116] Maximum concurrency for 32768 tokens per request: 3.56x
INFO 03-04 06:26:17 model_runner.py:1433] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|██████████| 35/35 [00:19<00:00,  1.79it/s]
INFO 03-04 06:26:37 model_runner.py:1561] Graph capturing finished in 20 secs, took 2.06 GiB
INFO 03-04 06:26:37 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 46.95 seconds
INFO 03-04 06:26:37 api_server.py:858] Using supplied chat template:
INFO 03-04 06:26:37 api_server.py:858] None
INFO 03-04 06:26:37 api_server.py:995] Starting vLLM API server on http://0.0.0.0:8000
INFO 03-04 06:26:37 launcher.py:23] Available routes are:
INFO 03-04 06:26:37 launcher.py:31] Route: /openapi.json, Methods: HEAD, GET
INFO 03-04 06:26:37 launcher.py:31] Route: /docs, Methods: HEAD, GET
INFO 03-04 06:26:37 launcher.py:31] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 03-04 06:26:37 launcher.py:31] Route: /redoc, Methods: HEAD, GET
INFO 03-04 06:26:37 launcher.py:31] Route: /health, Methods: GET
INFO 03-04 06:26:37 launcher.py:31] Route: /ping, Methods: GET, POST
INFO 03-04 06:26:37 launcher.py:31] Route: /generate, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /tokenize, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /detokenize, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /v1/models, Methods: GET
INFO 03-04 06:26:37 launcher.py:31] Route: /version, Methods: GET
INFO 03-04 06:26:37 launcher.py:31] Route: /v1/chat/completions, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /v1/completions, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /v1/embeddings, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /pooling, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /score, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /v1/score, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /v1/audio/transcriptions, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /rerank, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /v1/rerank, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /v2/rerank, Methods: POST
INFO 03-04 06:26:37 launcher.py:31] Route: /invocations, Methods: POST
INFO:     Started server process [7]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

[ywang96]

I think this is expected? In V0 we assume that multimodal requests need to be fully prefilled (therefore max_num_batched_tokens == max_model_len), thus we profile with limit-mm-per-prompt number of biggest possible images, but in V1 since we natively support chunked prefill for multimodal input, so we only profile with max_num_batched_tokens=2048 tokens, and the number of biggest possible images that can fit in this batch (in this case, one image)

This means now we there's way less memory needed in V1. You can tell this from the number of GPU blocks after profiling

V1

GPU KV cache size: 71,936 tokens

V0

cuda blocks: 848 (13568 tokens)

[ywang96]

But, If I load it using vllm v0.7.2, the model somehow loaded without OOM without VLLM_USE_V1=1

@fahadh4ilyas Hmmm.. now that is more concerning, and I do see a big difference in terms of cuda blocks: 7285 available

[DarkLight1337]

Not sure why V0 is broken now. Did the memory profiling in V0 change recently? @youkaichao @tlrmchlsmth

[youkaichao]

I also got several reports about OOM.

can you follow https://blog.vllm.ai/2025/01/10/dev-experience.html to bisect which commit caused the regression?

FYI, installing vllm from any commit is easy:

# use full commit hash from the main branch
export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd
pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl

[fahadh4ilyas]

I also got several reports about OOM.

can you follow https://blog.vllm.ai/2025/01/10/dev-experience.html to bisect which commit caused the regression?

FYI, installing vllm from any commit is easy:

use full commit hash from the main branch

export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd
pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl

This commit is the culprit ba5106e

Here is the log

log vllm using commit ba5106e

INFO 03-05 16:30:24 __init__.py:207] Automatically detected platform cuda.
INFO 03-05 16:30:24 api_server.py:911] vLLM API server version 0.7.4.dev50+gba5106e5
INFO 03-05 16:30:24 api_server.py:912] args: Namespace(subparser='serve', model_tag='models/Qwen2-VL-7B-INT8', config='', host='0.0.0.0', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='models/Qwen2-VL-7B-INT8', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 3, 'video': 1}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Qwen2-VL-7B-Int8'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function ServeSubcommand.cmd at 0x7d9233ea8f40>)
INFO 03-05 16:30:24 api_server.py:208] Started engine process with PID 157177
INFO 03-05 16:30:26 __init__.py:207] Automatically detected platform cuda.
INFO 03-05 16:30:27 config.py:560] This model supports multiple tasks: {'classify', 'score', 'generate', 'reward', 'embed'}. Defaulting to 'generate'.
INFO 03-05 16:30:28 gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-05 16:30:30 config.py:560] This model supports multiple tasks: {'generate', 'embed', 'reward', 'score', 'classify'}. Defaulting to 'generate'.
INFO 03-05 16:30:30 gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-05 16:30:30 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.4.dev50+gba5106e5) with config: model='models/Qwen2-VL-7B-INT8', speculative_config=None, tokenizer='models/Qwen2-VL-7B-INT8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2-VL-7B-Int8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
INFO 03-05 16:30:30 cuda.py:229] Using Flash Attention backend.
INFO 03-05 16:30:30 parallel_state.py:948] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 03-05 16:30:30 model_runner.py:1110] Starting to load model models/Qwen2-VL-7B-INT8...
WARNING 03-05 16:30:31 vision.py:94] Current `vllm-flash-attn` has a bug inside vision module, so we use xformers backend instead. You can run `pip install flash-attn` to use flash-attention backend.
INFO 03-05 16:30:31 config.py:3127] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
INFO 03-05 16:30:31 gptq_marlin.py:235] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:02<00:05,  2.76s/it]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:07<00:04,  4.04s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:12<00:00,  4.44s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:12<00:00,  4.21s/it]

INFO 03-05 16:30:43 model_runner.py:1117] Loading model weights took 9.5229 GB and 12.797676 seconds
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
WARNING 03-05 16:30:44 model_runner.py:1291] Computed max_num_seqs (min(256, 32768 // 65536)) to be less than 1. Setting it to the minimum value of 1.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
Token indices sequence length is longer than the specified maximum sequence length for this model (65536 > 32768). Running this sequence through the model will result in indexing errors
WARNING 03-05 16:30:48 profiling.py:196] The context length (32768) of the model is too short to hold the multi-modal embeddings in the worst case (65536 tokens in total, out of which {'image': 49152, 'video': 16384} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.
INFO 03-05 16:31:00 worker.py:267] Memory profiling takes 16.60 seconds
INFO 03-05 16:31:00 worker.py:267] the current vLLM instance can use total_gpu_memory (23.64GiB) x gpu_memory_utilization (0.90) = 21.28GiB
INFO 03-05 16:31:00 worker.py:267] model weights take 9.52GiB; non_torch_memory takes 0.08GiB; PyTorch activation peak memory takes 11.02GiB; the rest of the memory reserved for KV Cache is 0.66GiB.
INFO 03-05 16:31:00 executor_base.py:111] # cuda blocks: 777, # CPU blocks: 4681
INFO 03-05 16:31:00 executor_base.py:116] Maximum concurrency for 32768 tokens per request: 0.38x
ERROR 03-05 16:31:00 engine.py:400] The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (12432). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
ERROR 03-05 16:31:00 engine.py:400] Traceback (most recent call last):
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
ERROR 03-05 16:31:00 engine.py:400]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 03-05 16:31:00 engine.py:400]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 124, in from_engine_args
ERROR 03-05 16:31:00 engine.py:400]     return cls(ipc_path=ipc_path,
ERROR 03-05 16:31:00 engine.py:400]            ^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 76, in __init__
ERROR 03-05 16:31:00 engine.py:400]     self.engine = LLMEngine(*args, **kwargs)
ERROR 03-05 16:31:00 engine.py:400]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 276, in __init__
ERROR 03-05 16:31:00 engine.py:400]     self._initialize_kv_caches()
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 434, in _initialize_kv_caches
ERROR 03-05 16:31:00 engine.py:400]     self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 122, in initialize_cache
ERROR 03-05 16:31:00 engine.py:400]     self.collective_rpc("initialize_cache",
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 03-05 16:31:00 engine.py:400]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 03-05 16:31:00 engine.py:400]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/utils.py", line 2232, in run_method
ERROR 03-05 16:31:00 engine.py:400]     return func(*args, **kwargs)
ERROR 03-05 16:31:00 engine.py:400]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/worker/worker.py", line 291, in initialize_cache
ERROR 03-05 16:31:00 engine.py:400]     raise_if_cache_size_invalid(num_gpu_blocks,
ERROR 03-05 16:31:00 engine.py:400]   File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/worker/worker.py", line 544, in raise_if_cache_size_invalid
ERROR 03-05 16:31:00 engine.py:400]     raise ValueError(
ERROR 03-05 16:31:00 engine.py:400] ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (12432). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 402, in run_mp_engine
    raise e
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 124, in from_engine_args
    return cls(ipc_path=ipc_path,
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 76, in __init__
    self.engine = LLMEngine(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 276, in __init__
    self._initialize_kv_caches()
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 434, in _initialize_kv_caches
    self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 122, in initialize_cache
    self.collective_rpc("initialize_cache",
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/utils.py", line 2232, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/worker/worker.py", line 291, in initialize_cache
    raise_if_cache_size_invalid(num_gpu_blocks,
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/worker/worker.py", line 544, in raise_if_cache_size_invalid
    raise ValueError(
ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (12432). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
[rank0]:[W305 16:31:00.120215099 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
Traceback (most recent call last):
  File "/home/fahadh/anaconda3/envs/test-vllm/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
    args.dispatch_function(args)
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/entrypoints/cli/serve.py", line 34, in cmd
    uvloop.run(run_server(args))
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 946, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 138, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/fahadh/anaconda3/envs/test-vllm/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 232, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

As a proof, here is the log from the commit before it d5ca211

log vllm using commit d5ca211

INFO 03-05 16:37:38 __init__.py:207] Automatically detected platform cuda.
INFO 03-05 16:37:38 api_server.py:911] vLLM API server version 0.7.4.dev49+gd5ca2110
INFO 03-05 16:37:38 api_server.py:912] args: Namespace(subparser='serve', model_tag='models/Qwen2-VL-7B-INT8', config='', host='0.0.0.0', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='models/Qwen2-VL-7B-INT8', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 3, 'video': 1}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Qwen2-VL-7B-Int8'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function ServeSubcommand.cmd at 0x7687806acf40>)
INFO 03-05 16:37:38 api_server.py:208] Started engine process with PID 158449
INFO 03-05 16:37:40 __init__.py:207] Automatically detected platform cuda.
INFO 03-05 16:37:41 config.py:560] This model supports multiple tasks: {'reward', 'embed', 'generate', 'score', 'classify'}. Defaulting to 'generate'.
INFO 03-05 16:37:42 gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-05 16:37:43 config.py:560] This model supports multiple tasks: {'embed', 'reward', 'score', 'classify', 'generate'}. Defaulting to 'generate'.
INFO 03-05 16:37:44 gptq_marlin.py:143] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 03-05 16:37:44 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.4.dev49+gd5ca2110) with config: model='models/Qwen2-VL-7B-INT8', speculative_config=None, tokenizer='models/Qwen2-VL-7B-INT8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2-VL-7B-Int8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
INFO 03-05 16:37:44 cuda.py:229] Using Flash Attention backend.
INFO 03-05 16:37:44 parallel_state.py:948] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 03-05 16:37:44 model_runner.py:1110] Starting to load model models/Qwen2-VL-7B-INT8...
WARNING 03-05 16:37:44 vision.py:94] Current `vllm-flash-attn` has a bug inside vision module, so we use xformers backend instead. You can run `pip install flash-attn` to use flash-attention backend.
INFO 03-05 16:37:44 config.py:3127] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
INFO 03-05 16:37:44 gptq_marlin.py:235] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:02<00:05,  2.76s/it]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:07<00:04,  4.04s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:12<00:00,  4.44s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:12<00:00,  4.21s/it]

INFO 03-05 16:37:57 model_runner.py:1117] Loading model weights took 9.5229 GB and 12.806197 seconds
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
WARNING 03-05 16:37:58 model_runner.py:1291] Computed max_num_seqs (min(256, 32768 // 65536)) to be less than 1. Setting it to the minimum value of 1.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
Token indices sequence length is longer than the specified maximum sequence length for this model (65536 > 32768). Running this sequence through the model will result in indexing errors
WARNING 03-05 16:38:02 profiling.py:192] The context length (32768) of the model is too short to hold the multi-modal embeddings in the worst case (65536 tokens in total, out of which {'image': 49152, 'video': 16384} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.
INFO 03-05 16:38:07 worker.py:267] Memory profiling takes 9.61 seconds
INFO 03-05 16:38:07 worker.py:267] the current vLLM instance can use total_gpu_memory (23.64GiB) x gpu_memory_utilization (0.90) = 21.28GiB
INFO 03-05 16:38:07 worker.py:267] model weights take 9.52GiB; non_torch_memory takes 0.08GiB; PyTorch activation peak memory takes 5.51GiB; the rest of the memory reserved for KV Cache is 6.17GiB.
INFO 03-05 16:38:07 executor_base.py:111] # cuda blocks: 7215, # CPU blocks: 4681
INFO 03-05 16:38:07 executor_base.py:116] Maximum concurrency for 32768 tokens per request: 3.52x
INFO 03-05 16:38:09 model_runner.py:1437] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:09<00:00,  3.82it/s]
INFO 03-05 16:38:18 model_runner.py:1565] Graph capturing finished in 9 secs, took 2.07 GiB
INFO 03-05 16:38:18 llm_engine.py:436] init engine (profile, create kv cache, warmup model) took 20.61 seconds
INFO 03-05 16:38:18 api_server.py:957] Starting vLLM API server on http://0.0.0.0:8000
INFO 03-05 16:38:18 launcher.py:26] Available routes are:
INFO 03-05 16:38:18 launcher.py:34] Route: /openapi.json, Methods: GET, HEAD
INFO 03-05 16:38:18 launcher.py:34] Route: /docs, Methods: GET, HEAD
INFO 03-05 16:38:18 launcher.py:34] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 03-05 16:38:18 launcher.py:34] Route: /redoc, Methods: GET, HEAD
INFO 03-05 16:38:18 launcher.py:34] Route: /health, Methods: GET
INFO 03-05 16:38:18 launcher.py:34] Route: /ping, Methods: GET, POST
INFO 03-05 16:38:18 launcher.py:34] Route: /tokenize, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /detokenize, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /v1/models, Methods: GET
INFO 03-05 16:38:18 launcher.py:34] Route: /version, Methods: GET
INFO 03-05 16:38:18 launcher.py:34] Route: /v1/chat/completions, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /v1/completions, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /v1/embeddings, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /pooling, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /score, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /v1/score, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /v1/audio/transcriptions, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /rerank, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /v1/rerank, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /v2/rerank, Methods: POST
INFO 03-05 16:38:18 launcher.py:34] Route: /invocations, Methods: POST
INFO:     Started server process [158422]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

[DarkLight1337]

cc @Isotr0py