[Usage]: Qwen2-VL keyword argument `max_pixels` is not a valid argument for this processor and will be ignored.

[gaoshangle]

Your current environment

branch:qwen25vl

How would you like to use vllm

VLLM_ARGS="--limit-mm-per-prompt image=2 \
--tensor-parallel-size 1 \
--max-model-len 16384
--served-model-name Qwen2.5-VL-7B-Instruct/ \
--mm_processor_kwargs {\"max_pixels\":1000000} \
--gpu_memory_utilization 0.9 \
--model Qwen/Qwen2.5-VL-7B-Instruct/"
CUDA_VISIBLE_DEVICES=0 python3 -m vllm.entrypoints.openai.api_server ${VLLM_ARGS} --port 8000

bash run in the preceding way, the configured max_pixels is invalid in the following log

INFO 02-12 03:01:35 cuda.py:230] Using Flash Attention backend.
[W212 03:01:36.265895118 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
INFO 02-12 03:01:36 model_runner.py:1110] Starting to load model Qwen/Qwen2.5-VL-7B-autoglm-android-wechat-test-250211/...
INFO 02-12 03:01:36 config.py:2930] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:01<00:03,  1.15s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:02<00:01,  1.00it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.08it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.42it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.23it/s]
INFO 02-12 03:01:39 model_runner.py:1115] Loading model weights took 16.7361 GB
WARNING 02-12 03:01:41 model_runner.py:1288] Computed max_num_seqs (min(256, 10384 // 11025)) to be less than 1. Setting it to the minimum value of 1.


Keyword argument `max_pixels` is not a valid argument for this processor and will be ignored.


It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
WARNING 02-12 03:01:43 profiling.py:184] The context length (10384) of the model is too short to hold the multi-modal embeddings in the worst case (11025 tokens in total, out of which {'image': 2450, 'video': 8575} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.
INFO 02-12 03:01:45 worker.py:266] Memory profiling takes 5.11 seconds
INFO 02-12 03:01:45 worker.py:266] the current vLLM instance can use total_gpu_memory (23.65GiB) x gpu_memory_utilization (0.90) = 21.28GiB
INFO 02-12 03:01:45 worker.py:266] model weights take 16.74GiB; non_torch_memory takes 0.08GiB; PyTorch activation peak memory takes 1.38GiB; the rest of the memory reserved for KV Cache is 3.09GiB.
INFO 02-12 03:01:45 executor_base.py:108] # CUDA blocks: 3613, # CPU blocks: 4681
INFO 02-12 03:01:45 executor_base.py:113] Maximum concurrency for 10384 tokens per request: 5.57x
INFO 02-12 03:01:47 model_runner.py:1430] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:14<00:00,  2.44it/s]
INFO 02-12 03:02:02 model_runner.py:1558] Graph capturing finished in 14 secs, took 1.89 GiB
INFO 02-12 03:02:02 llm_engine.py:429] init engine (profile, create kv cache, warmup model) took 22.17 seconds
INFO 02-12 03:02:02 api_server.py:754] Using supplied chat template:
INFO 02-12 03:02:02 api_server.py:754] None
INFO 02-12 03:02:02 launcher.py:19] Available routes are:
INFO 02-12 03:02:02 launcher.py:27] Route: /openapi.json, Methods: HEAD, GET
INFO 02-12 03:02:02 launcher.py:27] Route: /docs, Methods: HEAD, GET
INFO 02-12 03:02:02 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 02-12 03:02:02 launcher.py:27] Route: /redoc, Methods: HEAD, GET
INFO 02-12 03:02:02 launcher.py:27] Route: /health, Methods: GET
INFO 02-12 03:02:02 launcher.py:27] Route: /ping, Methods: POST, GET
INFO 02-12 03:02:02 launcher.py:27] Route: /tokenize, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /detokenize, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /v1/models, Methods: GET
INFO 02-12 03:02:02 launcher.py:27] Route: /version, Methods: GET
INFO 02-12 03:02:02 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /pooling, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /score, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /v1/score, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /rerank, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /v1/rerank, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /v2/rerank, Methods: POST
INFO 02-12 03:02:02 launcher.py:27] Route: /invocations, Methods: POST
INFO:     Started server process [6960]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

[DarkLight1337]

Which branch are you talking about?

[DarkLight1337]

It looks like the HF image processor doesn't support passing min_pixels and max_pixels at call time, only at initialization time.

[tzjtatata]

I passed min_pixels and max_pixels works. However, the warning is still reported at every steps.
Code:

self.llm = LLM(
    model=model.name_or_path,
    device=vllm_device,
    gpu_memory_utilization=self.args.vllm_gpu_memory_utilization,
    dtype=torch.bfloat16,
    mm_processor_kwargs= { "max_pixels": max_pixels, "min_pixels": min_pixels} ,  
    # Automatic Prefix Caching caches the KV cache of existing queries, so that a new query can
    # directly reuse the KV cache if it shares the same prefix with one of the existing queries.
    # This is particularly useful here because we generate completions from the same prompts.
    enable_prefix_caching=True,
    enforce_eager=True,
    max_model_len=self.max_prompt_length,
)

At the init time, warning like:

Keyword argument `min_pixels` is not a valid argument for this processor and will be ignored. 
Keyword argument `max_pixels` is not a valid argument for this processor and will be ignored.

And when inference, the warning continues like:
WARNING 02-13 07:31:53 utils.py:1473] The following intended overrides are not keyword-only args and and will be dropped: {'min_pixels', 'max_pixels'}

But I am sure that the max_pixels actually works.

[gaoshangle]

Which branch are you talking about?

branch:qwen25vl

[DarkLight1337]

I think I figured out the issue - the kwargs are being passed into HF processor.__call__ in vLLM's BaseMultiModalProcessor._call_hf_processor which triggers the warning. However, it is difficult for us to determine whether the kwargs are supposed to be passed in __init__ or in __call__ without actually calling the function because HF processor.__call__ forwards all kwargs (so direct introspection doesn't work).

@ArthurZucker @hmellor any suggestions for detecting which keyword arguments are valid?

[hmellor]

Thank you for investigating this @DarkLight1337.

it is difficult for us to determine whether the kwargs are supposed to be passed in __init__ or in __call__

There shouldn't be kwargs that can only be set in __init__ and not __call__, so this isn't something that vLLM should need to worry about.

This will be fixed in the fast image processors that will be available in Transformers v4.49.0

[DarkLight1337]

@hmellor I found that Qwen2VLImageProcessor and Qwen2VLImageProcessorFast in the current v4.49.0 still don't support passing min_pixels and max_pixels in __call__ (i.e. preprocess method). Only the size argument can be passed, which fails to be propagated to smart_resize inside the image processor since that call uses min_pixels and max_pixels rather than size.

[hmellor]

Yeah it appears that the PRs that should resolve this are yet to be merged. For reference they are:

• Most processors - Remove differences between init and preprocess kwargs for fast image processors huggingface/transformers#36186
• Qwen2-VL specifically - Change Qwen2_VL image processors to have init and call accept the same kwargs huggingface/transformers#36207

[alexxchen]

I found the following code in vllm/inputs/registry.py

    def process_input(self, model_config: "ModelConfig",
                      inputs: ProcessorInputs) -> ProcessorInputs:
        """
        Apply an input processor to an instance of model inputs.

        The model is identified by ``model_config``.
        """
        # Avoid circular import
        from vllm.model_executor.model_loader import get_model_architecture

        model_cls, _ = get_model_architecture(model_config)
        processor = self._get_model_input_processor(model_cls)

        # Handle multimodal processor kwargs with priority:
        #     Inference kwargs -> Init kwargs -> {}
        # If it's empty, it'll fall back to the default kwarg values
        mm_processor_kwargs = resolve_mm_processor_kwargs(
            model_config.mm_processor_kwargs,
            inputs.get("mm_processor_kwargs", {}),  # type: ignore
            processor,
        )

        processed_inputs = processor(
            InputContext(model_config),
            inputs,
            **mm_processor_kwargs,
        )

        if is_encoder_decoder_inputs(processed_inputs):
            self._ensure_mm_kwargs(processed_inputs["encoder"],
                                   mm_processor_kwargs)
            self._ensure_mm_kwargs(processed_inputs["decoder"],
                                   mm_processor_kwargs)
        else:
            self._ensure_mm_kwargs(processed_inputs, mm_processor_kwargs)

        return processed_inputs

The function 'resolve_mm_processor_kwargs' in vllm/utils.py prevent passing max_pixels into the processor

def resolve_mm_processor_kwargs(
    init_kwargs: Optional[Mapping[str, object]],
    inference_kwargs: Optional[Mapping[str, object]],
    callable: Callable[..., object],
    *,
    requires_kw_only: bool = True,
    allow_var_kwargs: bool = False,
) -> Dict[str, Any]:
    """Applies filtering to eliminate invalid mm_processor_kwargs, i.e.,
    those who are not explicit keywords to the given callable (of one is
    given; otherwise no filtering is done), then merges the kwarg dicts,
    giving priority to inference_kwargs if there are any collisions.

    In the case that no kwarg overrides are provided, returns an empty
    dict so that it can still be kwarg expanded into the callable later on.

    If allow_var_kwargs=True, allows for things that can be expanded into
    kwargs as long as they aren't naming collision for var_kwargs or potential
    positional arguments.
    """

How about let the user decide how to filter the parameters by setting requires_kw_only and allow_var_kwargs, this will prevent the same problem from happening in the future.