[Model][Gemma3] Cast image pixel values already on CPU

[lgeiger]

Gemma3 by default uses the bfloat16 dtype, however the hugging face processor outputs the pixel values of the 896x896 images in float32 which means these values need to be casted to the gemma3 data type. This currently happens on GPU.

This PR moves the casting to CPU during the input processing which halves the amount of image data that needs to be copied from CPU to GPU and should also save a bit of GPU memory in prompts with lots of images.

This change seems to improve throughput by 2.4 % of the 4B model on a L40s GPU and improves latency, measured using the following script:

vllm serve google/gemma-3-4b-it --disable-log-requests
python benchmarks/benchmark_serving.py --backend openai-chat --model google/gemma-3-4b-it --endpoint /v1/chat/completions --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat --hf-split train --num-prompts 5000

Baseline (based on #18710):

============ Serving Benchmark Result ============
Successful requests:                     985
Benchmark duration (s):                  89.50
Total input tokens:                      95454
Total generated tokens:                  115301
Request throughput (req/s):              11.01
Output token throughput (tok/s):         1288.35
Total Token throughput (tok/s):          2354.93
---------------Time to First Token----------------
Mean TTFT (ms):                          52647.14
Median TTFT (ms):                        44711.12
P99 TTFT (ms):                           84957.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          112.01
Median TPOT (ms):                        111.66
P99 TPOT (ms):                           283.39
---------------Inter-token Latency----------------
Mean ITL (ms):                           109.95
Median ITL (ms):                         50.29
P99 ITL (ms):                            423.10
==================================================

This PR:

============ Serving Benchmark Result ============
Successful requests:                     985
Benchmark duration (s):                  87.41
Total input tokens:                      95454
Total generated tokens:                  115232
Request throughput (req/s):              11.27
Output token throughput (tok/s):         1318.31
Total Token throughput (tok/s):          2410.35
---------------Time to First Token----------------
Mean TTFT (ms):                          52161.96
Median TTFT (ms):                        43351.32
P99 TTFT (ms):                           82530.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          102.06
Median TPOT (ms):                        100.74
P99 TPOT (ms):                           263.27
---------------Inter-token Latency----------------
Mean ITL (ms):                           100.50
Median ITL (ms):                         48.11
P99 ITL (ms):                            405.93
==================================================

I'm very new to the vllm code base, so please let me know if this could lead to problems with other dtype/quantisation configs. I'm also not sure whether this is the best benchmark to evaluate this change or how noisy these benchmarks tend to be.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Looks reasonable, thanks for fixing! cc @ywang96 @Isotr0py perhaps we should do this for the other processors as well?

[Isotr0py]

perhaps we should do this for the other processors as well?

I think we can cast dtype in BaseMultimodalProcessor's apply method, so that we con't need to make modification for each processor.