[Model] Always use Transformers backend for PaliGemma and Gemma3-MM #26715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

DarkLight1337 merged 20 commits into vllm-project:main from DarkLight1337:drop-gemma

Oct 17, 2025

Member

DarkLight1337 commented Oct 13, 2025 •

edited by github-actions bot

Loading

Purpose

Since vLLM doesn't support the special attention mask used by PaliGemma and Gemma3-MM (not to be confused with Gemma3n), this PR removes our custom implementations so Transformers backend is used for these models.

@NickLucche it would be great if you could test if gemma3 works with Transformers backend on TPU!

Test Plan

Transformers backend tests should pass.

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.


          [Chore] Always use Transformers backend for PaliGemma and Gemma3-MM

26da7c5

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 requested a review from hmellor

October 13, 2025 16:25

DarkLight1337 requested a review from ywang96 as a code owner

October 13, 2025 16:25

DarkLight1337 added the ready label

mergify bot commented Oct 13, 2025

Documentation preview: https://vllm--26715.org.readthedocs.build/en/26715/

mergify bot added documentation multi-modality new-model rocm labels


          Fix unintentional change

488b9fb

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

gemini-code-assist bot reviewed

View reviewed changes

Contributor

gemini-code-assist bot left a comment

Code Review

This pull request correctly identifies that the custom vLLM implementations for PaliGemma and Gemma3-MM do not properly handle their special attention masks. The solution to remove these custom implementations and fall back to the Hugging Face Transformers backend is a sound approach that prioritizes correctness. The changes are implemented thoroughly, with corresponding updates to model registries, documentation, and test suites. Notably, the removal of now-obsolete skipped tests and the addition of new tests for the Transformers backend demonstrate good testing practices. I find no high or critical issues in this pull request; it is a solid improvement.

DarkLight1337 added 2 commits

October 13, 2025 16:28


          Keep alphabetical order

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Migrate test

1ab1d79

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 changed the title ~~[Chore]~~ [Chore] Always use Transformers backend for PaliGemma and Gemma3-MM

github-project-automation bot added this to Transformers backend

github-project-automation bot moved this to Todo in Transformers backend


          Fix registry

ab96586

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 removed the ready label

DarkLight1337 added 2 commits

October 13, 2025 16:38


          Update test

60f7c8f

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Remove outdated workaround

93b9e70

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 changed the title ~~[Chore] Always use Transformers backend for PaliGemma and Gemma3-MM~~ [Model] Always use Transformers backend for PaliGemma and Gemma3-MM


          Update doc

81ddc2d

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

hmellor reviewed

View reviewed changes

vllm/model_executor/models/registry.py Show resolved Hide resolved

Member Author

DarkLight1337 commented Oct 13, 2025

@hmellor @zucchini-nlp Seems that do_pan_and_scan=True doesn't work for the Transformers backend of Gemma 3, can you take a look?

DarkLight1337 added 2 commits

October 13, 2025 16:43


          Use registry

e4798c4

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Update prompts

8ddf827

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

hmellor reviewed

View reviewed changes

vllm/model_executor/models/registry.py Outdated Show resolved Hide resolved


          Remove noqa

c0d5db2

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

hmellor approved these changes

View reviewed changes

Member

hmellor left a comment

LGTM

Contributor

zucchini-nlp commented Oct 13, 2025

Yeah, option with do_pan_and_scan=True is not coded for the utility functions since the official released checkpoints have it set as False. I haven't seen much usage of this flag personally, but I can add the code in transformers repo if we want to support it

Though it'll be in v5 release with several breaking changes

mergify bot added the needs-rebase label

DarkLight1337 added 2 commits

October 17, 2025 02:54


          Merge branch 'main' into drop-gemma

6cd442b

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

mergify bot removed the needs-rebase label

DarkLight1337 added the ready label

DarkLight1337 enabled auto-merge (squash)

October 17, 2025 03:08

DarkLight1337 merged commit 8c017b3 into vllm-project:main

56 checks passed

DarkLight1337 deleted the drop-gemma branch

October 17, 2025 05:03

github-project-automation bot moved this from In Progress to Done in Transformers backend

Zhuul pushed a commit to Zhuul/vllm that referenced this pull request


          [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (v…

f2622a5

…llm-project#26715)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request


          [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (v…

9ae68ee

…llm-project#26715)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

lucianommartins mentioned this pull request

[Model] Revert PR #26715: Restore custom PaliGemma and Gemma3-MM impl… #27309

Merged

6 tasks

vllm-bot pushed a commit that referenced this pull request


          [Model] Revert PR #26715: Restore custom PaliGemma and Gemma3-MM impl… (

e05a675

#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>

JorgenTrondsen pushed a commit to JorgenTrondsen/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

1d64e23

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Signed-off-by: jorgentrondsen <jrtrondsen@gmail.com>

JorgenTrondsen pushed a commit to JorgenTrondsen/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

5feb737

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Signed-off-by: jorgentrondsen <jrtrondsen@gmail.com>

JorgenTrondsen pushed a commit to JorgenTrondsen/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

62938f4

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Signed-off-by: jorgentrondsen <jrtrondsen@gmail.com>

Kay-Tian mentioned this pull request

vLLM PR #27309 变更核心文件提醒 Kay-Tian/vllm#10

Closed

usberkeley pushed a commit to usberkeley/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

d34f0b4

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>

albertoperdomo2 pushed a commit to albertoperdomo2/vllm that referenced this pull request


          [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (v…

1db83fa

…llm-project#26715)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

albertoperdomo2 pushed a commit to albertoperdomo2/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

3558eb9

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

alhridoy pushed a commit to alhridoy/vllm that referenced this pull request


          [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (v…

67981a0

…llm-project#26715)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request


          [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (v…

c3448c8

…llm-project#26715)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>

xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request


          [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (v…

406f095

…llm-project#26715)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>

kingsmad pushed a commit to kingsmad/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

fb87bfb

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>

0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request


          [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (v…

cda86b7

…llm-project#26715)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

dc93147

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request


          [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (v…

bee6bf4

…llm-project#26715)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

be4d919

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

Kay-Tian mentioned this pull request

vLLM PR #27309 变更核心文件提醒 Kay-Tian/vllm#46

Closed

Chenyaaang pushed a commit to Chenyaaang/vllm that referenced this pull request


          [Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…

df47c7a

…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

hmellor hmellor approved these changes

ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

Isotr0py Awaiting requested review from Isotr0py

+1 more reviewer

gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Labels

documentation multi-modality new-model ready rocm