Skip to content

Refactor Transformers backend to use mixins #26906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 21 commits into from
Oct 16, 2025
Merged

Refactor Transformers backend to use mixins #26906

merged 21 commits into from
Oct 16, 2025

Conversation

@hmellor
Copy link
Member

@hmellor hmellor commented Oct 15, 2025 •
edited by github-actions bot
Loading

This PR refactors the Transformers backend to use mixin classes to add model specific functionality like multi-modality, mixture-of-experts or pooling.

This enables us to compose these functionalities to create new Transformers backend classes composing any of these features. The two new classes added by this PR are for multimodal embedding and multimodal sequence classification.

It also adds __getattr__ to __init__.py so that if a user requests a new combination of these functionalities, they are prompted to create an issue asking us to add it.

This PR enables #26715 by adding support for multimodal embedding.

This PR also:

@hmellor
Allow ModelConfig to create any Transformers backend cls name
156c631 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Update the Transformers backend to use mixins (+mm pooling classes)
a0ed0ca 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Use compilation in transformers test to catch dynamo issues
9cf3b62 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@mergify mergify bot added ci/build deepseek Related to DeepSeek models new-model Requests to new models labels Oct 15, 2025
@hmellor
typo
6a2c728 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
gemini-code-assist[bot]
gemini-code-assist bot reviewed Oct 15, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request is a significant and well-executed refactoring of the Transformers backend. It modularizes the backend by introducing mixin classes for different functionalities like multi-modality, Mixture-of-Experts (MoE), and pooling. This greatly improves code organization, maintainability, and extensibility. The new structure, with a dedicated transformers package, is much cleaner. I've found one potential high-severity issue related to the torch.compile configuration for multimodal models that could lead to incorrect behavior or errors.

@hmellor
Revert change that should not have been committed
69a5877 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Oct 15, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

DarkLight1337
DarkLight1337 reviewed Oct 15, 2025
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clean up, just one comment

@hmellor
Copy link
Member Author

hmellor commented Oct 15, 2025

Still running local testing, so no need to enable CI just yet

@mergify
Copy link

mergify bot commented Oct 15, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Oct 15, 2025
@hmellor
Fix syntax error
95fb99d 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add hf_config and hf_text_config as non-init fields to ModelConfig
3d0ddf4 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Fix forward references
98297b5 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Move torch compile util function to utils.py
eda1b76 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Isotr0py
Isotr0py reviewed Oct 15, 2025
View reviewed changes
@DarkLight1337 DarkLight1337 enabled auto-merge (squash) October 15, 2025 15:30
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 15, 2025
@hmellor
Fix get_language_model
a39cde7 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Skip Qwen because mrope was changed recently
5af9639 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@mergify mergify bot added the multi-modality Related to multi-modality (#4194) label Oct 15, 2025
@hmellor
Copy link
Member Author

hmellor commented Oct 15, 2025

#24172 broke Qwen VL models for the Transformers backend because it relied on MRotaryEmbedding.get_input_positions_tensor, which has now been removed.

We will forward fix this.

@hmellor
Fix mrope support
536a2ae 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Copy link
Member Author

hmellor commented Oct 15, 2025

It ended up not being too hard, I've fixed mrope for Transformers backend in this PR

@hmellor
Copy link
Member Author

hmellor commented Oct 16, 2025

royokong/e5-v has a very old config that is not compatible with the way processors currently work in Transformers.

I've switched to the same Gemma 3 checkpoint used in the multimodal classifier test from your PR @DarkLight1337

@hmellor
Switch to Gemma 3
c28097d 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor hmellor mentioned this pull request Oct 16, 2025
Closed
1 task
@hmellor hmellor moved this from Todo to In Progress in Transformers backend Oct 16, 2025
@hmellor
Gemma 3 scaling fix
b5819a1 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Merge branch 'main' into transformers-backend-mixins
66427a6
@DarkLight1337 DarkLight1337 merged commit fb5e10d into vllm-project:main Oct 16, 2025
53 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Transformers backend Oct 16, 2025
@hmellor hmellor deleted the transformers-backend-mixins branch October 16, 2025 21:51
Zhuul pushed a commit to Zhuul/vllm that referenced this pull request Oct 17, 2025
@hmellor @Zhuul
Refactor Transformers backend to use mixins (vllm-project#26906)
8dcec71 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@chaunceyjiang chaunceyjiang mentioned this pull request Oct 17, 2025
Merged
5 tasks
lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025
@hmellor @lywa1998
Refactor Transformers backend to use mixins (vllm-project#26906)
470b10e 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
albertoperdomo2 pushed a commit to albertoperdomo2/vllm that referenced this pull request Oct 23, 2025
@hmellor @albertoperdomo2
Refactor Transformers backend to use mixins (vllm-project#26906)
06c9b9e 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025
@hmellor @alhridoy
Refactor Transformers backend to use mixins (vllm-project#26906)
04166e4 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 24, 2025
@hmellor @xuebwang-amd
Refactor Transformers backend to use mixins (vllm-project#26906)
7357a97 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 24, 2025
@hmellor @xuebwang-amd
Refactor Transformers backend to use mixins (vllm-project#26906)
e4d385e 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025
@hmellor @0xrushi
Refactor Transformers backend to use mixins (vllm-project#26906)
e426ef9 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025
@hmellor @0xrushi
Refactor Transformers backend to use mixins (vllm-project#26906)
4e3fae6 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@Isotr0py Isotr0py Isotr0py approved these changes

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo simon-mo is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@houseroad houseroad Awaiting requested review from houseroad houseroad is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg ProExpertProg is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build deepseek Related to DeepSeek models multi-modality Related to multi-modality (#4194) new-model Requests to new models ready ONLY add when PR is ready to merge/full CI is needed

Projects

Transformers backend
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: google/embeddinggemma-300m when using transformers backend doesn't match the output of Sentence Transformers (or model_impl="vllm")

3 participants

@hmellor @Isotr0py @DarkLight1337