Skip to content

[V1][Pixtral-HF] Add custom slice_encoder_output for Pixtral #13080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 34 commits into from
Closed

[V1][Pixtral-HF] Add custom slice_encoder_output for Pixtral #13080

wants to merge 34 commits into from

Conversation

lk-chen
Copy link
Collaborator

@lk-chen lk-chen commented Feb 11, 2025 •
edited by github-actions bot
Loading

Prepare for #11409

This PR allows model to override _gather_encoder_outputs logic. Models like Pixtral need to take special tokens (break/end token) into consideration while gathering soft tokens.

See this comment for more context.

Test

Tested by running

VLLM_USE_V1=1 python examples/offline_inference/vision_language.py --model pixtral_hf --num-prompts=2

with #11409 patched, results are the same as V0.

cc @WoosukKwon @comaniac @ywang96

Linkun Chen and others added 30 commits November 18, 2024 05:52
Patch multi_modal_placeholders to RequestOutput
678c291 
* confirm that `offline_inference_vision_language.py` and `benchmark_throughput.py` runs
* FIXME: the placeholders in output, however, is empty
 - will fix in next commit

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
pipe multi_modal_placeholders from intput to final output
a1cdcb3 
* add test for pixtral
* fix a typo

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@WoosukKwon
[V1] Add code owners for V1 (vllm-project#10397)
f60964a 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@youkaichao
[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…
578e482 
…ect#10383)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@ywang96
[V1] Refactor model executable interface for all text-only language m…
7fa97cf 
…odels (vllm-project#10374)

Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@xuechendi
[CI/Build] Fix IDC hpu [Device not found] issue (vllm-project#10384)
629f512 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@Isotr0py
[Bugfix][CPU] Fix CPU embedding runner with tensor parallel (vllm-pro…
7539ab8 
…ject#10394)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@youkaichao
[platforms] refactor cpu code (vllm-project#10402)
bce660d 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@jikunshang
[Hardware] [HPU]add mark_step for hpu (vllm-project#10239)
305708b 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@imkero
[Bugfix] Fix mrope_position_delta in non-last prefill chunk (vllm-pro…
871a773 
…ject#10403)

Signed-off-by: imkero <kerorek@outlook.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@wchen61
[Misc] Enhance offline_inference to support user-configurable paramet… (
242bb53 
vllm-project#10392)

Signed-off-by: wchen61 <wchen61@foxmail.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@DarkLight1337
Fix initialization
f5312d3 
Signed-off-by: Cyrus Leung <tlleungac@connect.ust.hk>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@DarkLight1337
Run isort
439e324 
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@DarkLight1337
isort
60815f2 
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@DarkLight1337
isort
ec46755 
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@Isotr0py
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
ce3ae6f 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@HollowMan6
[Bugfix] Ignore ray reinit error when current platform is ROCm or XPU (…
466b2cf 
…vllm-project#10375)

Signed-off-by: Hollow Man <hollowman@opensuse.org>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
update RequestOutput.__init__() to take multi_modal_placeholders as…
3f092ce 
… optional argument

also require it to be passed as kwargs, to avoid breaking existing code.

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
update RequestOutput.__init__() to take multi_modal_placeholders as…
dd8427e 
… optional argument

also require it to be passed as kwargs, to avoid breaking existing code.

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
update RequestOutput.__init__() to take multi_modal_placeholders as…
76ac8b0 
… optional argument

also require it to be passed as kwargs, to avoid breaking existing code.

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@lk-chen
Merge branch 'vllm-project:main' into main
904e925
disable mypy type check
c963a25 
mypy is not smart enough to validate kwargs

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
disable mypy type check
470fbd3 
mypy is not smart enough to validate kwargs

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
remove unnecessary debug code
550be23 
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@lk-chen
Merge branch 'vllm-project:main' into main
1eb4d96
@lk-chen
Merge branch 'vllm-project:main' into main
9b002b0
@lk-chen
Merge branch 'vllm-project:main' into main
436beb2
@lk-chen
Merge branch 'vllm-project:main' into main
bbc6420
@lk-chen
Merge branch 'vllm-project:main' into main
5254415
@lk-chen
Merge branch 'vllm-project:main' into main
6077919
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@lk-chen lk-chen mentioned this pull request Feb 11, 2025
Closed
@mergify mergify bot added the v1 label Feb 11, 2025
@lk-chen
[V1] Enhance check when clicing encoder output
d8f785a 
Prepare for vllm-project#11409

For pixtral model, we need to insert placeholders in the middle of
encoder output, to fit into whole soft embedding. This case makes
slicing operation tricky. This PR raises assertion if something's
off.

Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen
[V1][Pixtral-HF] Add custom slice_encoder_output for Pixtral
89f243b 
Signed-off-by: Linkun Chen <github@lkchen.net>
@ywang96 ywang96 self-assigned this Feb 12, 2025
@DarkLight1337
Copy link
Member

DarkLight1337 commented Feb 12, 2025 •
edited
Loading

For your reference, I have added a mapping from encoder outputs to embeddings in the outputs of Molmo multi-modal processor (#12966, see feat_is_patch and embed_is_patch) so that there is no need to define a custom hook inside the model. Do you think it's feasible to do the same for this model?

@lk-chen lk-chen mentioned this pull request Mar 5, 2025
Merged
@lk-chen lk-chen marked this pull request as draft March 5, 2025 21:31
@lk-chen
Copy link
Collaborator Author

lk-chen commented Mar 6, 2025

Closing in favor of #14275

@lk-chen lk-chen closed this Mar 6, 2025
@lk-chen lk-chen deleted the pixtral_hf branch April 29, 2025 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

Assignees

@ywang96 ywang96

Labels
v1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.