[Core] Cast multimodal input in hf processor #18862
Open
+22
−25
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lgeiger
requested review from
WoosukKwon,
robertgshaw2-redhat,
njhill,
ywang96,
comaniac,
alexm-redhat and
DarkLight1337
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
mergify
bot
added
multi-modality
lgeiger
commented
May 28, 2025
|
Overall this approach does make more sense than what I originally did in #18756, thanks! |
lgeiger
force-pushed
the
non-blocking-casting
branch
from
b6d42a2 to
a25576c
Compare
DarkLight1337
approved these changes
May 29, 2025
github-actions
bot
added
the
ready
|
Can you fix the failing tests? Looks like you need to import |
lgeiger
commented
May 29, 2025
auto-merge was automatically disabled
May 29, 2025 16:46
Head branch was pushed to by a user without write access
lgeiger
force-pushed
the
non-blocking-casting
branch
from
e61670b to
0292449
Compare
Ah sorry, should be fixed in 0292449. Looks like the type annotation needs to be a string to support older versions of python. |
auto-merge was automatically disabled
May 29, 2025 18:50
Head branch was pushed to by a user without write access
|
Looks like not all items in the dict are tensors which breaks CI. 9d0d47c should fix that. |
|
Looks like V1 test is hanging in this PR, can you investigate it? |
lgeiger
force-pushed
the
non-blocking-casting
branch
from
9d0d47c to
b48017f
Compare
I rebased onto main, let's see if this fixes it or gives some more actionable error message |
|
Looks like the v1-tests are still failing. I'm waiting for huggingface to approve my model access request, then I can try and reproduce locally |
lgeiger
force-pushed
the
non-blocking-casting
branch
from
95deab1 to
b1344b4
Compare
|
Looks like the deadlocking tests are due to the fact that vllm uses fork as a default multiprocessing method. On and huggingface tokenizers already complain about not being compatible with fork On main this doesn't cause any deadlocks but it looks like with this change it breaks. I can reproduce this locally as well. Whereas running them in one go deadlocks: Setting Not sure how to best deal with this. I guess switching the default vllm multiprocessing method might be a larger change. |
|
Why would this PR cause the tests to fail when they weren't before though? |
You're right it's surprising especially since we don't actually change where the huggingface preprocessor is called. |
|
cc @njhill any idea? |
|
The tokenizers warning is a red herring and shouldn't be an issue. I don't think we should change the mp method in the test to workaround. If you can repro locally, you could check where things may be stuck by running with env var |
Here are the dumps from both processes: |
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
lgeiger
force-pushed
the
non-blocking-casting
branch
from
b1344b4 to
5d5c4b4
Compare
|
Looks like it's stuck in transformers here: huggingface/transformers@a31fa21#diff-d3478155ac25ae1107d16a4464001bfd54770bf12cd9bab881233a1b4d216e3fR240 🤔 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a follow up to #18756 and instead directly does the multimodal input casting as part of the huggingface preprocessing.
I think this is a bit cleaner and has two advantages: It moves the blocking casting/copy off the main thread and now does the conversion before serialisation which also reduces the amount of data that needs to be serialised and de-serialised. @DarkLight1337 Let me know if you see any disadvantages of doing this.
Before:
After:
I've done some quick benchmark and for some models I'm seeing very small improvement in throughput (0.5%-0.7%).