[Core][VLM] Add precise multi-modal placeholder tracking #8346

petersalas · 2024-09-10T22:02:12Z

Currently, multi-modal prompt placeholders are related to the multi-modal embeddings exclusively by index, i.e. the first placeholder token in the prompt must correspond to the first MM embedding vector, etc. This adds a mechanism for tracking multi-modal placeholder ranges precisely which allows multi-modal models to be used with chunked prefill and is a prerequisite for allowing multi-modal models to be used with prefix caching enabled (see #8348).

For a model to use precise tracking, it:

Sets the placeholder ranges on the LLM inputs (these should align 1-1 with the multi-modal items for the matching key)
Uses the computed placeholder/embedding index tensors from the AttentionMetadata to merge the embeddings instead of matching on token ID.

github-actions · 2024-09-10T22:02:26Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

DarkLight1337

Some initial comments.

tests/multimodal/test_utils.py

vllm/model_executor/models/clip.py

vllm/model_executor/models/ultravox.py

vllm/multimodal/image.py

vllm/model_executor/models/ultravox.py

DarkLight1337

Added some new comments.

Let's also test whether chunked prefill works with multimodal data for online serving since only prompt_token_ids are passed in that case.

vllm/inputs/data.py

vllm/inputs/registry.py

vllm/sequence.py

DarkLight1337 · 2024-09-24T11:02:14Z

As a preliminary test, I ran the VLM CI against this PR. Please take a look at the CI failures and fix them.

DarkLight1337 · 2024-09-25T02:01:29Z

PTAL at the failing workers test as well.

DarkLight1337 · 2024-09-27T01:55:11Z

Please merge from main again to resolve the test failures.

DarkLight1337 · 2024-09-29T02:01:17Z

It looks like the format of multi_modal_data inputted to the input mapper has been changed by you to always be in multi-input form (i.e. list of single input instances), breaking some of the models that don't allow multi-input.

vllm/multimodal/base.py

alexm-redhat · 2024-10-09T19:12:58Z

vllm/multimodal/base.py

+            intersection = range(max(positions.start, placeholder.start),
+                                 min(positions.stop, placeholder.stop))
+
+            if not intersection:


Is there a real use case where intersection is an empty set?

Yes, a couple scenarios:

If prefix caching is enabled (following integration with [Core][VLM] Add support for prefix caching for multi-modal models #8348) we can skip multi-modal handling altogether for any multi-modal items whose corresponding blocks are cached.

In chunked prefill, if a multi-modal item is in a section of the prompt that isn't currently being prefilled it can be ignored for that inference.

alexm-redhat · 2024-10-09T19:24:52Z

vllm/attention/backends/flash_attn.py

@@ -466,6 +484,7 @@ def build(self, seq_lens: List[int], query_lens: List[int],
            num_prefill_tokens=self.num_prefill_tokens,
            num_decode_tokens=num_decode_tokens,
            seq_lens=seq_lens,
+            multi_modal_placeholder_maps=placeholder_maps,


What code part is using this added parameter here? Ditto for all other attention changes.

Any model that uses merge_multimodal_embeddings_from_map -- this change only adds it for Ultravox but that's really only to limit scope. In time I'd expect any multi-modal model that merges multi-modal and text embeddings to use merge_multimodal_embeddings_from_map instead of merge_multimodal_embeddings (barring any tradeoffs I might be neglecting).

alexm-redhat · 2024-10-10T14:25:38Z

vllm/model_executor/models/ultravox.py

-                inputs_embeds = merge_multimodal_embeddings(
-                    input_ids, inputs_embeds, audio_embeddings,
-                    _AUDIO_PLACEHOLDER_TOKEN)
+                merge_multimodal_embeddings_from_map(


It looks like "merge_multimodal_embeddings_from_map" is used only for ultravox model, where this kind of merging happens for the other models?

See the other comment -- in time I'd expect usage of merge_multimodal_embeddings to be replaced with merge_multimodal_embeddings_from_map.

How can i use it in other multimodal models such as llava? Do multi_modal_placeholder_index_maps have other keys besides "audio" use for image?

alexm-redhat

@petersalas did you had a chance to do performance comparison of non-chunked vs chunked prompt execution with your changes?

petersalas · 2024-10-10T22:59:07Z

@petersalas did you had a chance to do performance comparison of non-chunked vs chunked prompt execution with your changes?

Not rigorously -- I did some ad-hoc runs of offline_inference_audio_language.py at various batch sizes as a smoke test (and came to the conclusion that throughput was slightly worse with chunked prefill than without). Is there a particular setup I should benchmark?

edit: here's the ad-hoc test I ran:

chunked prefill on

python examples/offline_inference_audio_language.py --num-prompts 100
Processed prompts: 100%|█| 100/100 [00:06<00:00, 14.87it/s, est. speed input: 2156.88 toks/s, output: 758.77 toks/s]

chunked prefill off

python examples/offline_inference_audio_language.py --num-prompts 100
Processed prompts: 100%|█| 100/100 [00:06<00:00, 15.38it/s, est. speed input: 2230.79 toks/s, output: 813.09 toks/s]

afeldman-nm

Thanks for the PR. Left a few nits but overall LGTM.

vllm/multimodal/base.py

tests/models/decoder_only/audio_language/test_ultravox.py

vllm/attention/backends/abstract.py

vllm/multimodal/base.py

ywang96

@petersalas Thanks for this great work and apologies for the very delayed review! I have left comments/suggestions so please take a look!

I previously discussed with @WoosukKwon on how we want to support chunked prefill for multimodal models and we landed on the similar idea as this PR, so this is definitely on the right track.

A few things to note from this PR:

With this PR and chunked prefill enabled, embedding generation through encoder and projector for a single multimodal data item can be executed multiple times unnecessarily, especially if the total budget of a batch is small. Therefore, we're essentially trading off full decoder prefill + one-time encoder prefill for chunked decoder prefill + potential repeated encoder prefill.

This should be okay for now since the encoder call is not expensive compared to the decoder (I wonder if you have profiled this tradeoff yourself?), but eventually we should definitely design a cache system for these multimodal embeddings so that we only need to generate them once and extract needed portions every step.
The PR only enables models that use repeat_and_pad_placeholder_tokens. I think later we should decouple the ranges generation from this function to provide some flexibility, or provide another interface where the developer can provide their own ranges(I've been thinking about the best way to do this especially with our re-arch work.)
We also need to update both user and developer documentation for how to enable their models with chunked prefill, but that doesn't need to happen this PR.

Thanks again for this contribution!

vllm/attention/backends/abstract.py

vllm/model_executor/models/blip2.py

vllm/attention/backends/abstract.py

vllm/multimodal/base.py

vllm/worker/model_runner.py

vllm/attention/backends/utils.py

mergify · 2024-10-30T21:57:17Z

This pull request has merge conflicts that must be resolved before it can be
merged. @petersalas please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Richard Liu <ricliu@google.com>

Signed-off-by: Peter Salas <peter@fixie.ai>

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai>

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Loc Huynh <jc1da.3011@gmail.com>

…t#10045) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Loc Huynh <jc1da.3011@gmail.com>

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>

…t#10045) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai>

…t#10045) Signed-off-by: Peter Salas <peter@fixie.ai>

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>

…t#10045) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>

DarkLight1337 requested review from mgoin and ywang96 September 11, 2024 02:07

DarkLight1337 reviewed Sep 11, 2024

View reviewed changes

tests/multimodal/test_utils.py Outdated Show resolved Hide resolved

vllm/model_executor/models/clip.py Outdated Show resolved Hide resolved

vllm/model_executor/models/ultravox.py Outdated Show resolved Hide resolved

vllm/multimodal/image.py Outdated Show resolved Hide resolved

ywang96 self-assigned this Sep 11, 2024

DarkLight1337 mentioned this pull request Sep 11, 2024

[RFC]: Multi-modality Support on vLLM #4194

Open

53 tasks

DarkLight1337 reviewed Sep 12, 2024

View reviewed changes

vllm/model_executor/models/ultravox.py Outdated Show resolved Hide resolved

DarkLight1337 mentioned this pull request Sep 12, 2024

[Core] generate from input embeds #6869

Closed

mgoin reviewed Sep 12, 2024

View reviewed changes

vllm/model_executor/models/ultravox.py Outdated Show resolved Hide resolved

ywang96 mentioned this pull request Sep 12, 2024

[Hotfix][Core][VLM] Disable chunked prefill by default and prefix caching for multimodal models #8425

Merged

petersalas mentioned this pull request Sep 12, 2024

[Core][VLM] Add support for prefix caching for multi-modal models #8348

Closed

DarkLight1337 mentioned this pull request Sep 17, 2024

[Doc] Compatibility matrix for mutual exclusive features #8512

Merged

petersalas force-pushed the psalas/placeholder-positions branch from e457298 to aa756bf Compare September 18, 2024 19:07

DarkLight1337 reviewed Sep 19, 2024

View reviewed changes

vllm/inputs/data.py Show resolved Hide resolved

vllm/inputs/registry.py Outdated Show resolved Hide resolved

vllm/sequence.py Outdated Show resolved Hide resolved

alexm-redhat reviewed Oct 9, 2024

View reviewed changes

alexm-redhat reviewed Oct 10, 2024

View reviewed changes

petersalas requested review from WoosukKwon, zhuohan123, youkaichao and njhill as code owners October 10, 2024 23:15

afeldman-nm approved these changes Oct 11, 2024

View reviewed changes

vllm/multimodal/base.py Outdated Show resolved Hide resolved

tests/models/decoder_only/audio_language/test_ultravox.py Show resolved Hide resolved

vllm/attention/backends/abstract.py Outdated Show resolved Hide resolved

vllm/multimodal/base.py Outdated Show resolved Hide resolved

petersalas force-pushed the psalas/placeholder-positions branch from 84d12e7 to 366d65a Compare October 24, 2024 15:52

ywang96 reviewed Oct 28, 2024

View reviewed changes

richardsliu pushed a commit to richardsliu/vllm that referenced this pull request Nov 4, 2024

[Core][VLM] Add precise multi-modal placeholder tracking (vllm-projec…

a13a962

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Richard Liu <ricliu@google.com>

petersalas added a commit to fixie-ai/vllm that referenced this pull request Nov 5, 2024

[OpenVINO] Fix regression from vllm-project#8346

52247f4

Signed-off-by: Peter Salas <peter@fixie.ai>

DarkLight1337 pushed a commit that referenced this pull request Nov 6, 2024

[Model][OpenVINO] Fix regressions from #8346 (#10045)

ffc0f2b

Signed-off-by: Peter Salas <peter@fixie.ai>

hissu-hyvarinen pushed a commit to ROCm/vllm that referenced this pull request Nov 6, 2024

[Core][VLM] Add precise multi-modal placeholder tracking (vllm-projec…

9789558

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai>

DarkLight1337 mentioned this pull request Nov 9, 2024

[CI/Build] Split up models tests #10069

Merged

JC1DA pushed a commit to JC1DA/vllm that referenced this pull request Nov 11, 2024

[Core][VLM] Add precise multi-modal placeholder tracking (vllm-projec…

a143177

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Loc Huynh <jc1da.3011@gmail.com>

JC1DA pushed a commit to JC1DA/vllm that referenced this pull request Nov 11, 2024

[Model][OpenVINO] Fix regressions from vllm-project#8346 (vllm-projec…

b21585a

…t#10045) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Loc Huynh <jc1da.3011@gmail.com>

WoosukKwon mentioned this pull request Nov 12, 2024

[V1] Support VLMs with fine-grained scheduling #9871

Merged

DarkLight1337 mentioned this pull request Nov 13, 2024

[Feature]: Chunked prefill for multimodal models #10290

Closed

1 task

sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024

[Core][VLM] Add precise multi-modal placeholder tracking (vllm-projec…

f84ac74

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>

sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024

[Model][OpenVINO] Fix regressions from vllm-project#8346 (vllm-projec…

4edbd2f

…t#10045) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>

imkero mentioned this pull request Nov 16, 2024

[Bugfix] Fix M-RoPE position calculation when chunked prefill is enabled #10388

Merged

lk-chen mentioned this pull request Nov 18, 2024

[VLM] Report multi_modal_placeholders in output #10407

Merged

sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024

[Core][VLM] Add precise multi-modal placeholder tracking (vllm-projec…

5f2078c

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai>

sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024

[Model][OpenVINO] Fix regressions from vllm-project#8346 (vllm-projec…

7d87666

…t#10045) Signed-off-by: Peter Salas <peter@fixie.ai>

LeiWang1999 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Mar 26, 2025

[Core][VLM] Add precise multi-modal placeholder tracking (vllm-projec…

e4a8586

…t#8346) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>

LeiWang1999 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Mar 26, 2025

[Model][OpenVINO] Fix regressions from vllm-project#8346 (vllm-projec…

55df7fe

…t#10045) Signed-off-by: Peter Salas <peter@fixie.ai> Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>

Uh oh!

[Core][VLM] Add precise multi-modal placeholder tracking #8346

[Core][VLM] Add precise multi-modal placeholder tracking #8346

Uh oh!

Conversation

petersalas commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 10, 2024

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkLight1337 commented Sep 25, 2024

Uh oh!

DarkLight1337 commented Sep 27, 2024

Uh oh!

DarkLight1337 commented Sep 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexm-redhat Oct 9, 2024

Choose a reason for hiding this comment

Uh oh!

petersalas Oct 10, 2024

Choose a reason for hiding this comment

Uh oh!

alexm-redhat Oct 9, 2024

Choose a reason for hiding this comment

Uh oh!

petersalas Oct 10, 2024

Choose a reason for hiding this comment

Uh oh!

alexm-redhat Oct 10, 2024

Choose a reason for hiding this comment

Uh oh!

petersalas Oct 10, 2024

Choose a reason for hiding this comment

Uh oh!

QiuJingkai Nov 13, 2024

Choose a reason for hiding this comment

Uh oh!

alexm-redhat left a comment

Choose a reason for hiding this comment

Uh oh!

petersalas commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afeldman-nm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ywang96 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

petersalas commented Sep 10, 2024 •

edited

Loading

DarkLight1337 commented Sep 24, 2024 •

edited

Loading

DarkLight1337 commented Sep 29, 2024 •

edited

Loading

petersalas commented Oct 10, 2024 •

edited

Loading

ywang96 left a comment •

edited

Loading