Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Model] Implement merged input processor for LLaVA model #10676

Merged
merged 24 commits into from
Dec 7, 2024
Merged
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
7b6c4f1
Add `get_dummy_data` to `MultiModalProcessor`; fix and test `iter_pla…
DarkLight1337 Nov 26, 2024
de8332a
Use merged processor for llava model
DarkLight1337 Nov 26, 2024
8b6804e
format
DarkLight1337 Nov 26, 2024
26e3fdf
Fix typo
DarkLight1337 Nov 26, 2024
93d27bc
Enable the test to pass on V1
DarkLight1337 Nov 26, 2024
d697241
Handle embedding inputs
DarkLight1337 Nov 26, 2024
ca11cc9
format
DarkLight1337 Nov 26, 2024
c32cba9
Merge branch 'main' into llava-mm-processor
DarkLight1337 Nov 27, 2024
6c5c9ca
Fix wrong ndim
DarkLight1337 Nov 27, 2024
0194324
Factor out `merge_placeholders`
DarkLight1337 Nov 27, 2024
09618d0
Fix placeholder maps handling on V0
DarkLight1337 Nov 27, 2024
5501458
Remove unused dummy data code
DarkLight1337 Nov 27, 2024
f3673c7
Update dummy model
DarkLight1337 Nov 27, 2024
37bc008
Enable overriding hf processor and tokenizer; fix `_apply_prompt_repl…
DarkLight1337 Nov 27, 2024
4805a9e
Improve error handling in `_resolve_matches`; merge matches directly
DarkLight1337 Nov 27, 2024
8539008
Avoid hashing
DarkLight1337 Nov 27, 2024
00244c7
Update mapper tests
DarkLight1337 Nov 27, 2024
a00f541
Merge branch 'main' into llava-mm-processor
DarkLight1337 Dec 4, 2024
b31f8d4
Avoid calling input mapper in the first place
DarkLight1337 Dec 4, 2024
711cd38
Fix missing `multi_modal_kwargs` in dummy data
DarkLight1337 Dec 5, 2024
a11c6b2
Update dummy model
DarkLight1337 Dec 5, 2024
1d5a4d4
proper processing
ywang96 Dec 6, 2024
000736b
Patch pixtral processor
DarkLight1337 Dec 6, 2024
1485c05
Fix double counting of `mm_counts`
DarkLight1337 Dec 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Enable the test to pass on V1
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
  • Loading branch information
DarkLight1337 committed Nov 26, 2024
commit 93d27bc87671e28435202091d5bb828f876b6213
4 changes: 4 additions & 0 deletions vllm/v1/engine/mm_input_mapper.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ def __init__(
model_config: ModelConfig,
mm_registry: MultiModalRegistry = MULTIMODAL_REGISTRY,
):
self.model_config = model_config
self.mm_registry = mm_registry
self.multi_modal_input_mapper = mm_registry.create_input_mapper(
model_config)
Expand All @@ -22,6 +23,9 @@ def process_inputs(
mm_data: MultiModalDataDict,
mm_processor_kwargs: Optional[Dict[str, Any]],
) -> List[MultiModalKwargs]:
if self.mm_registry.has_processor(self.model_config):
return [MultiModalKwargs(mm_data)] # Already processed

image_inputs = mm_data["image"]
if not isinstance(image_inputs, list):
image_inputs = [image_inputs]
Expand Down