[Core][Frontend][Doc] Initial support for LLaVA-NeXT and GPT-4V Chat Completions API #3978

DarkLight1337 · 2024-04-10T17:35:49Z

To combat scope creep, this PR has been split into smaller ones.

VLM backend
- [Core] Support image processor #4197
- [Model] Initial support for LLaVA-NeXT #4199
OpenAI API server
- ~~[Frontend] Support GPT-4V Chat Completions API #4200~~ [Frontend] Add OpenAI Vision API Support #5237
QA
- [Bugfix] Fix LLaVA-NeXT #5380

The branch associated with this PR has been frozen (except for critical fixes). Once all dependencies have been merged, I will compare this branch against the merged (main) branch to verify that I didn't miss any changes.

- Refactor `OpenAIServingChat` and add function for loading image - Move `pillow` dev dependency to common - Add example chat template for LLaVA model

- Add general guide for using VLMs - Add LLavA to list of supported models

- Move `ServerRunner` to common file

- Incorrect loading of config (also rename `openai_api` to `image_openai`) - Incorrect await of stream generator

- Also, use the type definitions from `openai` directly

…ions API and legacy Completions API

DarkLight1337 · 2024-04-18T09:45:18Z

I have just added support for LLaVA-NeXT, with one big caveat: the size of the input image is fixed, otherwise the feature size (i.e. number of <image> tokens to duplicate) would vary depending on the runtime input. This prevents us from taking full advantage of the extra resolution. Still, this provides us access to a 34b model which should improve over their 7b and 13b LLaVA-1.5 models.

DarkLight1337 · 2024-04-19T07:15:02Z

These force pushes consolidate the fixes to the LLaVA test and example code.

- Note that we now load the images directly instead of from `.pt` files

DarkLight1337 · 2024-04-19T08:02:50Z

@ywang96 I think that this PR is suffering from scope creep. Perhaps I should break apart the changes into smaller segments to facilitate the conversation in #4194? I could split the changes as follows, with each item being its own PR:

VLM backend
a. Refactor MultiModalData to support image processing; refactor LLaVA-1.5 accordingly. ([Core] Support image processor #4197)
b. Introduce LLaVA-NexT ~~along with the refactored LLaVA-1.5~~ ([Model] Initial support for LLaVA-NeXT #4199) [depends on 1(a)]
OpenAI API server
a. Refactor OpenAPI backend ([Frontend] Refactor prompt processing #4028)
b. Add GPT-4V support and provide LLaVA chat template ([Frontend] Support GPT-4V Chat Completions API #4200) [depends on 1(a) 2(a)]

Edit: Added links to the child PRs.

ywang96 · 2024-04-19T08:06:54Z

@ywang96 I think that this PR is suffering from scope creep. Perhaps I should break apart the changes into smaller segments to facilitate the conversation in #4194? I could split the changes as follows (listed in the form of a dependency tree):

VLM backend
a. Refactor MultiModalData to support image processing.
b. Introduce LLaVA-NexT along with the refactored LLaVA-1.5

OpenAI API server
a. Refactor OpenAPI backend (i.e. Support VLM model and GPT4V API #2058)
b. Add GPT-4V support [also depends on 1(a)]

I agree - I think OpenAI API server will be a good starting point since the interface should agree with OpenAI protocol anyways, and I'm sorry that this PR suffered :/

One suggestion I have is for a big change like this - it's probably good to have a series of PRs anyways. Take a look at Speculative decoding or Chunked Prefill - those are great examples.

DarkLight1337 · 2024-04-19T09:25:58Z

I have created the child PRs.

- These changes are propagated to the child PRs

- Note that LLaVA-1.5 has been refactored to facilitate this

DarkLight1337 · 2024-06-10T15:41:22Z

All of the child PRs have been completed, so I'm closing this now.

DarkLight1337 added 3 commits April 10, 2024 16:31

Add basic support for OpenAI image input API

874a581

- Refactor `OpenAIServingChat` and add function for loading image - Move `pillow` dev dependency to common - Add example chat template for LLaVA model

Update documentation

607434e

- Add general guide for using VLMs - Add LLavA to list of supported models

Add tests for OpenAI image input API and image loader

aaa6bfe

- Move `ServerRunner` to common file

DarkLight1337 changed the title ~~[Doc][Frontend] Extexnd OpenAI-compatible server to support GPT-4V Chat Completions API~~ [Doc][Frontend] Support GPT-4V Chat Completions API Apr 10, 2024

ywang96 self-assigned this Apr 10, 2024

DarkLight1337 added 5 commits April 11, 2024 03:10

Merge branch 'upstream' into openai-vision-api

26e7b2a

Apply formatter

44829b5

Place image before text for llava-hf model

bccb367

Internally enable customization of merging image with text prompt

b9302e8

Fix errors in CI/CD

a44d7d1

- Incorrect loading of config (also rename `openai_api` to `image_openai`) - Incorrect await of stream generator

This was referenced Apr 11, 2024

[Bug]: Model architectures ['LlavaForCausalLM'] are not supported for now in vllm 0.4.0.post1 #4008

Closed

[Frontend] Support complex message content for chat completions endpoint #3467

Merged

DarkLight1337 and others added 18 commits April 12, 2024 02:56

Merge branch 'upstream' into openai-vision-api

561ad49

Fix some type errors along the way

4479605

Improve async behaviour of loading images

20852d9

- Also, use the type definitions from `openai` directly

Use discriminated union in prompt parsing

ce770f4

Fix some type errors along the way

6b016bc

Some more fixes

7620354

Apply formatter

7c3e6d9

Merge branch 'upstream' into openai-vision-api

e74b0a7

Move openai to common requirements

9925dcb

Fix typo in _parse_chat_message_image_input

ceb4e35

Refactor prompt parsing so that it can be shared between Chat Complet…

7bdc84e

…ions API and legacy Completions API

Make code more readable

a7d1098

Move assertion to a more appropriate place

8b9d636

Merge branch 'openai-typing' into openai-vision-api

9754142

Add code documentation

c48c13a

Decompose _validate_prompt_and_tokenize

3530362

Fix missing import due to renaming

b8feec9

Merge branch 'openai-typing' into openai-vision-api

9cae113

DarkLight1337 changed the title ~~[Core][Frontend][Doc] Improved support for VLMs and add GPT-4V Chat Completions API~~ [Core][Frontend][Doc] Initial support for LLaVA-NeXT and add GPT-4V Chat Completions API Apr 18, 2024

Apply formatter and fix typo

21aaf3d

DarkLight1337 changed the title ~~[Core][Frontend][Doc] Initial support for LLaVA-NeXT and add GPT-4V Chat Completions API~~ [Core][Frontend][Doc] Initial support for LLaVA-NeXT and GPT-4V Chat Completions API Apr 18, 2024

DarkLight1337 added 3 commits April 18, 2024 12:54

Fix input shape not being based on config value

ac95b79

Allow config to specify other image size for LLaVA-NeXT

9a9a4e7

Improve error message to show the expected image_feature_size

176ad2c

DarkLight1337 force-pushed the openai-vision-api branch 5 times, most recently from f66a08f to 72eb712 Compare April 19, 2024 05:01

Fix dtype mismatch in multi_modal_kwargs

91ea044

DarkLight1337 force-pushed the openai-vision-api branch from 72eb712 to 175b819 Compare April 19, 2024 07:14

Fix LLaVA example and test w.r.t. image processing refactor

cb19743

- Note that we now load the images directly instead of from `.pt` files

DarkLight1337 force-pushed the openai-vision-api branch from 175b819 to cb19743 Compare April 19, 2024 07:49

ywang96 mentioned this pull request Apr 19, 2024

[RFC]: Multi-modality Support Refactoring #4194

Open

68 tasks

Merge branch 'upstream' into openai-vision-api

019f473

This was referenced Apr 19, 2024

[Core] Support image processor #4197

Merged

[Model] Initial support for LLaVA-NeXT #4199

Merged

DarkLight1337 mentioned this pull request Apr 19, 2024

[Frontend] Support GPT-4V Chat Completions API #4200

Closed

Fix circular import and set return type

f882d99

- These changes are propagated to the child PRs

DarkLight1337 referenced this pull request in DarkLight1337/vllm-rocm May 5, 2024

Add LLaVA-NeXT architecture

ea4f8ed

- Note that LLaVA-1.5 has been refactored to facilitate this

DarkLight1337 closed this Jun 10, 2024

DarkLight1337 deleted the openai-vision-api branch June 10, 2024 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core][Frontend][Doc] Initial support for LLaVA-NeXT and GPT-4V Chat Completions API #3978

[Core][Frontend][Doc] Initial support for LLaVA-NeXT and GPT-4V Chat Completions API #3978

DarkLight1337 commented Apr 10, 2024 •

edited

Loading

DarkLight1337 commented Apr 18, 2024 •

edited

Loading

DarkLight1337 commented Apr 19, 2024 •

edited

Loading

DarkLight1337 commented Apr 19, 2024 •

edited

Loading

ywang96 commented Apr 19, 2024 •

edited

Loading

DarkLight1337 commented Apr 19, 2024 •

edited

Loading

DarkLight1337 commented Jun 10, 2024

[Core][Frontend][Doc] Initial support for LLaVA-NeXT and GPT-4V Chat Completions API #3978

[Core][Frontend][Doc] Initial support for LLaVA-NeXT and GPT-4V Chat Completions API #3978

Conversation

DarkLight1337 commented Apr 10, 2024 • edited Loading

DarkLight1337 commented Apr 18, 2024 • edited Loading

DarkLight1337 commented Apr 19, 2024 • edited Loading

DarkLight1337 commented Apr 19, 2024 • edited Loading

ywang96 commented Apr 19, 2024 • edited Loading

DarkLight1337 commented Apr 19, 2024 • edited Loading

DarkLight1337 commented Jun 10, 2024

DarkLight1337 commented Apr 10, 2024 •

edited

Loading

DarkLight1337 commented Apr 18, 2024 •

edited

Loading

DarkLight1337 commented Apr 19, 2024 •

edited

Loading

DarkLight1337 commented Apr 19, 2024 •

edited

Loading

ywang96 commented Apr 19, 2024 •

edited

Loading

DarkLight1337 commented Apr 19, 2024 •

edited

Loading