Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[LLM] OSS LLM Serving #50643

Merged
merged 61 commits into from
Feb 19, 2025
Merged

[LLM] OSS LLM Serving #50643

merged 61 commits into from
Feb 19, 2025

Conversation

GeneDer
Copy link
Contributor

@GeneDer GeneDer commented Feb 16, 2025 •
edited
Loading

Why are these changes needed?

This PR adds all the components required to run llm serving application. Main components added are:

  • build_vllm_deployment
  • build_openai_app
  • VLLMDeployment
  • LLMModelRouterDeployment

Things to follow up:

  • Add more test coverages
  • Docs and Example
  • Use Serve's DeploymentConfig and AutoscalingConfig in LLMConfig
  • Add release tests

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@GeneDer
WIP
b42f419 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
WIP
f7df29e 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
WIP
b2e0b5c 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
WIP
a6065e0 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
WIP
b5740de 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
WIP
03c8404 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
WIP
85c8975 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
WIP: code complete, pending tests
ceb2534 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
lint
46951b4 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer GeneDer added the go add ONLY when ready to merge, run all tests label Feb 16, 2025
@GeneDer
lint
3a09df9 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer GeneDer changed the title OSS LLM Serving [LLM] OSS LLM Serving Feb 16, 2025
@GeneDer
add asyncache dependency
df87d16 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
try again
030a0bc 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
move error handling into configs
314c44e 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
fix typo and LLMModelRouterDeployment
6de69a6 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
refactor build_vllm_deployment
31a70c9 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
lint
130d094 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
fix imports in test
f63a808 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
drop generation config and use hf prompt template by default
f6c9541 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
fix test imports
a0d3dd7 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
test build_openai_app
b8db1f0 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add test for using serve run
b07b2c4 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add test for build_vllm_deployment
59b3490 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add test for vllm engine
0ae54ad 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add test for telemetry
cf84767 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add test for image retriver
7782178 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add streaming_error_handler test
481186f 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add tests for lora
428e223 
Signed-off-by: Gene Su <e870252314@gmail.com>
@kouroshHakha kouroshHakha mentioned this pull request Feb 18, 2025
Merged
8 tasks
@GeneDer
Merge branch 'master' into oss-rayllm
3d6f887
@GeneDer
fix all download_model_ckpt
bc60c2e 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
fix hf_prompt_format test
e097712 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
test using GPU
23bab42 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add botocore as requirement
b49adfe 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add boto3
8e4df55 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
add async_timeout
b6200bb 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer GeneDer marked this pull request as ready for review February 18, 2025 07:34
@GeneDer
Copy link
Contributor Author

GeneDer commented Feb 18, 2025

All test green! Especially this one https://buildkite.com/ray-project/premerge/builds/35010#019517cc-1117-4c01-ba25-77e015f856b6/186-935

aslonnie
aslonnie reviewed Feb 19, 2025
View reviewed changes
Copy link
Collaborator

@aslonnie aslonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the python/__init__.py file required?

@GeneDer
Copy link
Contributor Author

GeneDer commented Feb 19, 2025

why is the python/__init__.py file required?

oh good call, maybe that got created by my IDE some how. let me drop it

aslonnie
aslonnie reviewed Feb 19, 2025
View reviewed changes
Copy link
Collaborator

@aslonnie aslonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(and does not seem to have anything substantial that needs my review?)

@GeneDer
drop python/__init__.py
e2ac46c 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
Copy link
Contributor Author

GeneDer commented Feb 19, 2025

(and does not seem to have anything substantial that needs my review?)

correct, this doesn't contain any dependencies related changes after all

@kouroshHakha kouroshHakha enabled auto-merge (squash) February 19, 2025 01:44
@kouroshHakha kouroshHakha merged commit ee2ed35 into ray-project:master Feb 19, 2025
6 checks passed
@GeneDer GeneDer deleted the oss-rayllm branch February 19, 2025 04:39
400Ping pushed a commit to 400Ping/ray that referenced this pull request Feb 20, 2025
@GeneDer @400Ping
[ray.serve.llm] OSS LLM Serving (ray-project#50643)
4f3e6bd 
This PR adds all the components required to run llm serving application.
Main components added are:
- `build_vllm_deployment`
- `build_openai_app`
- `VLLMDeployment`
- `LLMModelRouterDeployment`

Signed-off-by: Gene Su <e870252314@gmail.com>
Signed-off-by: 400Ping <43886578+400Ping@users.noreply.github.com>
israbbani pushed a commit that referenced this pull request Feb 25, 2025
@GeneDer @israbbani
[ray.serve.llm] OSS LLM Serving (#50643)
01f3835 
This PR adds all the components required to run llm serving application.
Main components added are:
- `build_vllm_deployment`
- `build_openai_app`
- `VLLMDeployment`
- `LLMModelRouterDeployment`


Signed-off-by: Gene Su <e870252314@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@comaniac comaniac comaniac left review comments

@aslonnie aslonnie aslonnie left review comments

@kouroshHakha kouroshHakha kouroshHakha approved these changes

Assignees

@kouroshHakha kouroshHakha

Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@GeneDer @comaniac @kouroshHakha @aslonnie