Skip to content

[Core] [Frontend] Priority scheduling for embeddings and in the OpenAI-API #8965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Oct 1, 2024
Merged

[Core] [Frontend] Priority scheduling for embeddings and in the OpenAI-API #8965

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a1d1b8b
Make policy settable from EngineArgs
schoennenbeck Sep 30, 2024
cff3fde
Improved documentation
schoennenbeck Sep 30, 2024
1781cb7
Improved documentation
schoennenbeck Sep 30, 2024
5d44fd8
typo
schoennenbeck Sep 30, 2024
0bb54f6
Support priority in openai-server
schoennenbeck Sep 30, 2024
bc92a92
Merge branch 'main' into feature/priority-in-openai-api
schoennenbeck Sep 30, 2024
73c4751
Priority in MQLLMEngineClient
schoennenbeck Sep 30, 2024
fac7111
Add priority for embedding endpoints
schoennenbeck Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Improved documentation
  • Loading branch information
@schoennenbeck
schoennenbeck committed Sep 30, 2024
commit cff3fdeaf6ea77d8c4b5c00dc7bca3d7aa28d0fb
7 changes: 5 additions & 2 deletions vllm/engine/arg_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -802,8 +802,11 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
'--scheduling-policy',
choices=['fcfs', 'priority'],
default="fcfs",
help='The scheduling policy to use. "fcfs" (default) or "priority"'
)
help='The scheduling policy to use. "fcfs" (first come first serve'
', i.e. requests are handled in order of arrival, this is the '
'default) or "priority" (requests are handled based on given '
'priority (lower value means earlier handling) and time of '
'arrival deciding any ties).')

return parser

Expand Down
Loading