Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update default max_num_batch_tokens for chunked prefill to 2048 #10544

Merged
merged 2 commits into from
Nov 23, 2024
Merged

Update default max_num_batch_tokens for chunked prefill to 2048 #10544

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5f19b68
Update default max_num_batch_tokens for chunked prefill to 2048
mgoin Nov 21, 2024
e7483c5
Merge branch 'main' into default-chunked-prefill-2048
mgoin Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Merge branch 'main' into default-chunked-prefill-2048
  • Loading branch information
@mgoin
mgoin authored Nov 22, 2024
commit e7483c5ce562ae8effe9350b1a3185668f294804
2 changes: 1 addition & 1 deletion vllm/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1124,7 +1124,7 @@ def __post_init__(self) -> None:
else:
# This value is chosen to have a balance between ITL
# and TTFT. Note it is not optimized for throughput.
max_num_batched_tokens = 2048
self.max_num_batched_tokens = 2048
else:
# If max_model_len is too short, use 2048 as the default value
# for higher throughput.
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.