Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[V1] Implement Cascade Attention #11635

Merged
merged 23 commits into from
Jan 1, 2025
Merged

[V1] Implement Cascade Attention #11635

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
bf06942
[V1] Implement Cascade Inference
WoosukKwon Dec 30, 2024
4faac41
Minor
WoosukKwon Dec 30, 2024
012775f
Merge branch 'main' into v1-cascade
WoosukKwon Dec 31, 2024
8093b2e
Minor
WoosukKwon Dec 31, 2024
21c988d
Merge branch 'main' into v1-cascade
WoosukKwon Jan 1, 2025
2dc2531
isort
WoosukKwon Jan 1, 2025
910752e
Comment
WoosukKwon Jan 1, 2025
c8b32de
Minor
WoosukKwon Jan 1, 2025
42efe0d
minor
WoosukKwon Jan 1, 2025
ca7b756
comment
WoosukKwon Jan 1, 2025
58af494
minor
WoosukKwon Jan 1, 2025
d6a7daf
comment
WoosukKwon Jan 1, 2025
d802be9
Minor
WoosukKwon Jan 1, 2025
afe8af7
docstring
WoosukKwon Jan 1, 2025
3766125
comment
WoosukKwon Jan 1, 2025
801b521
Fix
WoosukKwon Jan 1, 2025
1dfd2d4
Consider prefix only
WoosukKwon Jan 1, 2025
c47a449
comment
WoosukKwon Jan 1, 2025
34da6dd
Minor
WoosukKwon Jan 1, 2025
03a2809
Add debug
WoosukKwon Jan 1, 2025
bf94bfa
Fix
WoosukKwon Jan 1, 2025
350de8a
Add kernel test
WoosukKwon Jan 1, 2025
8b3291d
Add e2e test
WoosukKwon Jan 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
minor 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
  • Loading branch information
@WoosukKwon
WoosukKwon committed Jan 1, 2025
commit 42efe0d1deb5e1b59130504ad55ca9531dbd4af4
3 changes: 2 additions & 1 deletion vllm/v1/core/kv_cache_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,8 @@ def get_num_common_prefix_blocks(
request: Any request in the RUNNING state, used to identify the
common prefix blocks.
num_running_requests: The total number of requests in the RUNNING
state.
state. This can be different from the number of scheduled
requests in the current step.

Returns:
int: The number of common prefix blocks.
Expand Down
Loading