Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prefix Caching- fix t4 triton error #2517

Merged
merged 5 commits into from
Feb 16, 2024
Merged

Prefix Caching- fix t4 triton error #2517

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
15421a6
fix triton kernel block size for t4 gpu
caoshiyi Jan 20, 2024
2f25efa
format
caoshiyi Jan 20, 2024
30151a7
minor
caoshiyi Jan 20, 2024
47ea3ff
move cap check inside context_attention_fwd
caoshiyi Feb 15, 2024
1824cdb
clean
caoshiyi Feb 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion vllm/model_executor/layers/triton_kernel/prefix_prefill.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -618,7 +618,9 @@ def context_attention_fwd(q,
b_ctx_len,
max_input_len,
alibi_slopes=None):
BLOCK = 128

cap = torch.cuda.get_device_capability()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does prefix caching adapt other hardware? like AMD? This only considers cuda arch. Might it better that we define a global utility to get block size which handles different hardwares.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this kernel only works for NVIDIA right now. Let me merge this fix first and we can systematically test for AMD later.

BLOCK = 128 if cap[0] >= 8 else 64
# shape constraints
Lq, Lk, Lv = q.shape[-1], k.shape[-1], v.shape[-1]
assert Lq == Lk and Lk == Lv
Expand Down
Loading