Skip to content

[Attention] Support using BF16 sparse prefill kernel for FP8 kv-cache with DeepseekV32 #26152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 33 commits into
base: main
Choose a base branch
from
Draft

[Attention] Support using BF16 sparse prefill kernel for FP8 kv-cache with DeepseekV32 #26152

wants to merge 33 commits into from
+1,216 −195

Conversation

@LucasWilkinson
Copy link
Collaborator

@LucasWilkinson LucasWilkinson commented Oct 3, 2025 •
edited by github-actions bot
Loading

Support using BF16 sparse prefill kernel for FP8 kv-cache with DeepseekV32 by first up converting the cache

FIX: #27043

@mergify mergify bot added deepseek Related to DeepSeek models v1 labels Oct 3, 2025
@mergify
Copy link

mergify bot commented Oct 8, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/split-prefills-and-decode branch from 9572596 to dcfd9bd Compare October 9, 2025 03:46
@mergify mergify bot added the ci/build label Oct 9, 2025
@LucasWilkinson
Split prefills and decode
319c660 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/split-prefills-and-decode branch from dcfd9bd to 319c660 Compare October 9, 2025 03:53
@heheda12345 heheda12345 mentioned this pull request Oct 9, 2025
Open
@LucasWilkinson
workspace refactor
e28bda0 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
wip
36168f4 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
4a22155 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
clean-up
4399ec8 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@mergify
Copy link

mergify bot commented Oct 11, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Oct 11, 2025
@LucasWilkinson
booting
2eceb39 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
Merge remote-tracking branch 'origin/main' into lwilkinson/split-pref…
bb7ba88 
…ills-and-decode

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@mergify mergify bot removed the needs-rebase label Oct 11, 2025
@LucasWilkinson
cleanup
337698c 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@mergify
Copy link

mergify bot commented Oct 13, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Oct 13, 2025
@LucasWilkinson
Merge remote-tracking branch 'origin/main' into lwilkinson/split-pref…
cd5edd1 
…ills-and-decode

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@mergify mergify bot removed the needs-rebase label Oct 13, 2025
@LucasWilkinson
cleanup
5330f95 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
376eeb9 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
0fba98b 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
3bda203 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
d4d4522 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
05b957b 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
fix
a4e28d5 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
Merge remote-tracking branch 'origin/main' into lwilkinson/split-pref…
9e4c72c 
…ills-and-decode
@LucasWilkinson
wip
e4957f3 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@mergify
Copy link

mergify bot commented Oct 16, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Oct 16, 2025
@LucasWilkinson LucasWilkinson mentioned this pull request Oct 16, 2025
Closed
1 task
@LucasWilkinson
Merge remote-tracking branch 'origin/main' into lwilkinson/split-pref…
2105618 
…ills-and-decode

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@mergify mergify bot removed the needs-rebase label Oct 16, 2025
@LucasWilkinson
cleanup
aee40b9 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
fix
5904207 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
fix server boot
5bf86a9 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
fix precommit
6a6cee8 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
cb922d1 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
add back kernels
0fbee26 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
add back missing code
2ccdf18 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
b44cf4c 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
rename
cc27394 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson mentioned this pull request Oct 17, 2025
Merged
heheda12345
heheda12345 reviewed Oct 20, 2025
View reviewed changes
@mergify
Copy link

mergify bot commented Oct 20, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Oct 20, 2025
LucasWilkinson and others added 2 commits October 20, 2025 07:56
@LucasWilkinson @heheda12345
Update vllm/v1/attention/backends/mla/flashmla_sparse.py
cbcc9ad 
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
@LucasWilkinson
Merge branch 'lwilkinson/split-prefills-and-decode' of https://github…
ac4b00d 
….com/neuralmagic/vllm into lwilkinson/split-prefills-and-decode
@LucasWilkinson LucasWilkinson mentioned this pull request Oct 21, 2025
Closed
5 tasks
@LucasWilkinson
resvere MoE workspaces
f1fe1fd 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cleanup
ed47577 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@heheda12345 heheda12345 heheda12345 left review comments

@mgoin mgoin Awaiting requested review from mgoin mgoin will be requested when the pull request is marked ready for review mgoin is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth will be requested when the pull request is marked ready for review tlrmchlsmth is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon will be requested when the pull request is marked ready for review WoosukKwon is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 will be requested when the pull request is marked ready for review yewentao256 is a code owner

@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety will be requested when the pull request is marked ready for review pavanimajety is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ci/build deepseek Related to DeepSeek models needs-rebase v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: FlashMLA: invalid configuration argument

2 participants

@LucasWilkinson @heheda12345