Skip to content

Add cutlass support for blackwell fp8 blockwise gemm #14383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
May 8, 2025
Merged

Add cutlass support for blackwell fp8 blockwise gemm #14383

merged 7 commits into from
May 8, 2025

Conversation

wenscarl
Copy link
Contributor

@wenscarl wenscarl commented Mar 6, 2025 •
edited
Loading

This PR adds support for cutlass blackwell blockwise gemm for fp8.

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 6, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added the ci/build label Mar 6, 2025
@pathorn
Copy link
Contributor

pathorn commented Mar 6, 2025

It looks like scaled_mm_blockwise_sm100_fp8.cu and scaled_mm_blockwise_sm100_fp8_dispatch.cuh were committed only as symlinks. When the change is done and ready for review, make sure to double check the actual files are added.

I'm curious to compare it to the blockwise kernels I wrote in the blackwell-rebase-feb20 branch of deepinfra/vllm , since we had to change the scale factor to be fp8

@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from 2f5591d to 546b495 Compare March 10, 2025 20:00
@wenscarl wenscarl changed the title [wip] scaled_mm_bw_sm100 Scaled_mm_blockwise_sm100 support by cutlass Mar 10, 2025
@wenscarl wenscarl marked this pull request as ready for review March 10, 2025 20:03
@wenscarl wenscarl requested a review from tlrmchlsmth as a code owner March 10, 2025 20:03
@wenscarl wenscarl changed the title Scaled_mm_blockwise_sm100 support by cutlass Add cutlass support for blackwell fp8 blockwise gemm Mar 10, 2025
@kushanam
Copy link
Contributor

Thank you @wenscarl! Please correct thecutlass_scaled_mm_supports_block_fp8() function in scaled_mm_entry.cu to allow >100 arch as well.

@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from 3a34259 to 07e8083 Compare March 21, 2025 03:25
@WoosukKwon
Copy link
Collaborator

@tylertitsworth Can you please take a look?

tlrmchlsmth
tlrmchlsmth reviewed Mar 26, 2025
View reviewed changes
Copy link
Collaborator

@tlrmchlsmth tlrmchlsmth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! A couple of comments but looks good overall

@mergify Mergify
Copy link

mergify bot commented Mar 28, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wenscarl.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Mar 28, 2025
@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from 287cfb3 to ca3a3e2 Compare March 28, 2025 18:47
@wenscarl wenscarl requested a review from tlrmchlsmth March 28, 2025 18:48
@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from ca3a3e2 to 87d109d Compare March 28, 2025 21:55
@mergify mergify bot removed the needs-rebase label Mar 28, 2025
@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch 2 times, most recently from a06d3da to 3dfa546 Compare March 30, 2025 16:20
tlrmchlsmth
tlrmchlsmth reviewed Apr 4, 2025
View reviewed changes
@tlrmchlsmth
Copy link
Collaborator

Looks good to me now, thank you!

Please merge in latest main to fix the docker-build-image failure

thakkarV
thakkarV reviewed Apr 4, 2025
View reviewed changes
@mergify Mergify
Copy link

mergify bot commented Apr 23, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wenscarl.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Apr 23, 2025
@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch 2 times, most recently from 703ec2c to 1d17dd1 Compare April 25, 2025 17:19
@mergify mergify bot removed the needs-rebase label Apr 25, 2025
@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from 1d17dd1 to 92d6da8 Compare April 25, 2025 17:23
@wenscarl wenscarl requested a review from thakkarV April 25, 2025 18:54
@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from 92d6da8 to 86d58fd Compare April 25, 2025 19:05
LucasWilkinson
LucasWilkinson reviewed Apr 25, 2025
View reviewed changes
@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from 7ca702c to f207fec Compare May 5, 2025 02:29
@wenscarl wenscarl requested a review from LucasWilkinson May 5, 2025 02:43
LucasWilkinson
LucasWilkinson approved these changes May 5, 2025
View reviewed changes
Copy link
Collaborator

@LucasWilkinson LucasWilkinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now, thanks!

@LucasWilkinson LucasWilkinson enabled auto-merge (squash) May 5, 2025 13:56
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label May 5, 2025
auto-merge was automatically disabled May 6, 2025 18:42

Head branch was pushed to by a user without write access

@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from f207fec to cc21ba5 Compare May 6, 2025 18:42
wenscarl added 7 commits May 6, 2025 19:16
@wenscarl
Support cutlass w8a8 blockwise matmul for blackwell
d001b23 
Signed-off-by: Shu Wang <shuw@nvidia.com>
@wenscarl
Update CUDA_VERSION requirement.
b45e6bb 
Signed-off-by: Shu Wang <shuw@nvidia.com>
@wenscarl
Improve after review 1
5100a33 
Signed-off-by: Shu Wang <shuw@nvidia.com>
@wenscarl
Correct forward compatibility.
24a528e 
Signed-off-by: Shu Wang <shuw@nvidia.com>
@wenscarl
Branching with 1/2SMs
c030b94 
Signed-off-by: Shu Wang <shuw@nvidia.com>
@wenscarl
Add comments for multiple of 4 constraint
9bab9f6 
Signed-off-by: Shu Wang <shuw@nvidia.com>
@wenscarl
Fix precommit.
b6783db 
Signed-off-by: Shu Wang <shuw@nvidia.com>
@wenscarl wenscarl force-pushed the scale_mm_bw_sm100 branch from cc21ba5 to b6783db Compare May 6, 2025 19:18
@simon-mo simon-mo merged commit 376786f into vllm-project:main May 8, 2025
77 of 80 checks passed
princepride pushed a commit to princepride/vllm that referenced this pull request May 10, 2025
@wenscarl @princepride
Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383
765df60 
)

Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@wenscarl
Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383
5725ea8 
)

Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
mawong-amd pushed a commit to ROCm/vllm that referenced this pull request May 14, 2025
@wenscarl @mawong-amd
Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383
ce8f7fb 
)

Signed-off-by: Shu Wang <shuw@nvidia.com>
zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025
@wenscarl
Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383
e459097 
)

Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
minpeter pushed a commit to minpeter/vllm that referenced this pull request Jun 24, 2025
@wenscarl @minpeter
Add cutlass support for blackwell fp8 blockwise gemm (vllm-project#14383
255194b 
)

Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: minpeter <kali2005611@gmail.com>
@tanujtiwari1998 tanujtiwari1998 mentioned this pull request Jul 8, 2025
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

@LucasWilkinson LucasWilkinson LucasWilkinson approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@thakkarV thakkarV Awaiting requested review from thakkarV

Assignees

@tlrmchlsmth tlrmchlsmth

Labels
ci/build ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@wenscarl @pathorn @kushanam @WoosukKwon @tlrmchlsmth @LucasWilkinson @thakkarV @simon-mo