Skip to content

CUDA: topk-moe: add optional parameter for gpt-oss #16649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 21, 2025
Merged

CUDA: topk-moe: add optional parameter for gpt-oss #16649

merged 3 commits into from
Oct 21, 2025
+153 −62

Conversation

@am17an
Copy link
Collaborator

@am17an am17an commented Oct 18, 2025 •
edited
Loading

While looking at this kernel I realized that it is relatively easy to add it for gpt-oss, which does the softmax after the top-k.

Performance on a 4090:

Model Test t/s master t/s cuda_gpt_oss_opt Speedup
gpt-oss 20B MXFP4 MoE tg32 170.99 177.68 1.04
gpt-oss 20B MXFP4 MoE tg64 168.75 175.36 1.04
gpt-oss 20B MXFP4 MoE tg128 167.01 173.33 1.04

@am17an am17an requested a review from slaren as a code owner October 18, 2025 11:24
@github-actions github-actions bot added testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Oct 18, 2025
jeffbolznv added a commit to jeffbolznv/llama.cpp that referenced this pull request Oct 18, 2025
@jeffbolznv
vulkan: Update topk_moe fusion to handle gpt's late softmax
0111a34 
Based on ggml-org#16649.
@jeffbolznv jeffbolznv mentioned this pull request Oct 18, 2025
Merged
jeffbolznv added a commit to jeffbolznv/llama.cpp that referenced this pull request Oct 21, 2025
@jeffbolznv
vulkan: Update topk_moe fusion to handle gpt's late softmax
34d4122 
Based on ggml-org#16649.
@am17an am17an force-pushed the cuda_topk_moe_gpt_oss branch from 49a541e to 17c3927 Compare October 21, 2025 11:53
jeffbolznv added a commit to jeffbolznv/llama.cpp that referenced this pull request Oct 21, 2025
@jeffbolznv
vulkan: Update topk_moe fusion to handle gpt's late softmax
b2d689a 
Based on ggml-org#16649.
@am17an am17an merged commit 03792ad into ggml-org:master Oct 21, 2025
70 checks passed
@am17an am17an deleted the cuda_topk_moe_gpt_oss branch October 21, 2025 15:21
ye-NX pushed a commit to ye-NX/llama.cpp that referenced this pull request Oct 21, 2025
@am17an @ye-NX
CUDA: topk-moe: add optional parameter for gpt-oss (ggml-org#16649)
7869ac7
@am17an am17an mentioned this pull request Oct 22, 2025
Merged
@avidwriter
Copy link

how to use this?

@am17an
Copy link
Collaborator Author

am17an commented Oct 22, 2025

@avidwriter if you are using the CUDA backend, with the latest master it should already be included

Nexesenex added a commit to Nexesenex/croco.cpp that referenced this pull request Oct 22, 2025
@Nexesenex
Revert "CUDA: topk-moe: add optional parameter for gpt-oss (ggml-org#…
46a1ef1 
…16649)"
FMayran pushed a commit to FMayran/llama.cpp that referenced this pull request Oct 23, 2025
@am17an
CUDA: topk-moe: add optional parameter for gpt-oss (ggml-org#16649)
54be1a7
pwilkin pushed a commit to pwilkin/llama.cpp that referenced this pull request Oct 23, 2025
@am17an @pwilkin
CUDA: topk-moe: add optional parameter for gpt-oss (ggml-org#16649)
8c01a63
jeffbolznv added a commit to jeffbolznv/llama.cpp that referenced this pull request Oct 26, 2025
@jeffbolznv
vulkan: Update topk_moe fusion to handle gpt's late softmax
6cccaef 
Based on ggml-org#16649.
@am17an am17an mentioned this pull request Oct 28, 2025
Closed
0cc4m pushed a commit that referenced this pull request Oct 29, 2025
@jeffbolznv @slaren
vulkan: Update topk_moe fusion to handle gpt's late softmax (#16656)
10fcc41 
* vulkan: Update topk_moe fusion to handle gpt's late softmax

Based on #16649.

* Add ggml_check_edges

* Add sync logging to show fusion effects

* handle clamp added in #16655

* Update ggml/src/ggml-impl.h

Co-authored-by: Diego Devesa <slarengh@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JohannesGaessler JohannesGaessler JohannesGaessler approved these changes

@slaren slaren Awaiting requested review from slaren slaren is a code owner

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning Nvidia GPU Issues specific to Nvidia GPUs testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@am17an @avidwriter @JohannesGaessler