Skip to content

Add thin M GEMM dispatch - #4654

Merged
zcbenz merged 1 commit into
ml-explore:mainfrom
Isalia20:thin-m-gemm-opt
Oct 9, 2026
Merged

zcbenz merged 1 commit into
ml-explore:mainfrom
Isalia20:thin-m-gemm-opt

Conversation

@Isalia20

@Isalia20 Isalia20 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Followup to #4640. Adds thin M GEMM dispatche logic and the tests which I left out in the previous PR accidentally. Speed on bf16 M5 Pro:

Model, layer M x N x K main branch speedup
Qwen3.8-27B mlp.gate_proj 32 x 17408 x 5120 984.3 694.1 1.42x
Qwen3.8-27B mlp.down_proj 32 x 5120 x 17408 1039.4 735.4 1.41x
Llama-3.1-8B mlp.gate_proj 64 x 14336 x 4096 717.7 461.3 1.56x
Llama-3.1-8B mlp.down_proj 64 x 4096 x 14336 721.7 543.2 1.33x
Qwen3-8B mlp.gate_proj 16 x 12288 x 4096 622.4 395.0 1.58x
gemma-4-31B mlp.gate_proj 48 x 21504 x 5376 1314.4 917.6 1.43x
Qwen-Image text encoder mlp.gate_proj 40 x 18944 x 3584 792.0 541.2 1.46x
Qwen-Image text encoder mlp.down_proj 40 x 3584 x 18944 832.7 614.3 1.36x
  • AI usage disclosure:
    N/A

@Isalia20 Isalia20 changed the title add thin m dispatch Add thin M GEMM dispatch Oct 8, 2026
@zcbenz
zcbenz merged commit a87691d into ml-explore:main Oct 9, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants