perf(minimax): scatter-grouped i8-WMMA + WMMA-dense prefill (gfx1151) - #479
Closed
fivetide wants to merge 9 commits into
Closed
perf(minimax): scatter-grouped i8-WMMA + WMMA-dense prefill (gfx1151)#479fivetide wants to merge 9 commits into
fivetide wants to merge 9 commits into
Conversation
fivetide
requested review from
Kaden-Schutt,
nwoolmer and
unverbraucht
and removed request for
nwoolmer
June 30, 2026 04:55
Cherry-pick of the 3 additive kernel commits from upstream PR warpfront#476 (d1f6115, 80920ce, 586ab0c) — NOT a merge, since warpfront#476 branched off pre-minimax-perf master and would delete this branch's spec.rs + the mq3-lloyd batched-verify kernel. Brings in: - gemm_mq2g256_lloyd_moe_grouped_mmq.gfx1151.hip — i8-WMMA grouped GEMM for MQ2-Lloyd MoE experts (+ widened weight-index loads) - gemm_q8_0_mmq_4w.gfx1151.hip — i8-WMMA 64x64 dense Q8_0 GEMM - rdna-compute gemm.rs/kernels.rs wrappers - deepseek4 opt-in wiring (HIPFIRE_DEEPSEEK4_MOE_I8 / _Q8_I8, default-off) Foundation for porting grouped-GEMM MoE to minimax prefill (the 93% bottleneck per rocprofv3). No active minimax/qwen path changed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tok/s) forward_batch_hidden ran q/k/v/o/router through gemm_q8_0_batched (scalar 1-wave-per-row). Switch to gemm_q8_0_batched_chunked, which on gfx1151 (and all wave32-WMMA archs) routes to gemm_q8_0_wmma — documented as ~65% of A3B prefill GPU time when left scalar. Also lifts the B>64 forward_batch cap (attention/MoE kernels take B as a grid dim; chunked GEMM tiles the projections internally) so larger prefill chunks are now valid. Bench (MiniMax-M2.7.mq2, gfx1151, 2048 tok, chunk 64, fresh process, median of 3): 34.4 → 70.5 tok/s = 2.05x. Chunk size is neutral (73→69 across 64→1024) — the win is the kernel, not amortization. Correctness: forward_batch vs sequential decode_step cosine 0.9999 (B=2/4/8); chunk-64 vs chunk-512 cosine 1.000000; real-prompt greedy generation fluent (Rayleigh-scattering answer, no attractor). rocprofv3 now shows MoE indexed-GEMV = 93% of prefill (gate_up 55.6% + down 37.4%); dense projections dropped to 4.5%. Grouped-GEMM MoE is the next lever. Adds minimax_prefill_bench example (chunk sweep + --verify + MINIMAX_GEN coherence mode) mirroring the daemon prefill path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The indexed-batched MoE GEMV re-reads each expert weight ONCE PER ROUTED TOKEN — at 256 experts/top-8 it was 93% of prefill (rocprofv3). Add a scatter-grouped path in forward_batch_hidden that reads each expert weight ONCE PER CHUNK via WMMA grouped GEMM: scatter (moe_scatter_fused_k8) → grouped gate_up (mq2-lloyd WMMA) → unscatter → AWQ silu·mul·rotate → grouped down (mq3/mq2-lloyd WMMA) → weighted combine into x. Mirrors the deepseek4 SGLang-style pipeline; reuses the existing FP16 grouped kernels. Gated on chunk size (HIPFIRE_MINIMAX_MOE_GROUPED_GATE, default 256): below it the 256 experts get too few rows/expert to beat the indexed path, so small chunks (and spec-verify B<=64) stay on the indexed GEMV. Daemon prefill now chunks at 512 (HIPFIRE_MINIMAX_PREFILL_CHUNK) to activate it. Bench (MiniMax-M2.7.mq2, gfx1151, 2048 tok, fresh process, median): chunk-64 indexed 74.5 tok/s chunk-512 grouped 166.4 tok/s = 2.23x Combined with the WMMA dense-projection win this session: 34 -> 166 = 4.9x. Correctness: grouped vs indexed final-logits cosine 0.9982, argmax match (FP16 WMMA vs higher-precision indexed accumulate — the same FP16 grouped path deepseek4 ships); real-prompt greedy generation fluent and accurate (Rayleigh-scattering answer, no attractor). FP16 grouped only; i8 grouped (mq2 PR kernel + mq3 analog) is the next step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the i8 MMQ path to the scatter-grouped MoE prefill (gfx1151): - gate_up (MQ2-Lloyd) routes to the PR warpfront#476 i8 kernel gemm_mq2g256_lloyd_moe_grouped_mmq_gfx1151. - down (MQ3-Lloyd) has no i8 kernel upstream, so this adds gemm_mq3g256_lloyd_moe_grouped_mmq.gfx1151.hip — the MQ3 twin of the MQ2 i8 kernel: 8-entry Lloyd codebook quantized to int8 + group scale once/group, 3-bit indices decoded via a two-u32 byte-extract LUT, i8 WMMA (wmma_i32_16x16x16_iu8) at ~2x FP16. Plus the gemm.rs wrapper + kernels.rs SRC const. Opt-in via HIPFIRE_MINIMAX_MOE_I8=1 (gfx1151 only; FP16 grouped stays the default). Kept opt-in because i8 lowers grouped-vs-indexed cosine 0.9982 → 0.9948; default-on awaits a multi-prompt coherence gate. Bench (MiniMax-M2.7.mq2, gfx1151, 2048 tok, chunk 512, median): FP16 grouped 168.5 → i8 grouped 184.9 tok/s (best 190) = +9.7% Correctness: argmax matches indexed; full-i8 greedy generation fluent and accurate (Rayleigh-scattering answer, no attractor). Session prefill total: 34 → 185 tok/s = 5.4x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ry; i8 default-on Adds minimax_coherence_battery (hipfire-runtime example): 6 diverse prompts × 3 prefill modes (indexed / grouped-FP16 / grouped-i8), 256-tok greedy decode each, fed through the hipfire-detect bank. A cell is DEGENERATE only on a true single-token attractor (max_freq>0.50) or a garbage/structural detector — the spec-decode unique-ratio sub-check is excluded since it fires on legitimate greedy repetition (the indexed baseline trips it too). The battery FOUND A REAL BUG: the grouped path sized m_total_max as `b*k_top + n_exp*BLOCK_M`, which is only 16-aligned when b is even (k_top=8). An odd-length last prefill chunk (e.g. a 553-tok prompt → 512 + 41) left the grouped kernel grid (ceil(m_total/16) tiles) one tile longer than the expert_tile_ids buffer (m_total_max/16, integer-div) → OOB read → GPU page fault. Standalone tests only used even b=512 so never hit it; a production prompt with an odd final chunk would. Fixed by rounding m_total_max up to a whole BLOCK_M. With the fix the battery is clean: 18/18 cells coherent, no attractors (max_freq ≤ 0.28 everywhere, < 0.50 threshold), all three modes track the indexed baseline. On that result, i8 grouped is flipped DEFAULT-ON (gfx1151, opt out HIPFIRE_MINIMAX_MOE_I8=0): cosine vs indexed 0.9948, +10% over FP16 grouped. Warm default-path prefill measured up to ~261 tok/s (DPM-dependent; conservative back-to-back number is 166 FP16 → 185 i8). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… dispatcher The arch→kernel-variant choice (i8 MMQ on gfx1151 vs FP16 WMMA elsewhere) belongs in the dispatch layer, not the forward pass — mirroring how gemm_q8_0_batched_chunked already resolves scalar-vs-WMMA internally. Add arch-resolving wrappers gemm_mq2g256_lloyd_moe_grouped / gemm_mq3g256_lloyd_moe_grouped in rdna-compute; forward_batch_hidden now calls them arch-agnostically (drops the moe_i8 flag, the HIPFIRE_MINIMAX_MOE_I8 env knob, and the inline if/else). i8 is gated purely on ArchCaps::is_gfx1151() (the i8 kernels are .gfx1151.hip builtins). Behavior-preserving. Battery example modes simplified to indexed vs grouped (i8/FP16 is no longer env-toggleable — it's arch-determined). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
scripts/coherence-gate-minimax.sh drives the real serving path: sends a 6-prompt matrix as chat `messages` so the daemon renders the MiniMax chat template (model answers + emits EOS, no greedy rambling), in two MoE-prefill modes — grouped (production; i8 on gfx1151) and indexed baseline. Per-request attractor detection over the daemon JSONL (zero-tokens / max_freq>0.50 / unique<0.30 / panic = hard fail). Result: GATE PASS. Chat-templated outputs are healthy (unique 0.46–0.82, max_freq 0.04–0.09) — far cleaner than the raw-prompt example battery where greedy repetition is expected. grouped-i8 tracks the indexed baseline on every prompt, confirming the scatter-grouped i8 prefill is coherence- indistinguishable from indexed through the daemon. Complements the GPU-direct minimax_coherence_battery example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Coherence is confirmed (coherence-gate-minimax.sh PASS through the chat template), so the prefill optimizations need no opt-in/opt-out knobs. Remove: - HIPFIRE_MINIMAX_MOE_GROUPED / _GATE (forward.rs) → grouped MoE is on whenever b >= MOE_GROUPED_GATE (const 256) on a WMMA arch with MQ2-Lloyd gate_up. - HIPFIRE_MINIMAX_PREFILL_CHUNK (daemon.rs) → fixed 512. (HIPFIRE_MINIMAX_MOE_I8 was already removed; i8 is arch-gated in the dispatcher.) Simplify coherence-gate-minimax.sh to a single env-free production run and add a >=256-token passage so the grouped-i8 path is exercised through the daemon (short prompts → indexed, long → grouped). Gate re-run: PASS (7/7 coherent). Remove the now-redundant minimax_coherence_battery example (its indexed-vs- grouped A/B required the removed env knobs; minimax_prefill_bench MINIMAX_GEN and the daemon gate cover GPU-direct and serving-path coherence). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fill prereq) MiniMax-M2.7.mq2 is mixed k-map (gate_up=MQ2-Lloyd / down=MQ3-Lloyd). master's batched forward_batch handled MQ2/MQ4/MQ6 down but not MQ3, so forward_batch_supported returned false for this model (daemon fell back to per-token decode_step prefill) and a direct forward_batch at b<256 errored "down dtype MQ3G256Lloyd has no batched kernel yet". Add the MQ3 indexed-batched down path so the batched-prefill optimizations in this branch actually engage for the mq3-down model: - gemv_mq3g256_lloyd_moe_down_indexed_batched_k4.hip + rdna-compute wrapper + kernels.rs SRC const, - the MQ3G256Lloyd arm in forward_batch's indexed down match, and - MQ3G256Lloyd in forward_batch_supported's down tier. (Extracted from the minimax batched-verify work; the grouped MoE path's mq3 i8/FP16 kernels land in the prefill commits.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fivetide
force-pushed
the
minimax-prefill-perf
branch
from
July 7, 2026 08:20
3e3244e to
831ffe5
Compare
Kaden-Schutt
added a commit
that referenced
this pull request
Jul 11, 2026
…h too The dry-run resolution preview showed the "stack-clobber" #479 rebases cleanly onto master. Refine §11: staging is agent-owned, so the agent resolves ON STAGING — rebases any PR (fork or not) there (mechanical conflicts auto-resolve, semantic ones escalate to codex merge-fix), never needing to push to the PR branch. Fork limits only the Gate-4 in-PR fix, not stacking. Codex fork-resolutions logged for contributor review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Kaden-Schutt
added a commit
that referenced
this pull request
Jul 17, 2026
Collaborator
|
Folded into #534 as a cleanly rebased/cherry-picked delta on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MiniMax-M2 prefill optimizations on gfx1151 (Strix Halo, RDNA3.5), validated
on
MiniMax-M2.7.mq2(256 experts / top-8, mixed k-map gate_up=MQ2-Lloyd /down=MQ3-Lloyd). Two compounding wins on the batched prefill path:
gemm_q8_0_batched(1-wave-per-row) togemm_q8_0_batched_chunked→gemm_q8_0_wmma. (~2× the projection cost; rocprofv3 then showed MoE = 93%of prefill.)
weight once per routed token; the new path scatters tokens by expert and
runs a WMMA grouped GEMM (each weight read once per chunk), with an
i8-WMMA variant on gfx1151 (2-bit/3-bit Lloyd → int8 codebook LUT, i8
WMMA at ~2× FP16). Kernel selection (i8 vs FP16) is resolved in the
rdna-compute dispatcher by arch; the forward pass is arch-agnostic.
Also enables the MQ3-Lloyd indexed-batched down path so the mixed-k-map
MiniMax-M2.7.mq2 model can use batched prefill at all (it previously fell back
to per-token
decode_stepprefill).The grouped i8 kernels are
.gfx1151.hipbuiltins and arch-gated; other WMMAarches take the FP16 grouped path, non-WMMA archs the indexed path. No env
flags — all unconditional.
Prefill metrics (MiniMax-M2.7.mq2, gfx1151, 2048-token prompt, fresh process, median)
decode_stepprefill (mq3-down had no batched support)≈ 3.4× over indexed-batched within the same run; ≈ 9× over master's actual
path for this model (which couldn't batch mq3-down prefill).
Decode metrics (before/after)
Unchanged — this is a prefill-only PR. It does not touch
decode_step; ARdecode ≈ 25 tok/s before and after.
Correctness / validation
scripts/coherence-gate-minimax.sh(new) drives the real serving path:prompts as chat
messages→ daemon renders the MiniMax chat template → modelanswers with natural EOS. Runs short (indexed) + a ≥256-token passage
(grouped-i8). GATE: PASS (7/7 coherent, max single-token freq ≤ 0.09,
no attractor/leak/loop, 0 GPU faults).
0.995 (i8 grouped) — the i8 path is numerically close and coherence-clean
(the same FP16/i8 grouped machinery deepseek4 ships).
m_total_maxwas not16-aligned for odd batch (
k_top=8), so an odd-length last prefill chunk leftthe kernel grid one tile longer than the
expert_tile_idsbuffer → GPU pagefault. Fixed via
next_multiple_of(BLOCK_M).Notes
attributed); the MQ3-Lloyd i8 grouped kernel is new here.
grouped / indexed fallbacks).
🤖 Generated with Claude Code
Rebased onto current master (2026-07-07)
Was 97 commits behind master; rebased clean onto
master(no conflicts). Re-validated on gfx1151 (MiniMax-M2.7.mq2):cargo build --release --workspace --locked— green (exit 0).scripts/coherence-gate-minimax.sh— PASS (production indexed-short + grouped-i8 long path, no attractors, 0 GPU faults).Merge note — deepseek4 kernel-selection skew vs #476
This branch carries a cherry-pick of #476's deepseek4 i8 path. Since then, #476 evolved: deepseek4 Q8_0 kernel selection moved into the rdna-compute dispatcher (
gemm_q8_0_wmma_prefill_auto) and the i8-MoE gate became default-on (dropping theHIPFIRE_DEEPSEEK4_MOE_I8=1opt-in). Whichever of #476/#479 merges second should take the newer (dispatcher-based, default-on) deepseek4 side and keep this PR's genuinely-newgemm_mq3g256_lloyd_moe_grouped_mmq_gfx1151minimax kernel. (Validated in a local#480→#476→#479integration stack — builds green, minimax coherence PASS.)