Title: Measured note on routed/block attention at the kernel level: folding the routed compute is byte-exact and ~6.6×, but the routing glue is where the GPU idles
Not a bug report, and the weakest-fit of a few writeups I'm posting — MoBA is a method repo and this is kernel-level measurement, so close it freely if it's noise for you. Posting because we measured something specific about routed compute that I have not seen written down, and our project is shelved so it will otherwise be lost.
Context: we ran a mixture-of-experts conv/attention block (CLMConvMoE) on our own compiler stack — own PTX backend, own GEMM, no cuBLAS call — and spent a campaign trying to raise GPU utilization on it.
Finding 1 — folding E routed branches into one kernel is byte-exact, not approximate
Folding E separate expert branches into a single fused kernel, and separately into a GEMM-with-weight-reuse path, was bit-identical to the naive E × ModuleList reference:
E=30 T=16 d=8 K=3 dil=1 max|Δ| fused-vs-ModuleList = 0.0
E=30 T=16 d=8 K=3 dil=2 max|Δ| fused-vs-ModuleList = 0.0
E=4 T=24 d=8 K=3 dil=1 max|Δ| fused-vs-ModuleList = 0.0
max|Δ| = 0.0 is an equality, not a tolerance. The GEMM-conv path measured 6.57× @ d=6208. F-CLM-MOE-CONV-FUSE-FOLD-EQ, F-CLM-MOE-CONV-GEMM-FOLD-EQ
Worth knowing because "fusing the routed branches" is often assumed to be a numerics-changing optimization requiring a tolerance gate. For this class of fold it does not have to be — you can hold it to exact equality and still get the speedup, which makes it safe to enable by default.
Finding 2 — the routing glue is the utilization wall, and removing the host does not fix it
This is the part I would want to know if I were implementing block-sparse or routed attention at scale. We attacked utilization on the routed step from the host side, on an idle H100, D=1536:
| lever |
util MEAN |
note |
| async single stream |
10.3–12.5% |
byte-eq broken by a race |
| piecewise CUDA graph (fwd→bwd) |
13.17% |
byte-eq PASS |
| whole-step CUDA graph |
13.19% |
AdamW inside the graph |
And it got worse as the model grew: 11.94% at D=1536 → 10.39% at D=2560.
Verdict: CLOSED-NEGATIVE. The binding constraint is not host launch overhead — it is occupancy. The step is a chain of many small, largely sequential kernels (norm / element-wise / routing glue between the GEMMs) that under-fill the H100: median util 2%, peak 71–100% only during the brief GEMMs.
F-FUSION-OCCUPANCY-WALL
A cooperative megakernel moved utilization by −0.08 pp. Batch-fill gave a real self-speedup but flattened at ~3×.
The generalizable claim: in a routed architecture, the gather/route/combine glue between the dense GEMMs is where the GPU sits idle, and CUDA graphs and stream overlap do not recover it — they remove host latency, which was not the binding term. Raising occupancy needs the small inter-GEMM ops merged into fewer, larger kernels (or into GEMM epilogues), or a right-sized GPU: the same workload hit ~90% util on an RTX 5070 at D=1536. Utilization is GPU-relative, and an H100 can simply be too large for a given block size.
The question
For MoBA specifically: block selection is dynamic per query, so the routing glue is data-dependent in a way our static expert routing was not. Does that make the occupancy picture worse (more sync points, less foldable) or better (blocks are large enough to fill SMs on their own)? Our measurement says the glue dominates whenever the routed units are small; whether MoBA's blocks are past that threshold is exactly the thing I could not test.
Full post-mortem (MIT, includes three headline numbers we got wrong and retracted): dancinlab/hexa-lang#5035
Title: Measured note on routed/block attention at the kernel level: folding the routed compute is byte-exact and ~6.6×, but the routing glue is where the GPU idles
Not a bug report, and the weakest-fit of a few writeups I'm posting — MoBA is a method repo and this is kernel-level measurement, so close it freely if it's noise for you. Posting because we measured something specific about routed compute that I have not seen written down, and our project is shelved so it will otherwise be lost.
Context: we ran a mixture-of-experts conv/attention block (
CLMConvMoE) on our own compiler stack — own PTX backend, own GEMM, no cuBLAS call — and spent a campaign trying to raise GPU utilization on it.Finding 1 — folding E routed branches into one kernel is byte-exact, not approximate
Folding
Eseparate expert branches into a single fused kernel, and separately into a GEMM-with-weight-reuse path, was bit-identical to the naiveE ×ModuleList reference:max|Δ| = 0.0is an equality, not a tolerance. The GEMM-conv path measured 6.57× @ d=6208.F-CLM-MOE-CONV-FUSE-FOLD-EQ,F-CLM-MOE-CONV-GEMM-FOLD-EQWorth knowing because "fusing the routed branches" is often assumed to be a numerics-changing optimization requiring a tolerance gate. For this class of fold it does not have to be — you can hold it to exact equality and still get the speedup, which makes it safe to enable by default.
Finding 2 — the routing glue is the utilization wall, and removing the host does not fix it
This is the part I would want to know if I were implementing block-sparse or routed attention at scale. We attacked utilization on the routed step from the host side, on an idle H100, D=1536:
And it got worse as the model grew: 11.94% at D=1536 → 10.39% at D=2560.
A cooperative megakernel moved utilization by −0.08 pp. Batch-fill gave a real self-speedup but flattened at ~3×.
The generalizable claim: in a routed architecture, the gather/route/combine glue between the dense GEMMs is where the GPU sits idle, and CUDA graphs and stream overlap do not recover it — they remove host latency, which was not the binding term. Raising occupancy needs the small inter-GEMM ops merged into fewer, larger kernels (or into GEMM epilogues), or a right-sized GPU: the same workload hit ~90% util on an RTX 5070 at D=1536. Utilization is GPU-relative, and an H100 can simply be too large for a given block size.
The question
For MoBA specifically: block selection is dynamic per query, so the routing glue is data-dependent in a way our static expert routing was not. Does that make the occupancy picture worse (more sync points, less foldable) or better (blocks are large enough to fill SMs on their own)? Our measurement says the glue dominates whenever the routed units are small; whether MoBA's blocks are past that threshold is exactly the thing I could not test.
Full post-mortem (MIT, includes three headline numbers we got wrong and retracted): dancinlab/hexa-lang#5035