perf(cuda/quant): amortizing small-M quantized GEMM for batched decode on Blackwell #725

Context

This is the follow-up carved out of #637 (part of epic #623). #637 fixed the large-M quantized prefill GEMM on sm_120/121 by raising the CTA tile M cap to 128 (merged in PR #723, +38% prefill on llama-3.1-8b-4bit on GB10). But it does NOT address small-M batched decode.

The gap

On GB10 (Blackwell sm_121), batched decode does not amortize weight bandwidth across the batch:

• The qmm dispatch (patches/mlx/backend/cuda/quantized/quantized.cpp) routes M*B < 8 (batched decode with B in [2,8)) to per-row qmv (matrix-vector), which re-reads the full weight matrix for each of the B sequences -> O(B) weight reads.
• M*B >= 8 routes to qmm_sm80, but the Ampere 128-wide tile is ~94% wasted at M=8, so it is worse than per-row qmv (measured).
• Net: aggregate decode throughput is flat (~45 tok/s at B=1/2/4 on llama-3.1-8b-4bit on GB10); batched and sequential decode are a throughput wash. Measurements and dispatch map are in docs/benchmark_results/qmm-sm121-tile-tuning-gb10-2026-07-10.md, and the operator-facing caveat is in docs/CONTINUOUS_BATCHING.md.

Impact

This is why #714's --parallel 4 serving default does not improve aggregate decode throughput on CUDA (it is a TTFT/concurrency knob there, not a throughput lever). Also caps #638 (speculative continuous-batching) benefit on CUDA.

What's needed

An amortizing small-M path on Blackwell: either a weight-reusing batched qmv (load the weight tile once, apply across the small batch of B columns) or an sm_120/121 small-M quantized GEMM tile that keeps weights resident across the batch. This is likely upstream MLX territory (the qmm kernels are MLX JIT templates); an upstream MLX issue with the ncu evidence would be the path, tracked here.

Acceptance Criteria

• Small-M batched decode (B in [2,8)) on sm_120/121 amortizes weight reads across the batch instead of O(B) full-matrix re-reads.
• Aggregate decode throughput on llama-3.1-8b-4bit (GB10) scales with batch size B rather than staying flat at ~45 tok/s.
• ncu evidence captured; upstream MLX issue filed (if the fix lands in MLX JIT templates) and linked back here.
• docs/CONTINUOUS_BATCHING.md caveat updated once the amortizing path lands.

Technical Considerations

• Dispatch logic lives in patches/mlx/backend/cuda/quantized/quantized.cpp; the qmm/qmv kernels are MLX JIT templates, so the actual kernel change is likely upstream MLX.
• Refs perf(cuda/quant): sm_120/sm_121-tuned quantized GEMM for Blackwell prefill #637, epic: CUDA (MLX backend) prefill and decode acceleration program #623, perf(server): serving-throughput defaults: parallel decode, batched prefill, prompt cache #714, perf(speculative): CUDA drafter pairing matrix and continuous-batching integration #638.