PQ2_0 decode: FMA-fusion reassociation + F32-native staging-store bank-conflict fix #164

Background

Follow-up to #157/#159/#161 (all closed, merged to dev, PR #163). Decode is at 17.85 tok/s (~35.7% of the ~50 tok/s roofline). A fresh ncu --set full pass + advisor review (2026-07-24) on the post-#161 kernel state found:

• A real pipeline shift: LSU (load/store) now leads ALU as the top-utilized pipeline in every kernel variant (80-84% vs 58-65%, was 78%/68% ALU-led pre-PQ2_0 decode: eliminate F32<->F16 conversion launches + ALU-bound ternary unpack reduction #161). ncu's own text attributes this to frequent, individually-cheap LSU instructions, not a stall signal — the real remaining lever is total instruction count on both pipes.
• Two new call-site shapes identified: grid=768 (attention Q+Gate projection) and grid=128 (attention K+V fused projection), both real/permanent/16x-per-decode-token.

Candidates to work through

1. FP32 FMA-fusion gap (~8% ncu Est. Speedup on the two dominant kernels, ~7-9% on every variant). Plausibly a side-effect of PQ2_0 decode: eliminate F32<->F16 conversion launches + ALU-bound ternary unpack reduction #161's own algebraic ALU reduction (traded IADD3 subtracts for extra FADDs, 360->368 SASS instructions on the _small family — already flagged honestly in that commit). Candidate fix: reassociate (pq2_0_code_dot(p,gx) - gx.sum) * scale into two chained FMAs. Pure arithmetic, zero new sync/shared-mem. Est. 1-5% decode, medium confidence.
2. Shared-memory bank conflict in the F32-native staging store (new finding) — the PQ2_0 decode: eliminate F32<->F16 conversion launches + ALU-bound ternary unpack reduction #161 F32-native conversion writes 4 separate scalar half stores/thread instead of the pre-PQ2_0 decode: eliminate F32<->F16 conversion launches + ALU-bound ternary unpack reduction #161 F16-native kernel's single vectorized 128-bit copy, creating a real (SASS-confirmed) 2.1-way bank conflict, ~9% Est. Speedup, concentrated on the FFN down-proj kernel (k=17408). Fix: pack into 2 half2 stores via __floats2half2_rn. Est. 2-10% on that kernel (~1-4% decode-wide), medium-high confidence.

A third candidate (software-pipelined weight-byte prefetch on the FFN gate+up kernel) was flagged but sequenced behind these two — real but higher-effort/lower-value, not part of this issue's initial scope.

Full advisor report and complete history: see [[prismml-bonsai-model]] in Claude's memory system, and native/kernels/pq2_0_gemv.cu's header comments (documents every prior attempt, 6 negative results, multiple real wins).

Test plan

• Each candidate: model/verify before implementing (ptxas -v, cuobjdump --dump-sass, bank-index/stride arithmetic worked through explicitly — this investigation's standing discipline), bit-exact unit test, full CUDA suite (with the real GGUF fixture wired in so the Bonsai-27B prefill produces -inf logit at index 6 (pre-existing, unrelated to #157/#159/#161 perf work) #162 regression test actually runs), real Bonsai-27B dotllm bench measurement.
• Keep what measurably wins, revert+document what doesn't (established pattern in pq2_0_gemv.cu).