Repository navigation
CPU IQ1_S prefill diverges from both GPU backends by 0.24 nats (found by the #256 gate) #276
Description
Summary
The first full sweep of the #256 cross-backend quantization gate found that CPU disagrees with
both GPU backends on the IQ1_S prefill leg by ~0.24 nats — roughly 5x the gate's 0.05 nat bound,
and ~80x the widest healthy spread measured anywhere else in the matrix.
As with the MXFP4 decode finding, CUDA and Vulkan agree with each other and disagree with CPU,
so the CPU path is the outlier.
Measurement
Fixture: Llama-3.2-1B-pure-IQ1_S.gguf (--pure, so IQ1_S is the only quantized path exercised).
Prefill leg: teacher-forced perplexity over 4096 corpus tokens, identical token sequence on every
backend.
| mean NLL vs CPU | relative perplexity | |
|---|---|---|
| CUDA | 0.242997 nats | 27.5% |
| Vulkan | 0.246257 nats | 27.9% |
| CUDA vs Vulkan (implied) | 0.00326 nats | — |
The two GPUs agree with each other to 0.0033 nats — comfortably inside the 0.05 bound, and in the
same band as every other healthy cell in the matrix. Both are 0.24 nats from CPU.
For scale: the widest healthy three-way spread previously measured across 24 types was 0.018
nats (Q2_K). This is 13x that.
Why this was not caught before
quant_matrix.tsv recorded IQ1_S as passing, but that matrix scored each backend's absolute
perplexity and looked for implausible magnitudes. On a --pure 1-bit fixture the model is
near-destroyed by design, so a 27% perplexity difference between two large numbers reads as noise
if you are only checking that the number is not 1e11. It is only visible as a spread against a
second and third backend on identical tokens, which is exactly the gap #256 exists to close.
This is the fifth defect of this shape (cf. #254, #255, #257, #258) and the first found by the
gate rather than by a hand-driven sweep.
Notes
- The decode leg's cosines for IQ1_S sit at 1-cos = 2.8e-3 (CUDA) / 3.0e-3 (Vulkan), which is
inside the healthy continuum — the divergence shows up on prefill/GEMM, not on decode/GEMV.
Whatever is wrong is in the path prefill takes and decode does not. - Reference llama.cpp
ggml-quants.cdequantize_row_iq1_sfor the authoritative IQ1_S grid /
delta convention; note IQ1_S carries a per-blockqhfield encoding both the grid index high
bits and the scale, which is an easy place to get a shift wrong in a way that degrades quality
smoothly rather than catastrophically.
Acceptance criteria
- Root cause identified on the CPU IQ1_S dequant/prefill path.
- A regression test that is shown to fail without the fix — and one that discriminates,
not a degenerate-shape test. -
CrossBackendQuantGateTests.Backend_AgreesWithCpu(IQ1_S, Cuda)and(IQ1_S, Vulkan)green. - Cross-backend check per CLAUDE.md across
src/DotLLM.Cpu/,src/DotLLM.Cuda/,
src/DotLLM.Vulkan/,src/DotLLM.Hip/.
Activity
jamesburton commentedon Aug 5, 2026
Correction: ignore the "prefill/GEMM, not decode/GEMV" line in the Notes section
The 0.24-nat prefill measurement — the actual finding — is unaffected. But the Notes section says
the divergence "shows up on prefill/GEMM, not on decode/GEMV" and that "whatever is wrong is in
the path prefill takes and decode does not". That contrast does not hold, and following it would
narrow the search wrongly.
The gate's decode leg does not exercise the GEMV path.
QuantGateBackendRunner.RunDecodere-prefills the whole context on every step (deliberately, so a KV-cache difference between
backends cannot be scored as a kernel difference), so
seqLenis roughly 6–15 and never 1 — andthe CPU fused-decode/GEMV path is gated on
seqLen == 1(
TransformerModel.cs:1244and:1606). Both legs of the gate are GEMM-shaped.So the right reading is simply: IQ1_S diverges by 0.24 nats on a 4096-token teacher-forced
comparison against two independent GPU backends that agree with each other to 0.0033 nats. The
decode-leg cosines sitting inside the healthy band says nothing about GEMV either way; it is a
much weaker signal because it compares four single positions rather than 4096.
The rest of the issue — the IQ1_S grid/
qhpacking against llama.cpp'sdequantize_row_iq1_s,and the acceptance criteria — is unchanged and is still the right place to start.
jamesburton commentedon Aug 5, 2026
Investigation update (2026-08-05)
Did a from-scratch byte-level audit of IQ1_S across all three backends against a freshly re-fetched
ggml-quants.c/dequantize_row_iq1_s(not cached/memory):DequantizeIQ1_S— correct. Entire 2048-entry codebook diffed programmatically against the authoritative table: 0 mismatches.MatMulDequantRows.cs) — decode and prefill run the identical code path (GemvDequantRowsis literallyGemmDequantRows(n:1)). No separate CPU prefill-only branch exists that could hide a bug.dequant_iq1_s_f16/f32, sharediq1_s_decodehelper) — correct, matches reference exactly.matmul_iq1_s_f32_gemv.comp,matmul_iq1_s_f32_gemm.comp) — two independently written shaders (not sharing a common helper), exactly the shape where a transcription bug would hide. Traced both element-by-element against the reference. Both correct, including theqhfield's dual-purpose bit layout this issue specifically flagged as risky.No unpacking/shift/layout bug found in any backend.
One real, confirmed difference: CUDA's default forward path dequantizes IQ1_S to FP16 before cuBLAS for both decode and prefill (
DOTLLM_ENABLE_HIGH_PRECISION_IQUANTS=1opts into F32). But two pieces of counter-evidence argue against this being the (sole) cause of the prefill-specific gap:Best-supported hypothesis: not a code bug — IQ1_S's
--pure1.5625-bpw fixture pushes the model into a near-degenerate/near-uniform output regime by design. In that regime, ordinary FP reduction-order differences between CPU's sequential dot product and the GPUs' parallel-reduction paths (similar in kind between CUDA and Vulkan, explaining why they agree with each other far more than with CPU) compound through many layers/tokens into a large aggregate NLL gap without any single implementation being wrong.Concrete next step to confirm/refute, once a real fixture + the #256 gate harness exist: re-run the prefill measurement with
DOTLLM_ENABLE_HIGH_PRECISION_IQUANTS=1on CUDA.--puresub-2-bit fixtures in the test(coverage): no end-to-end cross-backend gate for quantization kernels — how #254 reached dev green #256 gate, not a dequant change, since three independently byte-verified-correct implementations don't have a bug to converge on.No fixture (
Llama-3.2-1B-pure-IQ1_S.gguf) or harness (CrossBackendQuantGateTests) exists on the machines available to me right now to run this experiment directly — both were built in a session/machine whose.docs/corpora/output was never committed (gitignored by design). Flagging this as the concrete blocker for whoever picks this up next with access to that fixture.jamesburton commentedon Aug 6, 2026
Resolved: the #256 gate confirms no bug, on a real fixture with real GPU verification
The #256 cross-backend gate harness is now built and was run for real (CPU + CUDA + Vulkan, RTX 3060). Result: 105 total, 99 passed, 0 failed, 6 skipped (the 6 skips are BF16/Q8_0/F16 x {CUDA,Vulkan}, fixtures not yet generated -- unrelated to this issue).
Backend_AgreesWithCpu(IQ1_S, Cuda)andBackend_AgreesWithCpu(IQ1_S, Vulkan)both PASSED against the 0.05 nat/cosine thresholds, on a freshly generated--pureIQ1_S fixture (Llama-3.2-1B, imatrix-calibrated, via Ollama's bundledllama-quantize.exe) -- a different fixture than whatever produced this issue's original 0.24-nat measurement.This is consistent with and confirms the earlier investigation's conclusion (no unpacking/layout bug found in any of the three backends, byte-verified against the reference in all three): the original divergence was fixture-dependent noise in IQ1_S's inherently ill-conditioned near-degenerate regime, not a code defect. With a properly imatrix-calibrated fixture, all three backends agree cleanly.
Closing. If a future run against a different IQ1_S fixture reproduces a large spread again, reopen with the specific fixture/corpus so it can be compared directly against this passing one.
jamesburton commentedon Aug 6, 2026
Resolved -- see comment above. Cross-backend gate passes cleanly on a real fixture with real GPU verification.
test(quant): add cross-backend quantization gate harness (#256)
merge: cross-backend quantization gate harness into dev (#256)
jamesburton commentedon Aug 6, 2026
Update: confirmed on all three legs, both backends
Re-measured on
issue/256-cross-backend-quant-gateatd062d80b. IQ1_S is now the only cell in the42-cell matrix that fails every leg on both backends:
seqLen == 1(GEMV), worst stepThe prefill arm remains the dominant signal by a wide margin — ~5x the 0.05 nat bound, against two
independent GPU implementations that agree with each other to 0.0033 nats. The two decode legs sit
just under the gate's 0.994 cosine bound, so they corroborate rather than add much.
The Vulkan column is now trustworthy in a way it was not when this issue was filed:
d062d80bfixed a Vulkan defect where descriptor sets were keyed on
VkBufferhandles that the driverrecycles after
vkDestroyBuffer, which was corrupting every Vulkan cell's cached-decode leg. IQ1_Ssurvived that fix unchanged, which is what makes the CPU-is-the-outlier reading safe here.
For context on scale: after the Vulkan fix, 37 of 42 cells pass. The five remaining failures are
IQ1_S on both backends, MXFP4 on both backends (#275), and one IQ3_S/CUDA cached-decode step.