You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Documented pre-existing flake (memory: known to pass standalone, fail in the full CUDA suite,
confirmed 5x across sessions) — surfaced again this session while shipping #170. Investigated per
user request to "explore the isolation flake."
Root cause (confirmed, not a real bug)
EagerVsGraphDecode_QuantizedKv_Match asserts maxDiff < 1e-3f on first-step logits, with a
comment claiming eager and graph paths are bit-identical since they invoke the same kernels.
Its sibling test right above it, EagerVsGraphDecode_Match, already documents and tolerates
(< 5.0f) the real cause: PTX-JIT cache state left behind by other CUDA tests in the same process
(module/kernel registration order shifts SASS scheduling, which shifts FP accumulation order in
otherwise-identical arithmetic) perturbs logits by up to ~0.25 max-abs in the full suite, with
argmax unaffected. The QuantizedKv test's stronger "bit-identical" claim just isn't true once other
tests have run first — reproduced directly this session:
Standalone: Step 0 logit max abs diff: 0.000000
Full DotLLM.Tests.Unit.Cuda suite: Step 0 logit max abs diff: 0.201172 — well inside the
sibling test's own accepted tolerance, comfortably failing the QuantizedKv test's 1e-3f.
Fix
Loosen the tolerance to 5.0f (matching the sibling test exactly) and correct the comment to
describe the real, benign cause instead of the disproven "bit-identical" claim.
Verification
Full DotLLM.Tests.Unit.Cuda suite, same conditions that previously reproduced the failure:
before fix — 1 failed (this test); after fix — 339 total, 303 passed, 36 skipped, 0 failed.
Fixed in #175 (merged to dev). Confirmed root cause: too-tight 1e-3f tolerance for a benign PTX-JIT/kernel-order FP drift already documented+tolerated (5.0f) in the sibling test. Full CUDA suite clean (339/303 passed/36 skipped/0 failed) under the exact conditions that previously reproduced the failure.
Activity
jamesburton commentedon Jul 24, 2026
Fixed in #175 (merged to dev). Confirmed root cause: too-tight 1e-3f tolerance for a benign PTX-JIT/kernel-order FP drift already documented+tolerated (5.0f) in the sibling test. Full CUDA suite clean (339/303 passed/36 skipped/0 failed) under the exact conditions that previously reproduced the failure.