Skip to content

test(cuda): CudaGraphCaptureEquivalenceTest.EagerVsGraphDecode_QuantizedKv_Match too-tight tolerance causes full-suite-only failures #174

Closed

Description

@jamesburton

Context

Documented pre-existing flake (memory: known to pass standalone, fail in the full CUDA suite,
confirmed 5x across sessions) — surfaced again this session while shipping #170. Investigated per
user request to "explore the isolation flake."

Root cause (confirmed, not a real bug)

EagerVsGraphDecode_QuantizedKv_Match asserts maxDiff < 1e-3f on first-step logits, with a
comment claiming eager and graph paths are bit-identical since they invoke the same kernels.

Its sibling test right above it, EagerVsGraphDecode_Match, already documents and tolerates
(< 5.0f) the real cause: PTX-JIT cache state left behind by other CUDA tests in the same process
(module/kernel registration order shifts SASS scheduling, which shifts FP accumulation order in
otherwise-identical arithmetic) perturbs logits by up to ~0.25 max-abs in the full suite, with
argmax unaffected. The QuantizedKv test's stronger "bit-identical" claim just isn't true once other
tests have run first — reproduced directly this session:

  • Standalone: Step 0 logit max abs diff: 0.000000
  • Full DotLLM.Tests.Unit.Cuda suite: Step 0 logit max abs diff: 0.201172 — well inside the
    sibling test's own accepted tolerance, comfortably failing the QuantizedKv test's 1e-3f.

Fix

Loosen the tolerance to 5.0f (matching the sibling test exactly) and correct the comment to
describe the real, benign cause instead of the disproven "bit-identical" claim.

Verification

Full DotLLM.Tests.Unit.Cuda suite, same conditions that previously reproduced the failure:
before fix — 1 failed (this test); after fix — 339 total, 303 passed, 36 skipped, 0 failed.

Activity

  1. jamesburton commented on Jul 24, 2026

    @jamesburton
    OwnerAuthor

    Fixed in #175 (merged to dev). Confirmed root cause: too-tight 1e-3f tolerance for a benign PTX-JIT/kernel-order FP drift already documented+tolerated (5.0f) in the sibling test. Full CUDA suite clean (339/303 passed/36 skipped/0 failed) under the exact conditions that previously reproduced the failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions