Skip to content

fix(cuda): ssm_update_kernel bf16 parity tests fail on GB10 #2142

Description

@inureyes

Problem

Running the CUDA arm of #1873 on spark-101 (GB10, sm_121, main at a4974fab, make verify-test-cuda, --features cuda, test-fast, --test-threads=1), two of five tests in src/lib/mlxcel-core/src/ssm_update_parity_tests.rs fail. ..._matches_graph_step_f32, ..._refuses_unsupported_shapes and ssm_kernel_kill_switches_turn_predicate_off pass.

ssm_update_kernel_matches_graph_step_bf16: granite-4.0-h-tiny act dtype 12 A_log dtype 12 on Cuda: new state: normalized rms 2.532e-3 (tol 1.0e-5), normalized max 4.127e-2 (tol 1.0e-4)
ssm_update_kernel_keys_on_a_log_dtype: granite-4.0-h-tiny act dtype 12 A_log dtype 12 on Cuda: new state: normalized rms 1.811e-3 (tol 1.0e-5), normalized max 2.788e-2 (tol 1.0e-4)

Deterministic: identical in the full gate, in an isolated --exact run (not JIT cache contamination), and with MLX_ENABLE_TF32=0 (not TF32). Both panic at ssm_update_parity_tests.rs:128 on the granite case (bf16 activations, bf16 A_log): y meets its bf16 budget, while the f32 state misses tolerance_for(FLOAT32) (line 58) by about 200x. The f32-A_log cases (Nemotron-H, second keys_on_a_log_dtype case) are never reached.

The file came from #2099 (57d8ed29, ROCm port), validated on gfx1151 only, so this is a new test failing on CUDA, not a regression. SSM_CUDA_SOURCE (src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp:659) and the HIP body share the arithmetic apart from the lane shuffle; both read (float)A_log[h_idx] and write the state as U (f32), so the cause is not obvious from reading.

Reproduce

cargo test -p mlxcel-core --profile test-fast --features cuda --lib --no-run
gpu-lock run --tag ssm-parity -- <mlxcel_core test binary> --test-threads=1 ssm_update_parity_tests

Ask

Determine whether the CUDA kernel or the MLX-op reference() (line 262) is off on CUDA, e.g. compare both against a CPU-stream f32 reference and against the production ssm_step graph on CUDA. Fix the kernel, or the test only if the reference is wrong. If the kernel is at fault, state the effect on granite-4.0-h (bf16 A_log) CUDA decode.

Acceptance Criteria

  • All five ssm_update_parity_tests pass under make verify-test-cuda on GB10
  • The PR states the root cause (kernel vs reference)
  • If the kernel was wrong, a granite-4.0-h CUDA decode comparison (kernel vs MLXCEL_SSM_KERNEL=0 graph) is recorded
  • Tolerances are not loosened to pass without a stated reason

Refs #1873, #2099

Activity

  1. added
    type:bugBug fixes, error corrections, or issue resolutions
    area:coremlxcel-core: MLX FFI, primitives, KV cache, layers
    platform:linuxLinux (CUDA / packaging) specific
    on Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:coremlxcel-core: MLX FFI, primitives, KV cache, layersplatform:linuxLinux (CUDA / packaging) specificpriority:highHigh prioritystatus:readyReady to be worked ontype:bugBug fixes, error corrections, or issue resolutions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions