Problem
Running the CUDA arm of #1873 on spark-101 (GB10, sm_121, main at a4974fab, make verify-test-cuda, --features cuda, test-fast, --test-threads=1), two of five tests in src/lib/mlxcel-core/src/ssm_update_parity_tests.rs fail. ..._matches_graph_step_f32, ..._refuses_unsupported_shapes and ssm_kernel_kill_switches_turn_predicate_off pass.
ssm_update_kernel_matches_graph_step_bf16: granite-4.0-h-tiny act dtype 12 A_log dtype 12 on Cuda: new state: normalized rms 2.532e-3 (tol 1.0e-5), normalized max 4.127e-2 (tol 1.0e-4)
ssm_update_kernel_keys_on_a_log_dtype: granite-4.0-h-tiny act dtype 12 A_log dtype 12 on Cuda: new state: normalized rms 1.811e-3 (tol 1.0e-5), normalized max 2.788e-2 (tol 1.0e-4)
Deterministic: identical in the full gate, in an isolated --exact run (not JIT cache contamination), and with MLX_ENABLE_TF32=0 (not TF32). Both panic at ssm_update_parity_tests.rs:128 on the granite case (bf16 activations, bf16 A_log): y meets its bf16 budget, while the f32 state misses tolerance_for(FLOAT32) (line 58) by about 200x. The f32-A_log cases (Nemotron-H, second keys_on_a_log_dtype case) are never reached.
The file came from #2099 (57d8ed29, ROCm port), validated on gfx1151 only, so this is a new test failing on CUDA, not a regression. SSM_CUDA_SOURCE (src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp:659) and the HIP body share the arithmetic apart from the lane shuffle; both read (float)A_log[h_idx] and write the state as U (f32), so the cause is not obvious from reading.
Reproduce
cargo test -p mlxcel-core --profile test-fast --features cuda --lib --no-run
gpu-lock run --tag ssm-parity -- <mlxcel_core test binary> --test-threads=1 ssm_update_parity_tests
Ask
Determine whether the CUDA kernel or the MLX-op reference() (line 262) is off on CUDA, e.g. compare both against a CPU-stream f32 reference and against the production ssm_step graph on CUDA. Fix the kernel, or the test only if the reference is wrong. If the kernel is at fault, state the effect on granite-4.0-h (bf16 A_log) CUDA decode.
Acceptance Criteria
Refs #1873, #2099
Problem
Running the CUDA arm of #1873 on spark-101 (GB10, sm_121, main at
a4974fab,make verify-test-cuda,--features cuda, test-fast,--test-threads=1), two of five tests insrc/lib/mlxcel-core/src/ssm_update_parity_tests.rsfail...._matches_graph_step_f32,..._refuses_unsupported_shapesandssm_kernel_kill_switches_turn_predicate_offpass.Deterministic: identical in the full gate, in an isolated
--exactrun (not JIT cache contamination), and withMLX_ENABLE_TF32=0(not TF32). Both panic atssm_update_parity_tests.rs:128on the granite case (bf16 activations, bf16A_log):ymeets its bf16 budget, while the f32 state missestolerance_for(FLOAT32)(line 58) by about 200x. The f32-A_logcases (Nemotron-H, secondkeys_on_a_log_dtypecase) are never reached.The file came from #2099 (
57d8ed29, ROCm port), validated on gfx1151 only, so this is a new test failing on CUDA, not a regression.SSM_CUDA_SOURCE(src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp:659) and the HIP body share the arithmetic apart from the lane shuffle; both read(float)A_log[h_idx]and write the state asU(f32), so the cause is not obvious from reading.Reproduce
Ask
Determine whether the CUDA kernel or the MLX-op
reference()(line 262) is off on CUDA, e.g. compare both against a CPU-stream f32 reference and against the productionssm_stepgraph on CUDA. Fix the kernel, or the test only if the reference is wrong. If the kernel is at fault, state the effect on granite-4.0-h (bf16A_log) CUDA decode.Acceptance Criteria
ssm_update_parity_testspass undermake verify-test-cudaon GB10MLXCEL_SSM_KERNEL=0graph) is recordedRefs #1873, #2099