Repository navigation
fix(make): correct the TF32 note, klear reproduces too and is flaky - #1263
Merged
Merged
Conversation
#1262 recorded that `MLX_ENABLE_TF32=1` reds "three of the four" tests #1088 listed. That count came from single observations of each test and it is wrong. Repeating klear 10 times at `07474633` on GB10 (sm_121) fails 8 of them, with a max logit delta ranging 5.9e-4 to 2.0e-3 and a median of 1.2e-3, against a 1e-3 tolerance. All four tests depend on the pin, not three. The two green klear runs that produced the wrong count were the tail of that distribution, so the note now says explicitly that a single green run under `MLX_ENABLE_TF32=1` is not evidence that a test has stopped depending on the pin. Leaving the old wording in place would invite the next reader to treat klear as a standing exception and to conclude from one run that the pin no longer matters for it. The same 10-run comparison also quantifies the pinned side, which the previous note did not: with the pin active the delta is 3.6e-7 to 7.2e-7 over 10 runs, about 2500x smaller and effectively deterministic. MLX's reduced-precision kernel is nondeterministic run to run and the full-precision path is not, which is worth recording next to the policy it justifies. Comment-only change; `make -n verify-test-cuda` emits the same recipe. Refs #1088, #1262
inureyes
added
type:docs
5 tasks
inureyes
added
status:done
inureyes
added a commit
that referenced
this pull request
Aug 20, 2026
## Summary #1263 fixed the count in the `verify-test-cuda` TF32 note but overreached on the mechanism. It asserts that "MLX's reduced-precision kernel is also nondeterministic run to run". The evidence behind that sentence was klear alone, and the other three tests do not behave that way. This narrows the claim to what was actually measured. ## Related issues Refs #1088. Corrects the note as amended by #1262 and #1263. ## Type of change - [x] `docs` (comment-only, no recipe change) ## What the control run showed The other three tests #1088 listed, run 6 times each under `MLX_ENABLE_TF32=1` on the same host (GB10, sm_121) and commit, fail every run at a **bit-identical** magnitude. One distinct value each across all 6 runs: | test | magnitude, all 6 runs | | --- | --- | | bailing GLA | `state[11]: chunked -0.23453808 vs recurrence -0.23468243` | | deepseek_v2 MLA | `step 1 drifted by 0.00027007936` | | florence2 | `max abs diff 0.00014263391` | The florence2 figure is also identical to the one #1088 recorded at MLX pin `2c46b953`, so that path is stable across the pin bump as well. Nothing here supports a backend-wide nondeterminism claim. ## Why klear is the outlier It is the only one of the four that builds and runs a **whole model**. `small_args()` in `src/models/klear_tests.rs` configures 4-bit quantization (`group_size: 64, bits: 4`) with `num_experts = 8` and `num_experts_per_tok = 2`. The other three are isolated module-level parity checks over small dense f32 tensors: the bailing test drives the GLA kernel on noise arrays with no model, MoE or quantization at all, and the deepseek_v2 test drives the MLA attention module alone. klear also varies at **both** precision settings, 3.6e-7 to 7.2e-7 with the pin active against 5.9e-4 to 2.0e-3 without. So the pin bounds the magnitude, not the variance, and TF32 is amplifying a spread that is already present. ## What the note says now The operational advice is unchanged and is the part that survives: a single green run under `MLX_ENABLE_TF32=1` is not evidence that a test has stopped depending on the pin. The klear variance is described as klear's own, attributed structurally, and pointed at a separate issue rather than expanded on here, since it is not a property of the gate. ## Test plan - [x] 6 runs of the other three tests under `MLX_ENABLE_TF32=1`, magnitudes compared for bit-identity - [x] `make -n verify-test-cuda` emits the same recipe as before - [x] Comment-only diff, no Rust sources touched
inureyes
added a commit
that referenced
this pull request
Aug 20, 2026
The new comment on `the_prefill_is_causal_without_being_handed_a_mask` said a genuinely bidirectional prefill's 1.6 move on row 0 is "roughly 3e3x the default-precision figure". Against the default-precision delta the same comment quotes, 1.0485351e-3, the ratio is 1.6 / 1.0485351e-3 = 1526, so it is roughly 1.5e3x. The companion figure is right: 1.6 / 5.364418e-7 = 2.98e6, hence 3e6x under the pin. This comment block is the fourth pass over the same claim (#1262, #1263, #1264, then #1265), so a 2x-off ratio here would be a fifth. No behavior change; comment text only. Refs #1265
5 tasks done
4 tasks done
inureyes
added a commit
that referenced
this pull request
Oct 6, 2026
## Summary #1065's title and "Hardware dependence" section are wrong: the failures are not M5-specific. The cause is MLX's default `MLX_ENABLE_TF32=1`, which runs f32 GEMMs with 10-bit mantissa inputs on CUDA Ampere+ (cuBLAS `CUBLAS_COMPUTE_32F_FAST_TF32`) and Apple GPU generation 17 (NAX). The M1 Ultra passed only because it has no TF32-class path. The main fix already landed: #1260 pins `MLX_ENABLE_TF32=0` in both lib test binaries, #1262/#1263 document it at the `verify-test-cuda` gate, #1196 added the attention kill switch (which does not affect these tests). This PR closes what remained: - `mlxcel-core/src/tf32_pin_tests.rs`: an f32 GEMM (16x512x16, off MLX's gemv path) against an f64 host reference. Nothing in `mlxcel-core` went red on CUDA when the pin was lost, because the four tests #1065 named pass on CUDA even under TF32 (their shapes use gemv/SDPA, not cuBLAS). - Ctor doc comments in `src/lib.rs` and `mlxcel-core/src/lib.rs` no longer say "Apple only". - The two remaining chunked-SDPA assertions print the measured divergence (the two #1065 named already did). - `docs/environment-variables.md`: new `MLX_ENABLE_TF32` row with the measured decode impact; the `MLXCEL_METAL4_ATTENTION` row no longer names the route as #1065's suspect. Tolerances are unchanged. Shipped numerics stay on the MLX default, per the existing policy (unit tests verify full-f32 algorithm equivalence; TF32 output is not f32 evidence). ## Decision on production decode GB10, greedy, 256 tokens, default vs `MLX_ENABLE_TF32=0`, arms alternated: | checkpoint | weights | result | |---|---|---| | plamo-2-1b | f32 | 1 of 5 prompts diverges (after ~40 words), deterministic in both arms, 5/5 runs each; 38 tok/s both arms | | llama-3.1-8b-bf16 | f16 | identical, 3 prompts x 2 runs | | qwen2.5-0.5b-bf16 | bf16 | run-to-run variation in both arms alike, not attributable | | qwen3-0.6b | 4-bit | run-to-run variation in both arms alike, not attributable | TF32 changes greedy output only where dense f32 GEMMs exist, which in practice means f32 checkpoints. On CUDA, MLX sends unbiased single-row matmuls to a gemv kernel the setting does not reach, so prefill-sized GEMMs are the likely site; that split is read from MLX source, not measured. The default stays; the docs tell operators to set `MLX_ENABLE_TF32=0` for f32-exact output. ## Test plan - [x] At origin/main `5211f58d`, CUDA, `--test-threads=1`: the 4 `mlxcel-core` tests from #1065 pass by default and under `MLX_ENABLE_TF32=1` (3 runs each). The 3 root-lib tests from the 2026-08-17 comment pass by default and fail 3/3 under `MLX_ENABLE_TF32=1` (deepseek_v2 2.70e-4, florence2 1.43e-4) - [x] New sentinel: pass by default and with `=0`, fail 3/3 under `=1` (5.84e-3 vs 1e-4 bound) - [x] After the change: `layers::tests::chunked_`, `mla::decode::decode_tests`, `tf32_pin_tests` all pass on CUDA - [x] `cargo fmt --check`, `cargo clippy -p mlxcel-core --lib --tests --features cuda -- -D warnings` ## Not verified on this host - The four tests on M5 Max, and the M1 Ultra nightly runner: no Metal hardware here. #1065's 2026-08-17 comment already showed all four pass on M5 under `MLX_ENABLE_TF32=0`, and #1260 reported the suite green on M5 with the pin. - Sentinel failing under `MLX_ENABLE_TF32=1` on M5 (expected from the NAX dispatch, not run). - M5 vs M1 decode quality on real checkpoints. Closes #1065
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#1262 added a TF32 note to the
verify-test-cudacomment block and cited thatMLX_ENABLE_TF32=1reds "three of the four" tests #1088 listed. That count came from observing each test once, and it is wrong. klear reproduces too. All four tests depend on the pin.Related issues
Refs #1088. Corrects the note added in #1262.
Type of change
fix(documentation accuracy; comment-only, no recipe change)What was wrong and how it was found
klear was checked once under
MLX_ENABLE_TF32=1, passed, and got recorded as an exception. It is not an exception, it is flaky: its max logit delta straddles its own 1e-3 tolerance. Repeating it 10 times at07474633on GB10 (sm_121, Linux aarch64):MLX_ENABLE_TF32=0(pin active)MLX_ENABLE_TF32=1The two green runs behind the wrong count were the 20 percent tail. #1088's originally recorded 1.1e-3 sits at the median of this distribution, so klear behaves exactly as that issue described and nothing about it has changed.
This also refutes a hypothesis recorded in #1088's closing comment, that the MLX pin bump from
2c46b953to9a795735had nudged klear under its bound. It did not. klear was never under the bound; it straddles it.What the corrected note now says
MLX_ENABLE_TF32=1, three deterministically and klear 8 runs in 10.MLX_ENABLE_TF32=1is not evidence that a test has stopped depending on the pin. That is the exact mistake this PR corrects, and it is cheap to repeat.Test plan
models::klear::tests::the_prefill_is_causal_without_being_handed_a_mask, instrumented to print the max delta, instrumentation reverted afterwardsmake -n verify-test-cudaemits the same recipe as before