Repository navigation
fix(test): repair gate failures left by #2034 and #2037 - #2080
Merged
Merged
Conversation
Three checks fail on main after those merges; this fixes each without weakening it. - verify-binary-assets: re-encode tests/fixtures/nemotron_parse_page.png as 8-bit grayscale with optimize=True (41.1 KB to 20.6 KB). Every pixel of the page is gray, and decoding to RGB with Pillow or the image crate's to_rgb8 (what the Nemotron-Parse processor calls) gives byte-identical pixels. The generator script now saves the same encoding. - tests::family_order_is_exhaustive: add the Speech family (Nemotron VoiceChat) to FAMILY_ORDER after Text-to-speech, the order docs/supported-models.md already uses. - gelu_approx_matches_mlx_nn_bit_for_bit: replace the Metal-measured f32 and bf16 constants with a reference transcribed from the pinned mlx.nn.gelu_approx source, evaluated on the same device and compared bit for bit over a 4097-point grid in f32, bf16 and f16, plus f64 sanity values with a tolerance. The f32 constant at 4.1 was one ulp off on ROCm.
inureyes
added
type:test
This was referenced Sep 30, 2026
inureyes
added a commit
that referenced
this pull request
Oct 2, 2026
mlxcel's CUDA overlay of MLX's promotion table (src/lib/mlx-cpp/patches-cuda/dtype.cpp, kept for the single-dtype bf16 decode graph of #636) resolves bf16 with f32 to bf16 on every device of a CUDA build. Five lib tests and two Nemotron VoiceChat TTS sites assumed upstream's f32: the RVQ depth-sum and the MoG head's gathered proj_mus/low_mat slabs returned bf16 on CUDA. Both now cast the gathered rows explicitly (exact, bit-identical off CUDA). A probe test pins the per-build rule for add, matmul and fast_rms_norm (f32 off CUDA, bf16 with it). The f32_weights and perception tests build their reference with an explicit in-graph cast on CUDA, and the MoG test loads weights through promoted_subset as the model does. The gelu failure was already fixed by #2080. Validated on GB10 (--features cuda) and a CPU-only build: f32_weights, fastconformer, nemotron_voicechat::tts and gemma3_backbone all pass; clippy clean. Closes #2087
inureyes
added a commit
that referenced
this pull request
Oct 2, 2026
…2093) mlxcel's CUDA overlay of MLX's promotion table (src/lib/mlx-cpp/patches-cuda/dtype.cpp, kept for the single-dtype bf16 decode graph of #636) resolves bf16 with f32 to bf16 on every device of a CUDA build. Five lib tests and two Nemotron VoiceChat TTS sites assumed upstream's f32: the RVQ depth-sum and the MoG head's gathered proj_mus/low_mat slabs returned bf16 on CUDA. Both now cast the gathered rows explicitly (exact, bit-identical off CUDA). A probe test pins the per-build rule for add, matmul and fast_rms_norm (f32 off CUDA, bf16 with it). The f32_weights and perception tests build their reference with an explicit in-graph cast on CUDA, and the MoG test loads weights through promoted_subset as the model does. The gelu failure was already fixed by #2080. Validated on GB10 (--features cuda) and a CPU-only build: f32_weights, fastconformer, nemotron_voicechat::tts and gemma3_backbone all pass; clippy clean. Closes #2087
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three gate failures landed on main with #2034 and #2037. There is no issue for them; each is explained below.
Changes
make verify-binary-assets:tests/fixtures/nemotron_parse_page.png(feat(nemotron_parse): port Nemotron-Parse on the seq2seq worker #2034) was 41.1 KB, over the 32 KB ceiling ontests/fixtures/*.png. The page is black text on white, so every pixel is gray. Re-encoding it losslessly as 8-bit grayscale with Pillowoptimize=Truebrings it to 20.6 KB (an RGBoptimize=Truere-encode only reached 39.6 KB, and no oxipng, optipng, pngcrush or zopflipng is installed here). Decoding the new file to RGB gives byte-identical pixels to the old one, checked with Pillowconvert("RGB")and with theimage0.25.10 crate'sto_rgb8(), which is whatNemotronParseImageProcessor::preprocess_to_veccalls. No per-file rule was needed.generate_nemotron_parse_page.pynow saves the same encoding.tests::family_order_is_exhaustive: feat(nemotron_voicechat): load VoiceChat and run offline duplex inference #2037 added theSpeechfamily (Nemotron VoiceChat, speech-to-speech) without listing it inFAMILY_ORDER. It now sits afterText-to-speech, the same order as the ASR, TTS and speech-to-speech sections ofdocs/supported-models.md, so no docs change.gemma3_backbone_tests::gelu_approx_matches_mlx_nn_bit_for_bit: the test hardcoded values measured on Metal, and ROCm's f32 result at x = 4.1 is 4.0999565 against the hardcoded 4.099957 (one ulp). The test now builds the reference from the pinnedpython/mlx/nn/layers/activations.pyline (Python operator order,mx.powerforx**3, weak scalars built asarray(float(v), x.dtype)as the bindings do), runs it on the same device, and compares bit for bit withgelu_approxover 4097 points in [-8, 8] for f32, bf16 and f16. Backend-independent f64 sanity values with a tolerance keep it from passing on a formula that is wrong on both sides.The reference is a transcription of the upstream expression, so it has the same ops as the port by design. What the test checks is that the port stays op-for-op equal to upstream; the f64 sanity values check the formula itself. Temporarily breaking
gelu_approxfails the test in each case:x * x * xforpower(2 of 4097 f32 elements differ),0.0447for0.044715(2011 f32 elements), computing in f32 and rounding once (1270 bf16 elements), and0.0447changed in both the port and the reference (caught by the f32 sanity check).Verification (gfx1151,
--features rocm)cargo test -p mlxcel --lib models::gemma3_backbone::gemma3_backbone_tests: 5 passed.cargo test -p mlxcel --bin mlxcel tests::family_order: 2 passed.cargo test --test nemotron_parse_real_model -- --ignored: builds and runs; all three skip, since no Nemotron-Parse checkpoint is on this host.cargo test --test cli_help_consistency: 27 passed.make verify-versions verify-kernel-dtype-keys verify-kernel-port-dispatch verify-llama-compat verify-rocm-overlay verify-fmt verify-binary-assets: pass.cargo clippy -p mlxcel --lib --bin mlxcel --tests -- -D warnings: clean.Not verified: Metal and CUDA (not available on this host). The gelu test should still pass on Metal: the port and the reference issue the same MLX ops in the same order on the same stream, so each backend's kernels round them identically, and the sanity tolerances (1e-6 in f32, 4e-3 plus 1% in bf16) cover the values #2037 pinned on Metal. The same argument holds for CUDA. The other two changes are not backend-specific.