Repository navigation
fix: restore Gemma 4 31B MTP exactness - #1987
Merged
Merged
Conversation
Gemma 4 31B MTP compared multi-token verification with single-token decode using different full-attention, sliding-ring, and long-prefill arithmetic, causing the exactness gate to decline QAT and non-QAT checkpoints. Reproduce decode attention per query row for the measured 31B geometry, preserve reference-ring metadata across cache lifecycle operations, match scheduler prefill chunks, and localize failed probes. Add focused regressions and M5 Max evidence for both 31B checkpoints, shared-cache consumers, and Qwen 3.8 MTP. Validation: - Formatting, staged diff checks, and measurement-harness syntax checks passed. - M5 Max QAT/non-QAT parity, Gemma 3 and Qwen 2.5 classic smoke checks, and Qwen 3.8 MTP regression checks passed. - CUDA validation was unavailable because the GB10 host could not be accessed. Refs #1983, #1279
inureyes
added
status:review
Record the Gemma 4 31B exactness decisions, validation evidence, performance limits, and open backend and model follow-ups in English and Korean. Add the final Unified 12B short-probe pass to the benchmark report and machine-readable evidence. Validation: - JSON parsing and diff checks passed. - Full make verify and the direct buffered-cache donation regression passed before report generation. Refs #1987, #1983, #1279, #1986
This was referenced Sep 26, 2026
inureyes
added a commit
that referenced
this pull request
Sep 26, 2026
## Summary A plain macOS Cargo build enabled MLX Metal but compiled the runtime QMV setter as a no-op because its bridge used the optional Cargo feature instead of the actual backend setting. This explains the reported automatic-retry failure despite a fresh `MLXCEL_QMV_WIDE=0` process passing. - Resolve Metal once for CMake and both bridge controls, preserving explicit Metal-OFF and native non-macOS behavior. - Preserve initial and retry verdicts in warnings and cached decline reasons, including operator-pinned and unavailable-backend cases. - Document the old-binary workaround and add build-policy tests plus a real runtime-control probe. ## Test plan - Full local `make verify`: passed; after the final test-only adjustment, the affected 16-test suite passed with QMV unset, pinned to 0, and pinned to 1. - Release build without `--features metal,accelerate`: real QMV false/true and command-buffer control assertions pass. - Gemma 31B QAT: automatic retry and fresh narrow startup pass; 64 generated tokens match each other and the explicit-Metal release baseline. Acceptance is 33/84 in both fixed arms. - Qwen 3.8 MTP: both startup modes pass at block 3; 64 generated tokens and acceptance 35/52 match. Runtime validation used M5 Max. The user's original failure and fresh-narrow pass were on M5 Ultra; the corrected binary has not been run there. Cross-host compilation and physical Metal-OFF/CUDA execution remain unvalidated. This changes build wiring and diagnostics, not attention arithmetic. GB10-only CI remains unverified because its runner is unavailable; merge follows the user-approved exception for this investigation. Closes #1988 Refs #1987.
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
make verify, plus the explicitly ignored buffered-donation regression: pass. Final 12B startup also passes its retained short gate after the automatic narrow retry.Measurements and reproduction: benchmark report. Test-fast throughput varies by prompt and host state; no unchanged-throughput claim. GB10/CUDA and Qwen 3.6 MTP remain unvalidated (host inaccessible / no compatible local drafter). The experimental 12B long-context mismatch is tracked separately in #1986.
Maintainer approved merging with the three GB10-only CI jobs (
cargo-clippy,OpenXLA feature compile,WebUI installed artifact) still queued and unvalidated. All other active checks passed; the full local Metal gate passed.Closes #1983
Closes #1279