Repository navigation
fix(speculative): the Gemma 4 31B pairing's exactness probe fails both kernels on M3 Ultra, vetoing the burst #1217 just enabled #1279
Description
Activity
M5 Max cell: same both-kernel failure, so this is not an M3 Ultra quirk
Measured 2026-08-22 on Apple M5 Max 128 GB, macOS 26.6.1,
mainat92018a12, offline CLI,gemma-4-31b-it-4bit+gemma-4-31b-it-assistant-bf16.Probe: fails both kernels at every width tried, with a width-independent signature:
MTP declined: verify block position 0 differs from the single-token chain in 245722 of 524288 logit bytes. Disabling qmv_wide did not make it exact either.Identical byte count at
--draft-block-size2, 3 and 4 (M3 Ultra read 231782 of 524288). The CLI refuses with the actionable error; a server boot would warm the probe and fall back to classic decode with the WARN.What the decline costs here (single samples, code prompt, 200 tokens, quiet host):
arm tok/s classic decode 27.64 MTP forced ( MLXCEL_MTP_ALLOW_INEXACT=1), block 462.11 (2.25x) On this prompt the forced arm's 200 tokens came out byte-identical to classic despite the diverging logits, which is the usual near-tie story: the flips are prompt-dependent (the 2026-08-19 M1 Ultra sweep caught a real one on prose), so the probe's verdict stands and the default stays correct to fail closed.
Two implications for this issue:
- The gate-vs-gate conflict is generation-15+-wide, not M3U-specific: on every gen 15+ host measured (M3 Ultra, M5 Max), the batch-capable pairing perf(speculative): re-measure the batch-capable B=1 MTP hardware gate on M1 Ultra and M3 Ultra and adjust mtp_b1_default #1217 turned on by default is vetoed by the probe, so the static enable currently delivers nothing anywhere by default, at a measured cost of 1.95 to 2.65x (M3U) and 2.25x (M5 Max) left on the table.
- The mechanism is not the
qmv_wideM-split this family's gate machinery was built around: the divergence survives the narrow retry and its byte count is independent of the block width, which points at a T = 1 versus T > 1 path difference outside the quantized matmul dispatch (candidates worth checking: the single-query maskless SDPA fast path against the masked multi-token path, and the fix(core): f32 matmul and single-query SDPA lose ~fp16 precision on M5, breaking four numeric tests #1065 class of f32 op differences). The Unified 12B pairing passes the narrow retry on the same host, so whatever it is, it is specific to the non-Unified forward or this checkpoint's config (for examplefinal_logit_softcapping).
- added a commit that references this issue
on Aug 22, 2026 Tracking note: the diagnosis and fix for this decline now live in #1983 (per-layer localization of the first diverging layer and sub-op, then a verify-path fix), filed after the same failure reproduced on
gemma-4-31b-it-qat-4biton Apple Silicon and on GB10 CUDA (226171 of 524288 bytes). #1983 carries this issue's M5 Max findings (width-independent byte count, Unified 12B passing with the sameglobal_head_dim512) as input.This issue stays open for its policy and documentation criteria and closes with #1983's outcome:
- If fix: Diagnose and restore Gemma 4 MTP exactness on Metal #1983 restores byte-identity, the default gen 15+ configuration engages MTP and the remaining work here is refreshing
docs/benchmarks.md's 31B rows and themtp_b1_defaultcomment; the same PR can close both. - If fix: Diagnose and restore Gemma 4 MTP exactness on Metal #1983 concludes exactness is unreachable, the remaining work is stating in the docs and the
mtp_b1_defaultcomment that the 31B pairing needsMLXCEL_MTP_ALLOW_INEXACT=1on gen 15+ and what that forfeits.
Since #1983 was filed, the server's decline message names the probe verdict and the override once per process (PR #1978), so the "static gate says run, measured gate says decline" conflict is at least visible in the log.
- If fix: Diagnose and restore Gemma 4 MTP exactness on Metal #1983 restores byte-identity, the default gen 15+ configuration engages MTP and the remaining work here is refreshing
- added a commit that references this issue
on Sep 26, 2026 - added a commit that references this issue
on Sep 29, 2026
Problem
#1217 measured the batch-capable Gemma 4 31B + bf16 assistant pairing at 1.95x to 2.65x on M3 Ultra and switched
mtp_b1_defaultto enable the B = 1 burst for batch-capable targets from Apple GPU generation 15. But that measurement ran at9e2c6675, which predates #1258's Gemma exactness probe. On currentmain(dd21ada4) the probe for this pairing fails underqmv_wideand fails again with it disabled, so the default-env gate declines MTP at serve time on the very host and pairing #1217 enabled by default. The static gate says run, the measured gate says decline, and the decline wins, so the 1.95x to 2.65x is unreachable in a default configuration today.Evidence
Measured 2026-08-22 on a Mac Studio M3 Ultra 512 GB, macOS 26.6.1, during the issue #1261 measurement, recorded in
docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.mdin PR #1278.Two independent server boots of
mlxcel-server -m models/gemma-4-31b-it-4bit --model-draft models/gemma-4-31b-it-assistant-bf16 --draft-block-size 4withMLXCEL_ENABLE_MTP_B1=1 MLXCEL_MTP_ADAPTIVE=0both logged:The same host's Gemma 4 12B + 4-bit assistant pairing (block 5) and Qwen 3.8 27B + MTP head pairing (block 3) both pass after the narrow retry, so this is specific to the 31B pairing, not to the host or the gate.
A plausible mechanism is the one #1258's "What this does not settle" section names for the M1 Ultra prose divergence: a narrow-kernel divergence involving the 262144-wide LM head. The 31B's head is 5376 to 262144 where the 12B's is 3840 to 262144, so hidden width may be the discriminator, but that is a hypothesis, not a measurement.
Candidate directions
To be decided by measurement, not assumed:
mtp_b1_default's generation 15 batch-capable default should stay (it is harmless but misleading, since the exactness gate vetoes it) and updatedocs/benchmarks.md's 31B rows, which currently imply the speedup ships by default.MLXCEL_MTP_ALLOW_INEXACT=1throughput still matches perf(speculative): re-measure the batch-capable B=1 MTP hardware gate on M1 Ultra and M3 Ultra and adjust mtp_b1_default #1217's rows post-fix(speculative): probe Gemma 4 MTP exactness instead of assuming it #1258.Acceptance criteria
mtp_b1_defaultcomment state plainly that the pairing requiresMLXCEL_MTP_ALLOW_INEXACT=1on generation 15+ and what that forfeits.docs/benchmarks.md's 31B speedup rows carry the qualification.References
docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md(the record)