Repository navigation
fix(speculative): Gemma 4 MTP breaks byte-identity on generation 15+, and its gate returns unconditionally true #1188
Description
Activity
Measured: Gemma 4 projection shapes on M5 Max
Filling the "not yet measured" cell, using the cheap first step this issue names. Op-level only; this is not a Gemma 4 model-level chain run.
Apple M5 Max 128 GB, macOS 26.6.1, architecture generation 17,
mainate3f322e5. Shapes read from local checkpoints:gemma-4-31b-it-4bit(hidden 5376, intermediate 21504),gemma-4-12b-it-4bit(3840, 15360), both affine 4-bit group 64, vocab 262144.Affine 4-bit, the shipped configuration
shape T=2 T=3 T=4 T=8 31B MLP 5376 -> 21504 DIVERGES (2471 / 43008 bytes) DIVERGES (2471) DIVERGES (2471) DIVERGES (2471) 31B attn 5376 -> 5376 DIVERGES (539 / 10752) DIVERGES (539) DIVERGES (539) DIVERGES (539) 12B MLP 3840 -> 15360 DIVERGES (705 / 30720) DIVERGES (1612) DIVERGES (1612) not run The mechanism carries to Gemma 4. Every Gemma 4 projection shape breaks at
T = 2, and the MTP default block width is 4, so the shipped configuration is inside the broken range. The unconditionaltruearm is wrong on generation 15 and newer.Both thresholds are visible on one machine
Sweeping the 31B attention shape and reporting the magnitude rather than just equal/diverges shows two steps, not one:
T=1 equal T=2 DIVERGES (worst 539/10752 bytes) T=3 DIVERGES (worst 539/10752) T=4 DIVERGES (worst 539/10752) T=5 DIVERGES (worst 539/10752) T=6 DIVERGES (worst 539/10752) T=8 DIVERGES (worst 539/10752) T=10 DIVERGES (worst 1693/10752) <- second step T=11 DIVERGES (worst 1693/10752) T=12 DIVERGES (worst 1693/10752) T=16 DIVERGES (worst 1693/10752)The
T = 2step isuse_qmv_wide. TheT = 10step isget_qmv_batch_limitcrossing into the matrix-matrix kernel. Note the limit here is 10, not the 12 quoted in #1186: that 12 is thearch_size == 'd'branch, which is the Ultra. A Max lands in thedefaultbranch, which returns 10 for both operands above 4096 on a generation that is not 13 or 14.The mxfp4 claim does not hold on generation 17
#1186 infers from
use_qmv_wide(mode, d) = mode != "affine" || arch_gen >= 15that non-affine modes diverge on every generation, and measured mxfp4 diverging on M1 Ultra (3, 4, 2 of 10240 bytes atT = 3). On M5 Max the opposite is true. Measured at the same 5120 -> 5120 shape #1186 used, so this is directly comparable:mode T=1 T=2 T=3 T=4 mxfp4 group 32 equal equal equal equal affine 4-bit group 64 (control) equal DIVERGES (501 / 10240) DIVERGES (501) DIVERGES (501) nvfp4 group 16 equal equal equal DIVERGES (1 / 10752) The affine control diverging in the same run is what makes this a real result rather than a broken harness.
So the two modes behave oppositely across the two generations: M1 Ultra is affine-clean and mxfp4-broken, M5 Max is affine-broken and mxfp4-clean (nvfp4 marginal, one byte at
T = 4). Whatever the full dispatch story is, "mxfp4 diverges everywhere" is not it, and #1186's Impact section should be narrowed accordingly.This also strengthens the case for #1189's approach over a hardware table. Two generations, two quantization modes, and the answer inverts.
What this does not establish
- No Gemma 4 model-level block-vs-chain run. Op-level divergence is necessary but not sufficient for a token flip; the Qwen 3.5 case needed 241 tokens to produce 2 flips. A Gemma 4 chain test is still the thing that proves user-visible impact.
- No M1 Ultra control at Gemma 4 shapes. Predicted equal below
T = 12by the same mechanism, unmeasured. - Nothing here touches the structural work this issue identifies as the real cost, namely that Gemma 4's verify path drives model-owned
seq_id-keyed cache slots and needs a throwaway slot for the probe.
Reproduction was a scratch harness at these shapes rather than a committed test, since #1189 already carries
tests/metal_block_vs_chain_op_parity.rsand adding Gemma 4 shapes to it belongs on that PR or on this issue's own change.M5 Max confirms the exposure: all four Gemma 4 rows diverge at the production block width
Ran the diagnostic from #1192 on an M5 Max 128 GB (macOS 26.6.1, Xcode 26.5.0), branch
test/issue-1188-gemma4-op-parity-shapesat0e1f9f10. Build 4 minutes, test 0.75 s, no checkpoint.T = 3gemma4-12b q_proj 3840 -> 4096 (affine 4-bit g64) : DIVERGES (3147, 3231, 3356 of 8192) gemma4-12b kv_proj 3840 -> 2048 (affine 4-bit g64) : DIVERGES (1580, 1617, 1531 of 4096) gemma4-12b gate/up 3840 -> 15360 (affine 8-bit g64) : DIVERGES (24, 22, 30 of 30720) gemma4-12b down 15360 -> 3840 (affine 8-bit g64) : DIVERGES (8, 12, 9 of 7680) qwen3.8-27b gate/up 5120 -> 17408 (affine 4-bit g64) : DIVERGES (14002, 14264, 13650 of 34816) qwen3.8-27b o_proj 5120 -> 5120 (affine 4-bit g64) : DIVERGES (3853, 3951, 4099 of 10240) fast_rms_norm / fast_rope / SDPA : equalEvery Gemma 4 projection diverges, and the Qwen 3.8 affine rows diverge alongside them, which is the expected signature:
use_qmv_widekeys onarch_gen >= 15, not on the model family. The Gemma 4 arm ofmtp_capable_targetis admitting a host where its verify block is not byte-identical to the single-token chain. The model-level probe this issue asks for is justified.For contrast, the same commit on an M1 Ultra (generation 13) reports all four Gemma 4 rows
equalatT = 3.The 18/12 cliff does not appear here at all
T qwen q4 5120->5120 qwen q4 5120->17408 gemma q4 3840->4096 gemma q8 3840->15360 dense f16 1 equal equal equal equal equal 2 DIVERGES DIVERGES DIVERGES DIVERGES DIVERGES ... every width from 3 to 32: DIVERGESOn an M1 Ultra the same sweep shows Gemma 4's attention shapes holding to 17 and its MLP shapes breaking at 12, the two
get_qmv_batch_limitbuckets. Hereuse_qmv_widecatchesM >= 2first, so the batch limit is never reached and never observed. The per-shape cliff is a generation 13/14 phenomenon; from generation 15 there is a single threshold at 2.Amplitude varies by three orders of magnitude, and the gate must not care
The 8-bit MLP rows move 24 of 30720 bytes and 8 of 7680, while the 4-bit attention rows move about 38% of theirs. Byte-identity is broken in every one of them.
This is the argument for the gate deciding whether the arms differ rather than how much. A tolerance-based gate tuned on the loud rows would pass the quiet ones, and a quiet row still flips a near-tie argmax eventually: it is the same last-ulp mechanism, just rarer per position.
It also has a direct consequence for the probe design in #1186, filed separately below.
Follow-up on this issue
The measurement half of the acceptance criteria is now done for generation 17 and for the M1 Ultra control. What remains is the gate change: giving the Gemma 4 arms the same runtime probe, which needs a throwaway model-owned sequence slot because
forward_with_speculative_sinkskeys its cache onseq_idrather than taking a cache vector.- added a commit that references this issue
on Aug 16, 2026 - changed the title
[-]fix(speculative): the Gemma 4 MTP gate returns unconditionally true and is exposed to the same generation 15+ byte-identity break[/-][+]fix(speculative): Gemma 4 MTP breaks byte-identity on generation 15+, and its gate returns unconditionally true[/+]on Aug 18, 2026 Can the exactness cost be made conditional? Measured: no
Before wiring the gate, the obvious question is whether the 23.3% has to be paid unconditionally. The contract only breaks when a kernel difference flips an argmax, and a flip needs the top two logits to sit inside that difference. If near-ties were rare relative to the kernel error, the fast kernel could run every round and only ambiguous positions would be recomputed exactly.
Instrumented and measured. It does not work, and the numbers say why.
The two distributions
MLXCEL_MTP_GAP_LOG=1(added ine757aac1) printsposition, token, top1, top2, gapper verify position. Running a generation under each kernel selection and aligning the logs while the token streams still agree gives both halves. gemma-4-12b-it-4bit with the 4-bit assistant, block 5, 15 prompts, 444 aligned positions:quantity p50 p90 p99 max kernel delta on the winning logit 0.1250 0.3125 0.7500 3.0000 The delta is exactly zero at only 145 of 444 positions. The top-two gap has median 5.875, so typically the gap is about forty times the error, which is why most tokens agree. The problem is the tails: the gap is under 3.0 at 30.6% of positions.
The break-even
A guard recomputes the block exactly whenever any position's gap falls under
t. Always-narrow costs 1.180x the fast verify, so a guard has to come in under that.guard t covers delta up to delta coverage K=5 round ambiguous expected cost 0.25 0.125 70.9% 12.9% 1.153 0.50 0.250 89.9% 27.2% 1.321 1.00 0.500 98.0% 48.6% 1.573 3.00 1.500 99.8% 83.9% 1.990 6.00 3.000 100% 97.0% 2.145 Sound coverage costs nearly twice what simply running narrow costs. The one row that beats always-narrow,
t = 0.25, misses three potential flips in ten.Break-even needs the per-round ambiguity below 15.3%, which needs the per-position rate below 3.3%. Reaching that means guarding only
delta <= 0.125, and 29% of measured deltas exceed it.Why it fails
The kernel error and the top-two gap are the same order of magnitude. There is no threshold that is both cheap and safe, because the two distributions overlap where it matters rather than being separated by orders of magnitude.
A first pass with a thin sample looked promising, and that is worth flagging for anyone who repeats this: over a 16-position common prefix the max delta read 0.3125, which would have put the guard comfortably in the black. Across 444 positions the max is 3.0. The tail is the whole answer here, so a bound taken from a short run will mislead.
What this means for the decision
The choice in this issue stands as it was, without a third option: either pay the 23.3% and keep the contract, or keep the fast kernel and say plainly that Gemma 4 MTP is not byte-identical on generation 15+. There is no cheap conditional path between them.
Measured on Apple M5 Max (
Mac17,7), macOS 26.6.1, mlxcel ate757aac1.What the 23.3% actually buys, measured at the right shape
The decision here has been stuck between "pay 23.3% and keep byte-identity" and "keep the speed and admit it is not byte-identical", with nothing to say how much the second option costs a user. That is a gap in tooling rather than in judgement, so I built the missing measure:
examples/logit_traceplusscripts/compare_logit_traces.py(PR #1215 branch, commitbd97da0c). Teacher-forced, one trace per configuration, and it reports disagreement split by how decided the reference was.The answer
gemma-4-12b-it-4bit,
MLXCEL_QMV_WIDEoff as reference against on as candidate, forward width 5 behind 512 tokens of context, 300 positions:top-1 disagreement, all positions 4.0% top-1 disagreement, decided positions (reference top-two gap >= 2.0) 0.585%, 1 of 171 where the reference's choice landed when they disagreed rank 2 in 75%, rank 3 in 17%, beyond top-8 once logit delta on the reference's own choice p50 0.125, p90 0.500, max 1.50 Split by decidedness, the disagreement is almost entirely where the model had no preference:
reference top-two gap positions disagreed 0 to 0.5 41 21.95% 0.5 to 1 37 5.41% 1 to 2 51 0.00% 2 to 5 71 1.41% 5 to 10 66 0.00% 10 and up 34 0.00% So the trade is 23.3% of throughput against a 0.585% disagreement rate on positions the model had decided, three quarters of which are it picking its own runner-up. That is a materially different-looking decision from "23.3% for byte-identity", and it is the number this issue was missing.
One methodological point, because it inverts the answer
The forward width selects which kernel is measured, so it selects the answer. The same comparison reads:
forward width top-1 disagreement 8 20.6% 16 19.5% 32 0.0% 256 0.0% use_qmv_widesplits atM >= 2and the batch limit sends largerMto a matrix-matrix kernel both arms share, so anything traced at prefill width reports no difference at all and anything traced at a bare narrow width reports a catastrophe. Neither is the shape a verify runs at, which is a narrow forward behind a long context. The harness takes a prefill argument for exactly that reason.It also means the exposure is bounded in a useful way: prefill is unaffected, and the break lives only in the small-
Mwindow the verify block sits in.What I am not doing
Picking the default. The measurement narrows the question, it does not settle it: 0.585% on decided positions is small but it is not zero, and one of the twelve disagreements put the reference's choice outside the candidate's top eight. Whether that is worth 23.3% is a product call.
Measured on Apple M5 Max (
Mac17,7), macOS 26.6.1.- added 7 commits that reference this issue
on Aug 18, 2026
Status: measured and confirmed on 2026-08-18
This issue was filed as "the mechanism should carry over, nobody has run the comparison". The comparison has now been run on an Apple M5 Max and the mechanism does carry over. Gemma 4 MTP is not byte-identical to classic decode on generation 15+ hardware under the default kernel selection, and disabling
qmv_widerestores it exactly.The gate change is no longer speculative work. What is new, and what makes it a real decision rather than a formality, is the price.
The measurement
Host: Apple M5 Max (
Mac17,7), macOS 26.6.1, mlxcel at07ab8612,[profile.test-fast],--features metal,accelerate. Targetmodels/gemma-4-12b-it-4bit, draftermodels/gemma-4-12b-it-assistant-4bit,--draft-kind mtp,temperature 0, 250 tokens, two prompts.qmv_wideon (default)qmv_wideoffBlock width does not change the answer: block 2 and block 5 both diverge under the default, at the same character. That is consistent with the mechanism, since
use_qmv_widesplits atM >= 2and a block of 2 is already on the wrong side of it.Classic decode is deterministic run to run here (three runs, identical), so the divergence is systematic rather than the f16 jitter class.
One methodological trap, because it cost us a wrong answer first
Toggling
MLXCEL_QMV_WIDEchanges classic decode too, so the comparison has to hold the kernel fixed on both sides. Classic's multi-token prefill is itself anM >= 2forward, soclassic(wide on)andclassic(wide off)differ from each other at char 51 on the prose prompt. ComparingMTP(wide off)againstclassic(wide on)conflates the prefill effect with the verify effect and reports a divergence that says nothing about the contract.Second trap, for whoever reproduces this: the CLI echoes the prompt without a trailing newline and then appends
Loading MTP drafter from "..."to that same line, so a naive line filter leaves the banner in the compared text and every MTP run "differs from classic" at the character where the banner starts. Extract the generated text by splitting on the prompt echo and dropping the two drafter banner lines.What it costs
This is the number the decision turns on. Same host, code prompt, block 5, 300 tokens, three samples per arm alternated:
qmv_wideon (fast, not exact)qmv_wideoff (exact)Spreads were 0.04 and 0.37 tok/s, so the gap is not measurement noise: exactness costs 23.3% on this pairing. For comparison, the Qwen 3.5 arms pay about 17 to 20% for the same guarantee, and that cost is already being paid in production today because their probe runs.
So the change is not free, and it is not a bug fix that only removes a wrong claim. It converts an advertised 2.80x into an honest 2.14x on this prompt, or it keeps 2.80x and qualifies the claim. Both are defensible; silently advertising byte-identity while not measuring it is not.
What to change
mtp_exactness_gateinstead of returningtrue. Insrc/server/batch/speculative_burst.rs:The probe machinery in
src/models/speculative_exactness.rsis already family-agnostic: it compares aT = Kverify block against the single-token chain, and on failure retries withqmv_widedisabled before declining. Nothing in it is Qwen-specific.Decide what a failing probe should do for Gemma. Qwen's answer is "disable the kernel and keep the contract", which is why it pays 17 to 20%. Given the 23.3% measured here, the alternative worth putting on the table is to keep the fast kernel and state plainly that Gemma 4 MTP is not byte-identical on generation 15+, the way
MLXCEL_MTP_ALLOW_INEXACTalready allows. This issue should not pick that by default; it should make the choice explicit and measured.Qualify
docs/benchmarks.md. The MTP section overview still claims byte-identity with no qualification, and the Gemma 4 rows inherit it. The Qwen rows were qualified by fix(speculative): MTP temperature-0 byte-identity fails on Apple GPU generation 15+ and at block widths above the qmv batch limit #1186; these were not.Not covered here
The 31B and Unified pairings, and any generation 13 or 14 host. The measurement above is one target, one drafter, one machine. It is enough to settle that the exposure is real; it is not enough to characterise the cost across the family.