decide what the MTP temperature-0 contract should promise, now that byte-identity is unreachable on Metal #1191

Summary

MTP speculative decoding ships with a stated contract: at temperature 0 its output is byte-identical to classic decode. Measurement on M5 Max says that contract is not reachable on Metal by fixing any single kernel, because a T = K matmul is not bitwise equal to K single-token matmuls and every projection in the model is a matmul.

This issue is the decision, not the investigation. The investigation is done and reported at #1182 (comment), and the diagnostic that produced it is #1190.

Decide what the contract should say, then make the code and the gate say it.

What the measurement established

Apple M5 Max 128 GB, macOS 26.6.1, main at 115f6089, Qwen3.8-27B pairing.

The three real-model chain tests from #1182 fail deterministically here: block_verify_chain_matches_single_token_chain diverges at 2 of 241 positions (first at chain index 48), and both adapter chain tests diverge at 1 position (chain index 151). The offline CLI and the server path both show it: server token ids agree for 13 tokens then diverge at index 13, with each arm individually deterministic.

Attribution, in order of elimination:

1. Not the gated-delta chain-parity kernel. Its synthetic bitwise pin passes, and the pin is live here (it fails under MLXCEL_GDN_CHAIN_PARITY=0), yet flipping the toggle does not move the real-model divergence by a single position or argmax value.
2. Not the M5 Neural Accelerator attention route. Patching should_use_metal4_attention() with a temporary override and rebuilding produced identical divergence.
3. Matmul. Per-op, T = 3 block versus T = 1 chain: quantized_linear at 5120 -> 17408 and 5120 -> 5120, dense f16, and dense f32 all diverge. fast_rms_norm, fast_rope, and SDPA (including the attend_per_position form) are all equal.
4. Not the qmv/qmm threshold. The break is at T = 2 and never recovers through T = 32, while get_qmv_batch_limit computes to 10 or 12 for these operands.
5. Not NAX alone. With the NAX gemm disabled for float32 via MLX_ENABLE_TF32=0 (arm-attribution confirmed: the block digest moves, the chain digest does not), the divergence survives and only shrinks from 72% of bytes to 37%.

Why this is a contract problem rather than a bug

The complete fix is to compute the verify path's linears per position, the way attend_per_position already handles attention. That makes a T = K verify cost K single-token forwards, which is the entire speedup. Measured on this box: T = 3 verify is 34.4 ms against 3 x 30.5 = 91.5 ms for the equivalent chain. There is no version of that which is worth shipping.

So the residual is structural. What is left to decide is what we promise about it.

Note also that the same matrix-vector versus matrix-matrix kernel pair exists on M1 Ultra, since NAX is the only part M1 Ultra lacks and the divergence does not need NAX. If that holds, #1182's M1 Ultra pass was the residual perturbation failing to flip a near-tie argmax in those particular 241 / 241 / 161-token chains rather than a structural guarantee, and the current gate promises something it cannot deliver anywhere on Metal. Confirming that is the one prerequisite for this decision and it costs a second: run #1190's diagnostic on the M1 Ultra box.

The options

Option A: restate the contract as greedy-equivalent with a measured flip rate

Say what is true: at temperature 0 the speculative path takes the same greedy decisions as classic decode except where a near-tie is resolved differently by a numerically different but equally valid kernel, at a measured rate.

• Replace the byte-identity assertions in the three real-model chain tests with a flip-rate bound (for example, no more than N flips per 1000 generated tokens), keeping the exact-match reporting as diagnostic output.
• Keep test: pin which primitives keep block-vs-chain bitwise equality on Metal #1190's assertions on fast_rms_norm, fast_rope, and SDPA, which are the ops that genuinely must stay position-independent.
• Document the rate per hardware generation in docs/benchmark_results/.
• Drop the framing that positions the Metal gate as an exactness guarantee.

This matches what upstream stacks promise. Neither mlx-lm nor vLLM claims bitwise identity for speculative decoding.

Option B: keep byte-identity as the gate and decline where it fails

Keep the promise and let the feature turn itself off wherever it cannot be kept.

• Requires a runtime probe cheap enough to run at load, since the property is per-hardware and per-shape.
• On the current evidence this would decline on M5 for certain, and plausibly on every Metal machine once the M1 Ultra measurement lands, which would make the feature unreachable rather than safe.

Option C: hybrid

Keep byte-identity as an opt-in strict mode for users who need reproducibility (declining to classic decode when it cannot be held), and make greedy-equivalent the default.

Recommendation

Option A, contingent on the M1 Ultra measurement. If matmul diverges there too, byte-identity was never a property this feature had, and Option B reduces to disabling MTP everywhere. Option C is worth doing only if someone actually needs bit-reproducible output, which nothing in the tree currently asks for.

Acceptance criteria

• test: pin which primitives keep block-vs-chain bitwise equality on Metal #1190's diagnostic is run on M1 Ultra and the result recorded on this issue. This gates everything else.
• A contract option is chosen and written down in the MTP module docs and docs/, replacing the current byte-identity language wherever it appears.
• The three real-model chain tests in tests/qwen38_mtp_chain_parity.rs are updated to assert whatever the chosen contract actually promises, and pass on both M1 Ultra and M5 Max.
• The Metal-only gate's rationale is corrected: it currently reads as protecting an exactness guarantee, and if that guarantee is restated the gate's justification has to be restated with it or the gate removed.
• The near-tie flip rate is measured and recorded per hardware generation for at least M1 Ultra and M5 Max.
• Gemma 4 MTP is checked against the same contract, since it shares the verify machinery and carries the same claim.

Not in scope

The gated-delta chain-parity kernel stays. It removed a large systematic drift (a state rounding difference applied at every GDN layer at every step, reliably flipping tokens every 100 to 250) and it is measured at about 3.4% end to end. It is necessary; this issue is about it not being sufficient.

Drafter-side performance is #1185. The verify-side device walk is #1179.

References

• feat(inference): support the qwen3_5_mtp drafter for Qwen3.5/3.6/3.8 MTP speculative decoding #1165, feat(inference): support the qwen3_5_mtp drafter for Qwen 3.5 MTP speculative decoding #1182: the Qwen 3.5 MTP drafter and the M5 Max measurements.
• test: pin which primitives keep block-vs-chain bitwise equality on Metal #1190: tests/metal_block_vs_chain_op_parity.rs, the per-op diagnostic.
• perf(speculative): cut the MTP drafter's per-step cost, now the binding constraint on M5-class hardware #1185: drafter per-step cost, the other thing M5 Max measurement surfaced.