You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When speculative decoding verifies a Qwen 3.5 MTP draft block, the target model's per-position logits can disagree with single-token decode, which flips the accept/reject decision and breaks the temperature-0 byte-equality invariant (greedy speculative output should be identical to drafter-less greedy decode).
Root cause
The Qwen 3.5 MTP target-verify pass verifies the whole draft block in one forward. Its full-attention layers run the batched fused causal attention over the multi-token block, whose reduction order differs from single-query decode attention. The resulting per-position logit drift can flip the target's argmax at a verify position, diverging the accept/reject decision from the drafter-less greedy pass. Qwen 3.5 is especially sensitive because its gated attention and quantized projections amplify the drift through the gate.
Fix
Thread a target-verify flag through the Qwen 3.5 verify pass. When verifying a multi-token block, compute attention per query position: query i attends only to keys and values up to prefix_len + i + 1 with no mask, exactly as during single-token decode. The decode and prefill paths are untouched: the flag defaults to off and all existing call sites pass off, so non-speculative inference is byte-identical and pays zero overhead.
Acceptance criteria
The Qwen 3.5 MTP target-verify pass produces per-position argmax tokens identical to single-token decode at the same absolute positions.
Non-speculative decode and prefill remain byte-identical (no behavior change when the verify flag is off).
An executable regression test pins the verify-versus-decode argmax parity.
Activity