Skip to content

Fix batched causal mask (b>1) across Qwen3/GLM4/SmolLM3/Z-Image serving models - #38

Merged
astorise merged 2 commits into
mainfrom
fix/batched-causal-mask
Jul 30, 2026
Merged

astorise merged 2 commits into
mainfrom
fix/batched-causal-mask

Conversation

@astorise

Copy link
Copy Markdown
Owner

Fixes the batched causal-mask under-sizing bug across the fork's serving-optimized models, by cherry-picking upstream huggingface#3780 (@aashish-thapa, addresses huggingface#3582). Commits and authorship preserved.

The bug

8 models build the causal mask as Tensor::from_slice(&mask, (b, 1, tgt, tgt + offset), …) while the slice only holds tgt * (tgt + offset) elements — one batch element's worth. For b > 1 this under-sizes the storage: from_slice against a concrete shape skips the length check, so the GPU standard-attention path (l > 1 prefill) gets garbage/NaN, and CPU-flash builds fail outright. Affected: qwen3, quantized_qwen3, qwen3_moe, quantized_qwen3_moe, glm4_new, quantized_glm4, smol/smollm3, z_image/text_encoder — i.e. exactly the batched-serving path.

The fix

Build the mask batch-independently as (1, 1, tgt, tgt + offset) and let broadcast_add fan it across the batch — correct for any b, and cheaper than materializing b copies. Includes a CPU regression test (qwen3::tests::causal_mask_is_batch_independent) asserting the shape, the causal-triangle values, and that broadcasting onto b > 1 scores gives every row the same mask.

Why huggingface#3780 (arbitration among the cluster)

Several upstream PRs target this "Qwen3 mask for batch>1" family (huggingface#3646/huggingface#3624/huggingface#3610/huggingface#3586/huggingface#3611/huggingface#3609/huggingface#3484/huggingface#3743). huggingface#3780 is the right pick for this fork:

CI gate (candle's own, all green)

  • cargo fmt --all -- --check
  • cargo check --workspace
  • cargo clippy --workspace --tests --examples --benches -- -D warnings
  • cargo test -p candle-transformers --lib causal_mask — regression + sibling mask tests pass

@astorise
astorise merged commit ccda6b4 into main Jul 30, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants