WebGPU: Qwen2 models produce garbled output (repeated @ token)

[c0d3rman]

Description

Qwen2 models produce garbled output (repeated @ / token ID 31) when using the ggml WebGPU backend in the browser. Other architectures (TinyLlama/Llama) work correctly on the same setup.

Environment

• Browser: Chrome 146, Dia (Chromium-based) — same result on both
• GPU: Apple Metal-3 (M-series Mac)
• WebGPU adapter: vendor: "apple", arch: "metal-3", features include shader-f16, subgroups
• Wllama fork: reeselevine/wllama master branch (PR Add disk space requirements to README.md #201 to ngxson/wllama)
• JSPI: Available and used

Models tested

Model	GGUF	Output
TinyLlama-1.1B-Chat Q4_K_M	TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF	✅ Coherent
Qwen2.5-1.5B-Instruct Q4_K_M	Qwen/Qwen2.5-1.5B-Instruct-GGUF	❌ Repeats @
Qwen2.5-1.5B-Instruct Q8_0	Custom fine-tune	❌ Repeats @
Qwen2.5-1.5B-Instruct Q4_K_M	Custom fine-tune	❌ Repeats @

All Qwen2 models produce identical garbage. The same GGUF files work correctly on CPU (WASM-only wllama) and local mlx-lm inference.

Suspected cause

Qwen2-1.5B has dimensions that differ from Llama:

• num_attention_heads: 12 (not a power of 2)
• num_key_value_heads: 2 (GQA ratio 6:1)
• hidden_size: 1536 (not a power of 2)
• intermediate_size: 8960
• rope_freq_base: 1000000

TinyLlama has num_attention_heads: 32, num_key_value_heads: 4 (GQA 8:1), hidden_size: 2048 — all power-of-2 dimensions. This suggests the WebGPU matmul or attention shaders may have an issue with non-power-of-2 head counts or hidden dimensions.

Steps to reproduce

1. Build wllama with ggml WebGPU (reeselevine/wllama master branch)
2. Load any Qwen2 GGUF model with preferWebGPU: true
3. Generate text — output will be repeated @ characters

Expected behavior

Coherent text output matching CPU inference.

cc @reeselevine

Edit: my coding agent posted this during a debugging run without asking me 😬 Feel free to ignore if irrelevant

[reeselevine]

Thanks for the report! Generally I'd agree that letting coding agents post Github issues for you probably isn't a great idea, and in this case it got its suspected cause completely wrong. But, as it turns out I'm kind of in a bug-bashing mode anyways right now to increase the stability of the WebGPU backend before we make an announcement about it, so this is helpful.

Turns out the issue is that matrix multiplication accumulation in fp16 causes NaNs specifically in Qwen2.5 models. I've tested it out locally and running natively on my M3, I get coherent output from the q4_k_m model. Unfortunately, this doesn't seem to fix the issue in the browser, I no longer see @ but I also don't get coherent output. However, using a Q4_0 and Q8_0 model does give coherent output in the browser now.

I did try moving everything for the q4_k_m model to f32 (including dequantization and shared memory storage), but unfortunately that doesn't seem to help. Using a legacy/non-parallelized version of matmul did work however, so this does seem to be a very specific issue related to the weights used by Qwen2.5 and floating point precision.

At this point, since Qwen2.5 is a slightly older model and I haven't seen the same issues with more recent models yet, I don't think I'm going to investigate further right now. If Qwen2.5 q4_k_m is very important to you though, maybe your agent can investigate further and drill down where the floating poing accumulation drifts too far right now in the browser :).

I do have a fix in #21644 for the f32 accumulation, since that does seem to be more stable in general. Unfortunately it won't help with the more optimal subgroup matrix multiplication or flash attention, but those aren't supported in the browser yet so I just left a TODO for now.

I actually already have the fix in #21052, but I hadn't gotten around to getting that PR ready yet, and the reason for it was unrelated to this bug.

[c0d3rman]

Thanks, sorry again about the hassle

[c0d3rman]

Disclaimer: This comment was written by a coding agent (Claude Code) on behalf of @c0d3rman, who reviewed and approved it before posting.

Update: browser-specific K-quant tiled matmul issue isolated

Following up on @reeselevine's suggestion to investigate where the accumulation drifts in the browser. Here's what I found:

The f32 accumulation fix (PR #21644) is necessary but not sufficient for Q4_K_M in the browser. After applying it, Q4_0 and Q8_0 produce correct output (confirming your finding), but Q4_K_M still produces incoherent output.

The issue is specific to K-quant types (Q2_K through Q6_K) in the register-tiled matmul path under Emscripten. Q4_K_M models use Q6_K for some tensors (attn_v, ffn_down, token_embd/output), and all K-quant types go through the tiled path for mat-mat operations.

It's not the dequantization. I built standalone WebGPU compute shaders that run the exact init_shmem_src0 Q4_K and Q6_K dequant logic (including the load_src0_u32_at unaligned two-word read path for Q6_K's 210-byte blocks) and compared against CPU reference values. All tests pass with zero error across aligned and misaligned blocks, including blocks containing f16 NaN bit patterns in the Q6_K struct fields.

Workaround that produces correct output: Force K-quants to the legacy (scalar) matmul path when compiled under __EMSCRIPTEN__:

case GGML_TYPE_Q6_K:
#ifdef __EMSCRIPTEN__
    use_fast = false;
#else
    use_fast = true;
#endif
    break;
case GGML_TYPE_Q2_K:
case GGML_TYPE_Q3_K:
case GGML_TYPE_Q4_K:
case GGML_TYPE_Q5_K:
#ifdef __EMSCRIPTEN__
    use_fast = false;
#else
    use_fast = !is_vec;
#endif
    break;

This gives correct output for Qwen2.5-1.5B Q4_K_M in Chrome (10/10 on a benchmark suite, verified on Apple Metal-3). Performance is ~3s/query vs ~17s with the ngl=28 workaround — the legacy matmul is slower than tiled for prefill but still much faster than keeping the output projection on CPU.

What I haven't identified: The exact mechanism by which Chrome's WebGPU diverges from native Dawn for the tiled K-quant accumulation. Since the dequant is provably correct, the difference must be in how the tiled accumulation loop (shared memory + barriers + f16/f32 multiply-add) compiles differently through Chrome's Tint vs standalone Dawn's Tint. Could be a Tint codegen difference, a shader optimization pass reordering, or a workgroup memory initialization interaction.

Side finding: Q6_K data in this model contains ~3.5M f16 NaN bit patterns across its ql/qh/scales fields (88% of blocks affected). Not the cause here (the tiled path reads via raw array<u32>, immune to NaN canonicalization), but relevant to PR #21521.

Tested on: Chrome 146, Apple Metal-3, Wllama fork at commit d08f192.

[reeselevine]

I have as minimal a repro as I can here: https://github.com/reeselevine/dawn-metal-issue

There could be an oob access/undefined arithmetic in our shader but it also could be something going wrong in the Dawn/Metal integration