Phase 4 readiness: DSL Qwen and legacy LlamaRuntime diverge numerically on identical weights

[michalharakal]

Finding

The new DSL Qwen path (QwenNetworkLoader.fromWeights → OptimizedLLMRuntime DIRECT) and the legacy LlamaRuntime Qwen path produce substantially different logits when fed identical FP32 weights, even on a tiny 1-layer model. Discovered while attempting Phase 1B.4 (parity test) of the no-model-duplication refactor (refs #46, the Phase 1A/B/C/2 PRs #109/#110/#111/#112/#113).

Concrete numbers — first token, logit[0]:

• DSL: -13.231035
• Legacy: -7.451726
• Absolute diff: 5.7793093

That's six orders of magnitude above any reasonable tolerance for "same math, different op ordering". This is a real semantic divergence, not numerical drift.

Reproducer (sketch)

Synthetic Qwen3-flavoured weights, dim=32, 2 heads, kvHeads=2, headDim=16, ffnDim=64, 1 layer, vocab=64. Small integer weight values for reproducibility. Includes attn_q_norm / attn_k_norm so both paths build a real qkNorm=true Qwen network. RoPE base = 1_000_000, RMSNorm eps = 1e-6.

val weights = DecoderGgufWeights(metadata, tensors)  // identical for both paths

// DSL path
val dslModel = QwenNetworkLoader.fromWeights(weights)
val dsl = OptimizedLLMRuntime(dslModel, ctx, OptimizedLLMMode.DIRECT, FP32::class)

// Legacy path (with MemSegWeightConverter to pre-transpose, mirroring CLI)
Arena.ofConfined().use { arena ->
    val mapped = LlamaWeightMapper.map(weights)
    val converted = MemSegWeightConverter.convert(mapped, ctx, arena)
    val backend = CpuAttentionBackend(ctx, converted, FP32::class, ropeFreqBase = 1_000_000f)
    val legacy = LlamaRuntime(ctx, converted, backend, FP32::class, eps = 1e-6f)

    dsl.forward(1)     // → logit[0] = -13.231035
    legacy.forward(1)  // → logit[0] = -7.451726
}

I had this test in a draft branch; pulled it from PR #112/#113's follow-up rather than ship a failing test without a root-cause investigation.

Hypotheses (most likely first)

1. RoPE convention mismatch. DSL's rope() defaults to RoPEMode.INTERLEAVED pairing (i, i+1). Legacy's CpuAttentionBackend.applyRopeGqa may use split-half pairing (i, i + headDim/2). The Gemma DSL had this exact gotcha — see the long comment at GemmaNetworkDef.kt:172-187 explaining that HF / GGUF storage convention is split-half and using interleaved gives "correct-by-accident" outputs at small N but compounds wrong rotations across positions.
2. QK-norm formula. DSL MultiHeadAttention.qNorm uses an RMSNormalization module applied on a reshape; legacy LlamaRuntime.applyPerHeadRMSNorm is inline math (LlamaRuntime.kt:181-). Eps placement (sumSq/headDim + eps vs (sumSq + eps)/headDim) or weight broadcasting could differ.
3. Attention scale placement. Legacy may apply 1/sqrt(headDim) at a different stage than DSL's MultiHeadAttention.
4. Transpose semantics on square Q/O. Legacy linearProject's shape heuristic relies on MemSegWeightConverter having pre-transposed weights to [in, out]. With the converter included in the test the divergence persists, so the transpose handling is probably not the root cause — but worth ruling out via a non-square test.

Implication for Phase 4 (CLI swap)

The plan (snazzy-wibbling-dewdrop) Phase 4 swaps the CLI's Qwen branch from LlamaRuntime → DSL. This is a real behavior change, not just an internal refactor as previously framed. We do not yet know which path produces correct output (= matches HuggingFace transformers reference on a real Qwen3 model). The Q8 smoke test in #113 only proves DSL FP32 ≈ DSL Q8 (round-trip consistency, not absolute correctness).

Phase 4 readiness checklist

Holding Phase 4 until at least one of:

• Root-cause the parity divergence. Localize per-layer or per-stage diffs (post-attn-norm, post-Q-proj, post-RoPE-Q, post-attn, post-FFN, ...) to identify which op disagrees. Cross-check against HF transformers reference on a tiny model.
• HF-reference smoke in tests/smoke/. Run a real Qwen3 GGUF through both DSL (Phase 4 candidate) and legacy CLI paths, capture top-K next tokens, compare against HF transformers (or llama.cpp) reference for the same prompt. Whichever matches HF wins.
• Resolve issue Byte-level BPE broken for GPT-2/Qwen models (affects both GGUF and SafeTensors) #52 (byte-level BPE). Without this, Qwen3 tool-calling is broken end-to-end regardless of which inference path ships, so we can't even smoke-test Phase 4 against real Qwen3-Instruct.

What is NOT at risk

The architectural wins from #109/#110/#111/#112/#113 stand:

• decoderTransformerNetwork shared builder
• qwenNetwork no longer a stub
• Generic loader naming (Decoder*)
• DecoderGgufMemSegConverter for the DSL path
• QwenNetworkLoader.fromGgufNative + Q8 round-trip smoke

The DSL Qwen path works — it just produces different output than the legacy path. Either could be the correct one; we just need to know which.

[michalharakal]

Update — RoPE NEOX fix on its way (#116)

Confirmed hypothesis 1 (RoPE convention) was correct: decoderTransformerNetwork was using LLaMA's default RoPEMode.INTERLEAVED while Qwen 2/3 GGUFs are stored with LLAMA_ROPE_TYPE_NEOX (SPLIT_HALF). PR #116 fixes that path.

But — #116 alone does not close this issue. Discovered while reviving the parity test: position 0 still diverges by exactly the same 5.78 logits even with the RoPE fix applied. That's because sin(0) = 0 makes RoPE rotation identity at position 0 regardless of pairing convention, so a position-0 divergence cannot come from RoPE.

Residual divergence sources (still to investigate)

Most likely candidates, ranked:

1. QK-Norm eps default mismatch. MultiHeadAttention.kt:146-152 builds RMSNormalization(intArrayOf(headDim), name = \"$name.q_norm\", unitOffset = qkNormUnitOffset) — no eps argument, picks up RMSNormalization's default eps = 1e-5 (RMSNormalization.kt:45). The legacy LlamaRuntime.applyPerHeadRMSNorm uses eps = metadata.rmsNormEps, which for Qwen3 is 1e-6. This is small numerically but real. Fix: route Qwen3's eps through qkNorm instantiation.
2. MHA bias defaults. MultiHeadAttention.kt:134-143 allocates 4–8 weight params depending on bias. Need to verify Qwen3 GGUFs come with bias = false and the DSL builder agrees.
3. Embedding scale. Llama and Qwen do NOT scale token embeddings (Gemma does, by sqrt(dim)). Verify both paths agree on no-scale.
4. Linear projection default bias. Same concern as MHA — Qwen3 uses no bias on Q/K/V/O/FFN linear projections; verify DSL dense() agrees.

Phase-4 readiness — revised

The synthetic logit-parity test is unlikely to be a reliable gate because each fix only addresses one source of divergence. Better gate: real-GGUF smoke comparison against HF / llama.cpp reference output — already the tests/smoke/smoke-test.sh harness, just needs an HF-reference comparison script. That's what the original Phase-4 readiness checklist proposed.

#52 (byte-level BPE) is now fixed by #115. With #116 (RoPE NEOX) on the way, two of three Phase-4 readiness items are addressed. The third — proper parity validation — likely shifts from "synthetic logit equivalence" to "real-output coherence".

[michalharakal]

Probe data — divergence after #116 + #117 merged

Re-ran the synthetic Qwen parity probe on develop (post-#116 RoPE NEOX, post-#117 QK-norm eps). Output:

```
=== Qwen DSL ↔ legacy LlamaRuntime divergence probe (#114) ===
step=0 token=1 max|Δ|=49.259285 rms|Δ|=27.070175 sample dsl[0]=-13.231035 legacy[0]=-7.451726
step=1 token=5 max|Δ|=71.070076 rms|Δ|=42.20553 sample dsl[0]=17.55949 legacy[0]=-8.11031
step=2 token=3 max|Δ|=6.9622736 rms|Δ|=4.5580883 sample dsl[0]=15.54761 legacy[0]=17.572042
```

(logit[0] divergence at step 0 was 5.78; the max divergence across the full vocab logit vector is 49.26. My earlier number was misleading — looking only at logit[0] understates the divergence.)

What this tells us

• Position 0 is identical to before the fixes (sample dsl[0]=-13.23, legacy[0]=-7.45) — RoPE is irrelevant at pos 0 (sin(0)=0), and the qkNormEps fix with unit-scale qNorm/kNorm weights barely affects output. So neither fix is visibly moving the synthetic-parity needle for this test, even though both are independently correct.
• The divergence is non-monotonic across positions (49 → 71 → 7) which suggests numerical cancellations in this specific synthetic scenario rather than a clean "growing per-position drift". Hard to draw signal from the synthetic test.

What this changes about #114

The remaining residual sources I ranked (MHA bias defaults, embedding scale) have all been verified to be correct in code:

• MultiHeadAttention(... bias: Boolean = false, ...) — default false, matches Qwen.
• Embedding (Embedding.kt:25-) — clean row gather, no scaling.
• EmbeddingAdapter — pass-through, no scaling.
• VoidDense (output projection placeholder) — clean input @ W^T, declared bias not applied to output.

So the dominant ~49-magnitude divergence at pos 0 must come from somewhere I haven't enumerated yet. Plausible:

• swiGluFFN builder vs legacy's gate.silu() * up * down — operand ordering or a subtle scale factor.
• VoidDense post-mapper handling — if WeightMapper somehow binds output.weight to the void-bias slot, or vice versa.
• RMSNormalization's computation order (eps inside vs outside the sqrt) — though that's likely numerical, not 49-magnitude.
• The legacy LlamaRuntime.linearProject shape heuristic (w_rows == x_cols → use as-is) — for square weights post-transpose, this might be wrong despite my earlier analysis.

Recommendation

Phase 4 readiness should pivot off synthetic logit parity onto real-GGUF output coherence. The synthetic test scenario isn't sensitive to the fixes I've shipped (RoPE doesn't matter at pos 0, qkNormEps only matters at non-unit weight scale), and the residual-source candidate list I built is exhausted with several still-unknown sources remaining.

The right gate is tests/smoke/smoke-test.sh against a real Qwen3-1.7B-Q8 GGUF, comparing the DSL CLI path's output to a HuggingFace transformers reference for the same prompt. That's a real correctness signal, not a synthetic one.

Closing this with the recommendation: keep #114 open as a known divergence between paths (informational), but don't block Phase 4 on closing it. Phase-4 readiness gate: real-GGUF smoke + #52 (now resolved by #115).

[michalharakal]

Closing in favour of #118 with a focused scope.

Summary of what this issue achieved

Two of the four enumerated divergence-source candidates landed as fixes:

• Hypothesis 1 — RoPE NEOX pairing → fixed by fix(qwen): apply NEOX (SPLIT_HALF) RoPE pairing for Qwen3 GGUFs #116. Qwen3 GGUFs are stored with LLAMA_ROPE_TYPE_NEOX (SPLIT_HALF), the DSL was using INTERLEAVED. Independently correct fix; matches RopeType.forArchitecture("qwen3") in llama.cpp.
• Hypothesis 2 — QK-Norm eps default → fixed by fix(transformer): thread metadata RMSNorm eps through QK-norm #117. MultiHeadAttention.qNorm/kNorm were using RMSNormalization's default eps = 1e-5 rather than metadata.rmsNormEps (1e-6 for Qwen3). Threaded through MHA + DSL + decoder builder.

The other two candidates were verified to be already-correct in code (MHA bias = false default, no embedding scale).

Why closing without resolving the synthetic divergence

The synthetic-test still shows max |Δ| = 49.26 between paths at position 0, root cause unknown. But this is the wrong gate:

1. Legacy LlamaRuntime is deleted in Phase 4. Optimising for DSL-vs-legacy synthetic equivalence is debt.
2. Neither path is the source of truth. HF transformers / llama.cpp on the real Qwen3 GGUF is.
3. The fixes shipped from this investigation (fix(qwen): apply NEOX (SPLIT_HALF) RoPE pairing for Qwen3 GGUFs #116, fix(transformer): thread metadata RMSNorm eps through QK-norm #117) are independently correct architecturally — they don't need synthetic equivalence to justify themselves.

#118 reframes the readiness gate as "DSL Qwen output coherence vs HF reference on a real GGUF", which is what actually matters for shipping Phase 4.