Repository navigation
Vulkan MoE routed-bank device-resident coverage: add Q5_0 / IQ4_NL / Q2_K and make the F32-skip decision per bank instead of per model #327
Description
Summary
VulkanWeights.MoeRoutedRawDeviceQuantType can keep a routed MoE expert bank device-resident only when it is Q8_0, Q4_K, Q5_K, Q6_K, or (with coopmat) F16. Everything else falls back to a host F32 dequant, and CanSkipMoeF32HostDequant is model-global all-or-nothing: one unsupported bank forces every bank of every MoE layer through F32.
For DeepSeek-V2-Lite that is ~57 GB of host RAM, which OOMs on a 128 GB box (see #326 for the OOM triage and the exact per-file tensor census).
Two independent gaps
1. Missing quant types. Real, shipping llama.cpp GGUFs use formats we do not cover — and not by accident. DeepSeek-V2-Lite's moe_intermediate_size = 1408 is not a multiple of QK_K = 256, so llama.cpp is forced to fall back to a non-K format for every ffn_down_exps:
| GGUF | gate_exps | up_exps | down_exps |
|---|---|---|---|
mradermacher/DeepSeek-V2-Lite-GGUF Q4_K_M |
Q4_K (26) | Q4_K (26) | Q5_0 (14), Q8_0 (12) |
bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF Q2_K |
Q2_K (26) | Q2_K (26) | IQ4_NL (26) |
So Q5_0 and IQ4_NL are not exotic — they are what you get whenever the MoE intermediate size is not a multiple of 256, which is every DeepSeek-V2-Lite / Coder-V2-Lite build. Q2_K covers the whole low-bit MoE tier.
2. All-or-nothing granularity. skipF32MoeDequant is a single bool threaded into TransformerWeights.LoadDeepSeekMoeLayer. In the Q4_K_M file, 64 of 78 routed banks are already device-resident-capable; all 64 are host-dequantised anyway because 14 are Q5_0. Per-bank resolution would cut the footprint by ~80% for that file with no new kernels at all.
The all-or-nothing design is deliberate and documented: with skipF32MoeDequant: true, W1/W2/W3 are left as all-NULL pointer arrays, so a bank that then resolves to the F32 upload path reads a null pointer per expert — silent corruption, not a crash. Making this per-bank therefore requires the F32/raw decision to be represented per bank in MoeLayerWeights, not inferred from a model-wide flag.
Suggested direction
Reuse the indexed MoE MMVQ machinery from #137 (already shipped for Qwen3-MoE banks) to add routed-bank device-resident support for Q5_0, IQ4_NL and Q2_K, and make the skip decision per bank rather than per model. Gap 2 is the cheaper and higher-leverage half and can land first.
Acceptance criteria
- Per-bank F32-vs-raw resolution, with
MoeLayerWeightscarrying the per-bank decision so a null-pointer F32 read is structurally impossible. -
Q5_0,IQ4_NL,Q2_Krouted banks stay device-resident. -
deepseek-v2-lite-q4_k_manddeepseek-coder-v2-lite-q2_kload on Vulkan on a 128 GB box and passRealGgufVulkanParityTests. - Kernel-level parity tests for each newly covered quant against the CPU oracle.
Activity
jamesburton commentedon Aug 11, 2026
Scale correction (see #326 for the measurement). This host has 31.6 GiB of RAM, not the 128 GB assumed when the DeepSeek OOM was first recorded. The F32 fallback is 53.6 GiB, so on this box the fallback can never work for DeepSeek-V2-Lite — closing the coverage gap here is the only way these models run on Vulkan locally, not merely the tidier way.
That also raises the value of gap 2 (per-bank rather than model-global skip) relative to gap 1: for the Q4_K_M build, 64 of the 78 routed banks are already device-resident-capable today, so per-bank resolution alone would cut the fallback from 53.6 GiB to roughly a fifth of it — with no new kernels at all. Only the 14
Q5_0ffn_down_expsbanks would still need host F32, and that residue may well fit.The Q2_K build (78/78 unsupported: Q2_K gate/up + IQ4_NL down) gets nothing from gap 2 and needs the new quant coverage.
jamesburton commentedon Aug 11, 2026
Two follow-ups.
1. Memory framing corrected. The box has 128 GB installed but only 31.6 GiB OS-visible (~96 GB is the BIOS UMA carve-out for the iGPU). So my earlier "this host has 31.6 GiB of RAM" was imprecise; the operative number for a host allocation is still 31.6 GiB, so the conclusion stands — the 53.6 GiB fallback cannot fit, and closing this coverage gap is the only way DeepSeek-V2-Lite runs on Vulkan here.
2. A second unguarded caller.
VulkanPipelineTransformerModel.LoadFromGguf(src/DotLLM.Vulkan/VulkanPipelineTransformerModel.cs:90) callswith
skipF32MoeDequantleft at itsfalsedefault — so the dual-GPU pipeline path takes the host F32 expert dequant unconditionally, even for a model whose banks are entirely Q4_K/Q8_0 and would have been device-resident on the single-device path. That is a strictly worse case than the one triaged in #326 (which at least tries to skip), and it is not covered by #326's preflight either. Worth folding into this issue's per-bank work: whatever replaces the model-global flag should be applied at both call sites.Merge origin/dev into issue/344-unit0-q5_0
jamesburton commentedon Aug 15, 2026
Likely resolved as a side effect of PR #426
PR #426 (issue #407) adds
moe_indexed_matmul_q5_0_f32andmoe_indexed_matmul_iq4_nl_f32Vulkan shaders plus the residency wiring, so routed MoE expert banks of those types stay packed on device instead of expanding to F32.DeepSeek-V2-Lite Q4_K_M is the worked example in #344:
moe_intermediate_size = 1408, which is not a multiple ofQK_K = 256, so llama.cpp emits 14 of 26ffn_down_expsbanks as Q5_0 — and dotLLM expanded all of them.Verified independently during review of #426 (not taken from the PR body):
1408 % 32 == 0, so the Q5_0 block constraint is satisfied and the resolver returns Q5_0 for those banks.Both the arithmetic and the mechanism check out, so this should no longer occur once #426 lands.
Please confirm before closing
This was not verified against a real DeepSeek-V2-Lite GGUF — no copy was available locally during the review, and #426's own primary evidence is real-GGUF block parity on SmolLM2-135M pure-Q5_0/IQ4_NL fixtures plus synthetic MoE banks. The reasoning is sound and the numbers reconcile, but an actual load of DeepSeek-V2-Lite Q4_K_M showing zero host-F32 expansion is what should close this.
Note also a related finding from that review, which may matter here:
PlanMoeF32HostDequantandResolveMoeBankResidency(VulkanWeights.cs:2338, 2445) both early-return unlessconfig.MlaConfig is not null— so the residency-report evidence is structurally MLA/DeepSeek-shaped. That is fine for this issue (DeepSeek is MLA), but it means the same evidence cannot speak for non-MLA MoE models.Related: #344 (umbrella), #407, #426.