Repository navigation
diffusiongemma: first real-26B Vulkan generation — quantized Q5_0/Q8_0 down banks + UMA heap fallback (#40, #120) - #122
Merged
Conversation
#40) Env-gated (DOTLLM_DIFFUSIONGEMMA_GGUF + Vulkan device) denoise-loop generation on the real 26B GGUF via VulkanTransformerModel + DiffusionTextGenerator. Reports denoise-steps/sec, per-step canvas latency and effective tok/s (canvas/steps/prompt/PKV overridable via DOTLLM_DG_*). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ment (#40) GEMMA4-GPU-GAPS.md gets a SUPERSEDED header (the Vulkan/CUDA gemma4 gaps it scopes were closed the same overnight, merged via 41621d5); the DiffusionGemmaConfigExtractor remark no longer claims the removed ValidateGemma4UniformHeadDim guard blocks real-weight forward. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… fallback (#120, #40) Real unsloth Q4_K_M conversions ship ffn_down_exps as Q5_0/Q8_0 (never Q5_1), so every layer fell back to the F32 host-dequant path (~92 GB) and OOMed the 15.8 GiB strict device-local heap on Strix Halo. - Gemma4ExpertsKeepQuantized now accepts down in {Q5_1, Q5_0, Q8_0}. - Q5_0 downs repack bit-exactly to Q5_1 at upload (d*(q-16) = d*q + m with m = -16d, fp16-exact) and ride the validated Q5_1 scale-fold shader. - Q8_0 downs stay Q8_0 with ffn_down_exps.scale[e] folded into each block fp16 d; dispatched by the existing scale-less Q8_0 indexed kernel. - AllocateDeviceLocal: on VK_ERROR_OUT_OF_DEVICE_MEMORY retry the combined DEVICE_LOCAL|HOST_VISIBLE type (UMA/GTT heap), then host-visible (llama.cpp GGML_VK_ALLOW_SYSMEM_FALLBACK equivalent); opt-out via DOTLLM_VULKAN_STRICT_DEVICE_LOCAL=1; DeviceLocalFallbackCount reports use. - CPU unit tests: Q5_0->Q5_1 bit-exactness, Q8_0 scale-fold payload/rounding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first ladder picked the FIRST type matching the fallback flags, which on AMD APUs can share the exhausted carve-out heap - the retry then re-failed (canvas-256 real run OOMed in UploadMatrix despite the fallback). Enumerate all eligible types per rung instead: other heaps before the failed heap, larger heaps first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 3, 2026
Closed
There was a problem hiding this comment.
Pull request overview
Enables real-weight DiffusionGemma-26B end-to-end generation on the Vulkan backend by keeping Gemma-4 MoE down-expert banks quantized for real conversions (Q5_0/Q8_0) and improving Vulkan device-local allocation robustness via heap-aware fallback, with accompanying tests and measurement docs.
Changes:
- Keep Gemma-4 MoE experts quantized for real-world
ffn_down_expstypes by repacking Q5_0→Q5_1 at upload and folding per-expert scale into Q8_0 block scales. - Add heap-aware Vulkan
VK_ERROR_OUT_OF_DEVICE_MEMORYfallback across eligible memory types/heaps (opt-out via env var) and expose a fallback counter for harness reporting. - Add CPU-only exactness tests for the repack logic, an env-gated real-weight Vulkan generation integration test, and new/updated diffusiongemma documentation.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/DotLLM.Tests.Unit/Vulkan/VulkanWeightsQuantRepackTests.cs | Adds CPU-only tests validating Q5_0→Q5_1 repack and Q8_0 scale folding. |
| tests/DotLLM.Tests.Integration/Backends/DiffusionGemmaVulkanRealGenerationTests.cs | Adds env-gated real-weight DiffusionGemma-26B Vulkan end-to-end generation harness. |
| src/DotLLM.Vulkan/VulkanWeights.cs | Implements Q5_0→Q5_1 repack and Q8_0 scale-fold upload paths for MoE down banks; widens quantized eligibility. |
| src/DotLLM.Vulkan/VulkanDevice.cs | Adds heap-aware device-local OOM fallback logic and a counter for fallback occurrences. |
| src/DotLLM.Models/SafeTensors/DiffusionGemmaConfigExtractor.cs | Updates remarks documenting per-layer head_dim support (#36). |
| docs/diffusiongemma/REAL-26B-RESULTS.md | Adds real-weight CPU/Vulkan measurement notes and results matrix. |
| docs/diffusiongemma/GEMMA4-GPU-GAPS.md | Marks historical gap report as superseded and adds context pointers. |
| Half scale = *(Half*)s; | ||
| *(Half*)d = scale; | ||
| *(Half*)(d + 2) = (Half)(-16f * (float)scale); | ||
| new ReadOnlySpan<byte>(s + 2, 20).CopyTo(new Span<byte>(d + 4, 20)); // qh + qs verbatim |
| for (long b = 0; b < blockCount; b++) | ||
| { | ||
| *(Half*)d = (Half)((float)*(Half*)s * scale); | ||
| new ReadOnlySpan<byte>(s + 2, 32).CopyTo(new Span<byte>(d + 2, 32)); |
Comment on lines
+59
to
+63
| // Bit-exact: d·q and 16·d are both exact in F32 (11-bit × 5-bit significands), | ||
| // so d·q − 16d and d·(q−16) round identically. | ||
| for (int i = 0; i < elems; i++) | ||
| Assert.True(fromQ5_0[i].Equals(fromQ5_1[i]), | ||
| $"elem {i}: Q5_0={fromQ5_0[i]:G9} != converted Q5_1={fromQ5_1[i]:G9}"); |
Comment on lines
+107
to
+110
| Half foldedD = MemoryMarshal.Read<Half>(dst.AsSpan(b * Q8_0BlockBytes)); | ||
| int q = (sbyte)src[b * Q8_0BlockBytes + 2 + (i % GroupSize)]; | ||
| Assert.True(((float)foldedD * q).Equals(scaled[i]), | ||
| $"elem {i}: expected {(float)foldedD * q:G9}, got {scaled[i]:G9}"); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First real-weight DiffusionGemma-26B runs on the Vulkan backend, plus the fixes that made them possible. Closes #120.
What happened
DiffusionGemmaVulkanRealGenerationTests— env-gated (DOTLLM_DIFFUSIONGEMMA_GGUF+ Vulkan device) end-to-end masked-diffusion generation on the real 26B GGUF, reporting denoise-steps/sec, canvas latency and effective tok/s (DOTLLM_DG_CANVAS/STEPS/PROMPT/PKVoverridable). Passes on Strix Halo: 15 steps / 58.6 s at canvas 32, SC+PKV on, coherent factual completion.ffn_down_expsas Q5_0 ×16 + Q8_0 ×14 (never Q5_1) — all 30 layers previously fell into the F32 host-dequant fallback (~92 GB → OOM). Now: Q5_0 downs repack bit-exactly to Q5_1 at upload (d·(q−16) = d·q + m,m = −16d, fp16-exact — unit-tested) and ride the validated Q5_1 scale-fold shader; Q8_0 downs keep the existing scale-less Q8_0 indexed kernel withffn_down_exps.scale[e]folded into each block's fp16d.VK_ERROR_OUT_OF_DEVICE_MEMORY,AllocateInternalnow retries every eligible type — other heaps before the failed heap, larger heaps first, combined rung before plain host-visible (llama.cppGGML_VK_ALLOW_SYSMEM_FALLBACKequivalent). Opt-outDOTLLM_VULKAN_STRICT_DEVICE_LOCAL=1; occurrences surfaced viaVulkanDevice.DeviceLocalFallbackCount.REAL-26B-RESULTS.md(CPU + Vulkan generation results, throughput matrix vs llama.cpp), SUPERSEDED header on the staleGEMMA4-GPU-GAPS.md, stale#36guard remark fixed inDiffusionGemmaConfigExtractor.Measured (Strix Halo, same session, GPU idle-checked)
The diffusion gap is the dense SC soft-embed (~47 GFLOP/step at canvas 32) + host-side sampling — follow-up filed as #121 (also covers the canvas-256
VK_ERROR_DEVICE_LOSTTDR from monolithic dispatches).Tests
VulkanWeightsQuantRepackTests(CPU-only): Q5_0→Q5_1 dequant bit-exactness incl. ±0/subnormal scales; Q8_0 scale-fold payload/rounding invariants. Green.Gemma4DiffusionVulkanTests+Gemma4VulkanKvCacheTests(GPU, synthetic): green on gfx1151.DiffusionGemmaGgufForwardTests3/3 green; Vulkan generation test green at canvas 32.🤖 Generated with Claude Code