feat(qwen4_exp): support channel-wise fp8 dense weights and per-row PLE loaders - #498
Open
alvarorsouza-arch wants to merge 1 commit into
Open
alvarorsouza-arch wants to merge 1 commit into
alvarorsouza-arch wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds loaders for the
huginnfork/primitive-aicheckpoint variant of Qwen3.8-Flash-Next, which stores the dense backbone in channel-wise fp8 (separate per-channel scales) instead of the fused layout, and ships its PLE table in a per-row format.weight.py: fuse the channel-wise scale vectors in the expected order at load timeple.py/kernel/triton/ple.py: per-row PLE table loadermodel.py: wire the new loadersWhy
The un-quantized backbone of this checkpoint is ~11 GB of BF16. On a 16 GB card there is no room for a meaningful MoE cache with it. The fp8 backbone is 8.2 GB, which frees enough VRAM to enable
--moe-prefill-overlap.This addresses #409 (12 GB cards: ~9.9 GiB of unquantized attention/embedding weights leave no room for MoE cache + KV) for the 16 GB tier, the same way #390 did for the qwen3_5_moe family.
Measured (RTX 4070 Ti SUPER 16 GB, EPYC 7K62, 251 GB RAM)
Identical quality score against the same suite, A/B on the same server.
Test protocol
All numbers measured end-to-end over the OpenAI API. The quality suite is 40 known-answer questions plus 30 completions at temperature 0, A/B against the bf16 checkpoint on the same hardware. Happy to add a loader unit test if maintainers prefer.