Skip to content

feat(qwen4_exp): support channel-wise fp8 dense weights and per-row PLE loaders - #498

Open
alvarorsouza-arch wants to merge 1 commit into
FlashML-org:mainfrom
alvarorsouza-arch:qwen4-exp-channelwise-fp8
Open

alvarorsouza-arch wants to merge 1 commit into
FlashML-org:mainfrom
alvarorsouza-arch:qwen4-exp-channelwise-fp8

Conversation

@alvarorsouza-arch

Copy link
Copy Markdown

What

Adds loaders for the huginnfork/primitive-ai checkpoint variant of Qwen3.8-Flash-Next, which stores the dense backbone in channel-wise fp8 (separate per-channel scales) instead of the fused layout, and ships its PLE table in a per-row format.

  • weight.py: fuse the channel-wise scale vectors in the expected order at load time
  • ple.py / kernel/triton/ple.py: per-row PLE table loader
  • model.py: wire the new loaders

Why

The un-quantized backbone of this checkpoint is ~11 GB of BF16. On a 16 GB card there is no room for a meaningful MoE cache with it. The fp8 backbone is 8.2 GB, which frees enough VRAM to enable --moe-prefill-overlap.

This addresses #409 (12 GB cards: ~9.9 GiB of unquantized attention/embedding weights leave no room for MoE cache + KV) for the 16 GB tier, the same way #390 did for the qwen3_5_moe family.

Measured (RTX 4070 Ti SUPER 16 GB, EPYC 7K62, 251 GB RAM)

before (bf16 backbone) after (fp8 backbone)
Backbone on GPU 11 GB 8.2 GB
Prefill 424 t/s (no overlap) ~1000 t/s (overlap on)
Decode ~30 t/s 37 t/s
Quality (40 known-answer QA + 30 cloze, temp 0) 39/40 39/40

Identical quality score against the same suite, A/B on the same server.

Test protocol

All numbers measured end-to-end over the OpenAI API. The quality suite is 40 known-answer questions plus 30 completions at temperature 0, A/B against the bf16 checkpoint on the same hardware. Happy to add a loader unit test if maintainers prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant