[Kernel] Support deep_gemm for linear methods

[artetaout]

Insight

When using Qwen3-32B-FP8, the decode stage is using cutlass FP8, which makes it slower than sglang

• vLLM 27ms
  
  

• sglang 14ms
  
  

And as I checked, vLLM use DeepGEMM in MOE, but not in FP8LinearMethod
So I add an option to use DeepGEMM and JIT as sglang does

Accuracy Result

• origin cutlass FP8 decode

VLLM_USE_DEEP_GEMM=0 lm_eval --model vllm --model_args pretrained=/mnt/Qwen3-32B-FP8 --tasks gsm8k --batch_size auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.6224	±	0.0134
		strict-match	5	exact_match	↑	0.7445	±	0.0120

• our DeepGEMM FP8 decode

VLLM_USE_DEEP_GEMM=1 lm_eval --model vllm --model_args pretrained=/mnt/Qwen3-32B-FP8 --tasks gsm8k --batch_size auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.6346	±	0.0133
		strict-match	5	exact_match	↑	0.7430	±	0.0120

Performance Result

• origin cutlass FP8 decode

VLLM_USE_V1=1 VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3-32B-FP8/ --no-enable-prefix-caching

============ Serving Benchmark Result ============
Successful requests:                     10
Benchmark duration (s):                  4.56
Total input tokens:                      1000
Total generated tokens:                  1000
Request throughput (req/s):              2.19
Output token throughput (tok/s):         219.24
Total Token throughput (tok/s):          438.48
---------------Time to First Token----------------
Mean TTFT (ms):                          291.90
Median TTFT (ms):                        318.24
P99 TTFT (ms):                           319.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.06
Median TPOT (ms):                        42.83
P99 TPOT (ms):                           44.89
---------------Inter-token Latency----------------
Mean ITL (ms):                           43.06
Median ITL (ms):                         42.89
P99 ITL (ms):                            43.39
==================================================

• our DeepGEMM FP8 decode

VLLM_USE_V1=1 VLLM_USE_DEEP_GEMM=1 vllm serve Qwen/Qwen3-32B-FP8/ --no-enable-prefix-caching

============ Serving Benchmark Result ============
Successful requests:                     10
Benchmark duration (s):                  2.51
Total input tokens:                      1000
Total generated tokens:                  1000
Request throughput (req/s):              3.98
Output token throughput (tok/s):         398.03
Total Token throughput (tok/s):          796.07
---------------Time to First Token----------------
Mean TTFT (ms):                          287.71
Median TTFT (ms):                        314.18
P99 TTFT (ms):                           315.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          22.43
Median TPOT (ms):                        22.18
P99 TPOT (ms):                           24.44
---------------Inter-token Latency----------------
Mean ITL (ms):                           22.43
Median ITL (ms):                         22.24
P99 ITL (ms):                            22.69
==================================================

• to reproduce

python3 benchmark_serving.py  --model /data/Qwen3-32B-FP8 --num-prompts 10 --dataset-name random --random-input-len 100 --random-output-len 100 --ignore-eos

• need to pre-install DeepGEMM

git clone https://github.com/deepseek-ai/DeepGEMM.git --recursive
cd DeepGEMM && git checkout bfe983c
python setup.py install

• support for vLLM V1 and torch.compile is done

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[artetaout]

hi, I've fixed lint and please give some review advice ! @mgoin @robertgshaw2-redhat @tlrmchlsmth

[mgoin]

Could you move this import into the conditional in apply_w8a8_block_fp8_linear?

[artetaout]

OK, I will fix

[artetaout]

done and commit is squashed

[mgoin]

Make these names more specific e.g. dg_supported_dtype.
Also doesn't deepgemm only support Hopper GPUs? So we should be checking compute capability

[artetaout]

OK, I will fix

[artetaout]

done and squashed

[mgoin]

currently ONLY support vLLM V0

Why is this the case and could you support V1? We will deprecate V0 soon

For your benchmarking, what GPU did you use?

[artetaout]

currently ONLY support vLLM V0

Why is this the case and could you support V1? We will deprecate V0 soon

For your benchmarking, what GPU did you use?

1. Currently, It does not support torch.compile. But we will support V1 and torch.compile later
2. use H20

[artetaout]

currently ONLY support vLLM V0

Why is this the case and could you support V1? We will deprecate V0 soon

For your benchmarking, what GPU did you use?

Support V1 is done, and result for performance and accuracy is updated !

Besides, some comments on code are also fixed !

[bnellnm]

Can you explain how this method is different than deep_gemm.gemm_fp8_fp8_bf16_nt? There's a lot of lower level code copied from DeepGemm but it's not immediately obvious why it is necessary.

[artetaout]

Can you explain how this method is different than deep_gemm.gemm_fp8_fp8_bf16_nt? There's a lot of lower level code copied from DeepGemm but it's not immediately obvious why it is necessary.

In sglang, It provides command for pre-compile DeepGEMM, to save time during start stage.
And since now this commit does not plan to support pre-compile commands, so I've removed them.
Support for vllm pre-compile will be in another commit .

[shixianc]

Does this deepgemm kernel only work for block_quant (fine-grained) fp8 models? Can it work with general per-tensor/channel quant fp8 models?

[artetaout]

Does this deepgemm kernel only work for block_quant (fine-grained) fp8 models? Can it work with general per-tensor/channel quant fp8 models?

I've never try a channel/tensor-wise FP8 model, maybe you can provide one or two.

[shixianc]

@artetaout
I'm testing on our own model that quantized with AutoFP8 (per-tensor fp8)
Here's one on HF: RedHatAI/Mixtral-8x7B-Instruct-v0.1-AutoFP8

However I were looking at the deepgemm code https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/jit_kernels/gemm.py#L167-L170 it seems the weight scale tensors must be in per-block/group. Does it mean it won't work if the input weight scales are per-tensor?
I meant we could try expanding our per-tensor weight scales to the required shape, but do you think it still provide speed improvement by DeepGEMM itself?

[artetaout]

@artetaout I'm testing on our own model that quantized with AutoFP8 (per-tensor fp8) Here's one on HF: RedHatAI/Mixtral-8x7B-Instruct-v0.1-AutoFP8

However I were looking at the deepgemm code https://github.com/deepseek-ai/DeepGEMM/blob/main/deep_gemm/jit_kernels/gemm.py#L167-L170 it seems the weight scale tensors must be in per-block/group. Does it mean it won't work if the input weight scales are per-tensor? I meant we could try expanding our per-tensor weight scales to the required shape, but do you think it still provide speed improvement by DeepGEMM itself?

i think it will.

[artetaout]

hi reviewers! thx for review! but the CI seems to be failed, but the failure has nothing to do with current deepgemm
logic.... i wonder if i've missed something ? or please tell me what to do next @mgoin @youkaichao @bnellnm

[mgoin]

Nice simplification work, this looks much better! Just one nit and I'll help you look at the CI failures. We have some known failures

[artetaout]

Nice simplification work, this looks much better! Just one nit and I'll help you look at the CI failures. We have some known failures

fixed, and now it seems to be pending...

[artetaout]

hi reviewers , is there anything i need to do more? @mgoin @shixianc

[mgoin]

LGTM, thank you for the contribution! We should aim to document this and possibly make a script/tool for installing deepgeem to use this in the future, similar to nixl (https://github.com/vllm-project/vllm/blob/da9b523ce1fd5c27bfd18921ba0388bf2e8e4618/tools/install_nixl.sh)

[artetaout]

LGTM, thank you for the contribution! We should aim to document this and possibly make a script/tool for installing deepgeem to use this in the future, similar to nixl (https://github.com/vllm-project/vllm/blob/da9b523ce1fd5c27bfd18921ba0388bf2e8e4618/tools/install_nixl.sh)

the document about DeepGEMM installation will be release soon... by my colleague @HanHan009527