Add benchmark for DeepGEMM and vLLM Block FP8 Dense GEMM

[mgoin]

python benchmark_fp8_block_dense_gemm_table.py
INFO 02-26 21:55:13 [__init__.py:207] Automatically detected platform cuda.
===== STARTING FP8 GEMM BENCHMARK =====
PyTorch version: 2.5.1+cu124
CUDA version: 12.4
Triton version: 3.1.0
Using device: NVIDIA H100 80GB HBM3

===== PERFORMANCE COMPARISON =====

DeepGEMM Implementation:
+------+-------+-------+-----------+--------+--------+
| m    | n     | k     | Time (μs) | TFLOPS | GB/s   |
+------+-------+-------+-----------+--------+--------+
|    8 |  4096 |  7168 | 102.9     | 4.6    | 286.4  |
|    8 |  7168 | 18432 | 70.8      | 29.8   | 1868.8 |
|    8 | 18432 |  7168 | 69.3      | 30.5   | 1911.8 |
|   64 |  4096 |  7168 | 69.1      | 54.4   | 439.0  |
|   64 |  7168 | 18432 | 69.4      | 243.6  | 1933.6 |
|   64 | 18432 |  7168 | 70.4      | 240.3  | 1917.2 |
|   64 | 24576 |  1536 | 70.1      | 68.9   | 584.6  |
|   64 | 32768 |   512 | 68.4      | 31.4   | 307.1  |
|   64 |  7168 | 16384 | 69.5      | 216.3  | 1718.5 |
|  128 |  4096 |  7168 | 141.1     | 53.3   | 222.1  |
|  128 |  7168 | 18432 | 71.9      | 470.5  | 1896.1 |
|  128 | 18432 |  7168 | 69.3      | 488.2  | 1988.2 |
| 1024 |  4096 |  7168 | 89.7      | 670.1  | 502.5  |
| 1024 | 18432 |  7168 | 279.0     | 969.8  | 635.2  |
| 2048 |  4096 |  7168 | 175.1     | 687.0  | 347.4  |
| 4096 |  4096 |  7168 | 335.4     | 717.0  | 275.1  |
+------+-------+-------+-----------+--------+--------+

vLLM Triton Implementation:
+------+-------+-------+-----------+--------+--------+--------------+
| m    | n     | k     | Time (μs) | TFLOPS | GB/s   | vs DeepGEMM  |
+------+-------+-------+-----------+--------+--------+--------------+
|    8 |  4096 |  7168 | 74.0      | 6.3    | 398.2  | 1.39x faster |
|    8 |  7168 | 18432 | 89.6      | 23.6   | 1478.1 | 0.79x slower |
|    8 | 18432 |  7168 | 113.2     | 18.7   | 1170.4 | 0.61x slower |
|   64 |  4096 |  7168 | 79.4      | 47.3   | 382.2  | 0.87x slower |
|   64 |  7168 | 18432 | 98.5      | 171.7  | 1363.0 | 0.70x slower |
|   64 | 18432 |  7168 | 119.5     | 141.5  | 1129.4 | 0.59x slower |
|   64 | 24576 |  1536 | 37.6      | 128.4  | 1089.7 | 1.86x faster |
|   64 | 32768 |   512 | 38.7      | 55.5   | 542.6  | 1.77x faster |
|   64 |  7168 | 16384 | 86.1      | 174.5  | 1386.4 | 0.81x slower |
|  128 |  4096 |  7168 | 90.7      | 82.9   | 345.4  | 1.56x faster |
|  128 |  7168 | 18432 | 144.0     | 234.9  | 946.9  | 0.50x slower |
|  128 | 18432 |  7168 | 229.5     | 147.4  | 600.1  | 0.30x slower |
| 1024 |  4096 |  7168 | 242.3     | 248.2  | 186.1  | 0.37x slower |
| 1024 | 18432 |  7168 | 897.8     | 301.4  | 197.4  | 0.31x slower |
| 2048 |  4096 |  7168 | 463.0     | 259.7  | 131.4  | 0.38x slower |
| 4096 |  4096 |  7168 | 901.8     | 266.7  | 102.3  | 0.37x slower |
+------+-------+-------+-----------+--------+--------+--------------+

vLLM CUTLASS Implementation:
+------+-------+-------+-----------+--------+--------+--------------+--------------+
| m    | n     | k     | Time (μs) | TFLOPS | GB/s   | vs DeepGEMM  | vs Triton    |
+------+-------+-------+-----------+--------+--------+--------------+--------------+
|    8 |  4096 |  7168 | 34.6      | 13.6   | 852.3  | 2.98x faster | 2.14x faster |
|    8 |  7168 | 18432 | 78.9      | 26.8   | 1677.3 | 0.90x slower | 1.13x faster |
|    8 | 18432 |  7168 | 81.2      | 26.0   | 1631.1 | 0.85x slower | 1.39x faster |
|   64 |  4096 |  7168 | 36.9      | 101.9  | 822.9  | 1.87x faster | 2.15x faster |
|   64 |  7168 | 18432 | 87.4      | 193.4  | 1535.2 | 0.79x slower | 1.13x faster |
|   64 | 18432 |  7168 | 85.0      | 199.0  | 1587.6 | 0.83x slower | 1.41x faster |
|   64 | 24576 |  1536 | 28.0      | 172.8  | 1465.8 | 2.51x faster | 1.35x faster |
|   64 | 32768 |   512 | 28.8      | 74.5   | 728.5  | 2.37x faster | 1.34x faster |
|   64 |  7168 | 16384 | 77.9      | 193.0  | 1532.8 | 0.89x slower | 1.11x faster |
|  128 |  4096 |  7168 | 39.1      | 192.4  | 802.0  | 3.61x faster | 2.32x faster |
|  128 |  7168 | 18432 | 93.7      | 360.8  | 1454.2 | 0.77x slower | 1.54x faster |
|  128 | 18432 |  7168 | 85.7      | 394.8  | 1608.0 | 0.81x slower | 2.68x faster |
| 1024 |  4096 |  7168 | 99.7      | 603.1  | 452.2  | 0.90x slower | 2.43x faster |
| 1024 | 18432 |  7168 | 331.3     | 816.7  | 534.9  | 0.84x slower | 2.71x faster |
| 2048 |  4096 |  7168 | 198.3     | 606.6  | 306.7  | 0.88x slower | 2.34x faster |
| 4096 |  4096 |  7168 | 392.2     | 613.2  | 235.3  | 0.86x slower | 2.30x faster |
+------+-------+-------+-----------+--------+--------+--------------+--------------+

===== AVERAGE PERFORMANCE =====
+----------------+------------+----------+---------------+
| Implementation | Avg TFLOPS | Avg GB/s | Avg Time (ms) |
+----------------+------------+----------+---------------+
| DeepGEMM       | 310.98     | 1052.10  | 0.11          |
| vLLM Triton    | 144.30     | 715.60   | 0.23          |
| vLLM CUTLASS   | 286.78     | 1076.67  | 0.11          |
+----------------+------------+----------+---------------+

===== AVERAGE SPEEDUPS =====
+-----------------------------+--------------+
| Comparison                  | Speedup      |
+-----------------------------+--------------+
| DeepGEMM vs vLLM Triton     | 1.71x faster |
| DeepGEMM vs vLLM CUTLASS    | 0.94x slower |
| vLLM CUTLASS vs vLLM Triton | 1.84x faster |
+-----------------------------+--------------+

===== ACCURACY COMPARISON =====
+----------------+-----------------------+
| Implementation | Avg Diff vs Reference |
+----------------+-----------------------+
| DeepGEMM       | 0.000684              |
| vLLM Triton    | 0.000684              |
| vLLM CUTLASS   | 0.000684              |
+----------------+-----------------------+

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[houseroad]

Does the number reported match the numbers reported in their repo?

[mgoin]

Does the number reported match the numbers reported in their repo?

It seems DeepGEMM performance is very limited in performance to the shapes they support. I am looking at this

[benchislett]

Is it possible to reproduce the setup and configuration they used in their benchmark?

We test all shapes potentially used in DeepSeek-V3/R1 inference (including both prefilling and decoding, but without tensor parallelism) on H800 SXM5 with NVCC 12.8. All speedup metrics are calculated in comparison to our internally and carefully optimized implementation based on CUTLASS 3.6.

The summary claims the following for one of the sample shapes M=128, N=4096, K=7168:

533 TFLOPS | 2221 GB/s | 2.0x

whereas this benchmark has the following result:

=== Benchmarking shape: m=128, n=4096, k=7168 ===
Running correctness check...
DeepGEMM vs Reference difference: 0.000682
vLLM Triton vs Reference difference: 0.000682
vLLM CUTLASS vs Reference difference: 0.000682
vLLM Triton vs DeepGEMM difference: 0.000007
vLLM CUTLASS vs DeepGEMM difference: 0.000007
DeepGEMM: 0.109 ms, 68.76 TFLOPS
vLLM Triton: 0.091 ms, 82.65 TFLOPS
vLLM CUTLASS: 0.039 ms, 190.49 TFLOPS
DeepGEMM is 0.83x slower than vLLM Triton
DeepGEMM is 0.36x slower than vLLM CUTLASS
vLLM CUTLASS is 2.30x faster than vLLM Triton

It seems there might be some inconsistency between the benchmarking setups here...

[LucasWilkinson]

do you know how slow get_col_major_tma_aligned_tensor is? we can probably update per_token_group_quant_fp8 to handle this (if we do then I can also vectorize the scale loads in the cutlass implementation)

[mgoin]

I made a prettier script and updated the table above. The numbers will look a bit worse than their results because I am including the quantization of the input tensor, where they exclude this. I was able to improve their results a decent bit by using out existing triton kernel for input quantization, however it is still behind cutlass except for large M. The larger issue I face is I see many failures to compile for the shapes that they report on.

For instance:

=== Benchmarking shape: m=4096, n=7168, k=16384 ===
Running correctness check...
/home/mgoin/.deep_gemm/cache/kernel.gemm_fp8_fp8_bf16_nt.25e0f4716b93/kernel.cu:8:10: fatal error: cutlass/cutlass.h: No such file or directory
    8 | #include "cutlass/cutlass.h"
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
Traceback (most recent call last):
  File "/home/mgoin/code/vllm/benchmarks/kernels/deepgemm/benchmark_fp8_block_dense_gemm_table.py", line 396, in <module>
    run_benchmarks(verbose=True)
  File "/home/mgoin/code/vllm/benchmarks/kernels/deepgemm/benchmark_fp8_block_dense_gemm_table.py", line 269, in run_benchmarks
    result = benchmark_shape(m, n, k, verbose=verbose)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgoin/code/vllm/benchmarks/kernels/deepgemm/benchmark_fp8_block_dense_gemm_table.py", line 109, in benchmark_shape
    C_deepgemm = deepgemm_gemm()
                 ^^^^^^^^^^^^^^^
  File "/home/mgoin/code/vllm/benchmarks/kernels/deepgemm/benchmark_fp8_block_dense_gemm_table.py", line 79, in deepgemm_gemm
    deep_gemm.gemm_fp8_fp8_bf16_nt((A_deepgemm, A_scale_aligned),
  File "/home/mgoin/code/vllm/benchmarks/kernels/deepgemm/DeepGEMM/deep_gemm/jit_kernels/gemm.py", line 156, in gemm_fp8_fp8_bf16_nt
    runtime = jit_tuner.compile_and_tune(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgoin/code/vllm/benchmarks/kernels/deepgemm/DeepGEMM/deep_gemm/jit_kernels/tuner.py", line 40, in compile_and_tune
    kernels.append((build(name, arg_defs, code), tuned_keys))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgoin/code/vllm/benchmarks/kernels/deepgemm/DeepGEMM/deep_gemm/jit/compiler.py", line 139, in build
    assert subprocess.check_call(command) == 0, f'Failed to compile {src_path}'
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mgoin/.local/share/uv/python/cpython-3.12.4-linux-x86_64-gnu/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/cuda-12.5/bin/nvcc', '/home/mgoin/.deep_gemm/cache/kernel.gemm_fp8_fp8_bf16_nt.25e0f4716b93/kernel.cu', '-o', '/home/mgoin/.deep_gemm/tmp/nvcc.tmp.6d4720bb-9e87-4f2b-bfae-dd365f216cf6.82df462b59a9.so', '-std=c++17', '-shared', '-O3', '--expt-relaxed-constexpr', '--expt-extended-lambda', '-gencode=arch=compute_90a,code=sm_90a', '--ptxas-options=--register-usage-level=10', '--diag-suppress=177,174,940', '--compiler-options=-fPIC,-O3,-Wno-deprecated-declarations,-Wno-abi', '-I/home/mgoin/code/vllm/benchmarks/kernels/deepgemm/DeepGEMM/deep_gemm/jit/../include']' returned non-zero exit status 1.

The cutlass error is strange since several other shapes in this file work fine.

[mgoin]

@houseroad @LucasWilkinson @benchislett please check the latest results. For specific shapes it seems I am able to get close to their results

[houseroad]

Wondering if we can try more shapes provided from their side. Also curious about the Grouped GEMM comparison?

[LucasWilkinson]

Hmmm ya this is a bit inconclusive/underwhelming given that the CUTLASS blockwise kernels haven't really be tuned yet, with the exception of #12978 (which was actually targeted at H800 not H100), we use the same M tile size for everything. I was hoping it would be a bit more conclusive in either direction (but I guess that was was wishful thinking given these are gemms after all haha)

feels like this make its hard to say if its worth focusing more energy on DeepGEMM or tuning the blockwise kernels (and use smaller instruction shapes via transposing once this lands)

We'd probably have an easier time tuning DeepGEMM though since CUTLASS currently has some restrictions on tile size

[ProphetPeng]

Can you replace it with per_token_group_quant_fp8(A, block_size[1], column_major_scales=True)?

[mgoin]

This seems to make no difference

[youkaichao]

i think we can make this pr a draft (to save ci cost), since there's nothing we need to test in ci?

[benchislett]

How much effort would it be to plug this in and make some end-to-end benchmarks?

[mgoin]

@benchislett please see #13996

As noted here, microbenchmark performance is not good (except for very specific sizes) yet so we need to figure out how to fix this first.

[houseroad]

Btw, shall we land this benchmark scripts? We may reuse to expand to other kernel libraries.

[mgoin]

@houseroad Yes I think we should land this benchmark, please review!