[CUDA] Add MatMulBlockQuantizedFp8Weight contrib op

[Tianlei Wu (tianleiwu)]

Description

Adds a new CUDA contrib operator MatMulBlockQuantizedFp8Weight (ms domain, opset 1): a weight-only block-quantized FP8 (E4M3) matmul.

Inputs / outputs

• A (T): float16/bfloat16 activation, shape [..., K]
• B (T1): float8e4m3fn weight, shape [N, K] (1 byte/value), block-scaled along K
• b_scale (T2): float32, shape [N, ceil(K/block_size)] (required)
• a_scale (T2, optional): float32 scalar. When present, A is statically quantized to FP8 (W8A8 numerics: a_deq = fp8(A/a_scale) * a_scale).
• bias (T, optional): shape [N]
• Y (T): float16/bfloat16, shape [..., N]
• Attribute block_size (default 128)

Type constraints: T = {float16, bfloat16}, T1 = {float8e4m3fn}, T2 = {float}.

Implementation

• GEMV fast path for small M (≤ 8, with K % 16 == 0 and block_size % 16 == 0).
• Otherwise: dequantize B into a scratch buffer, then cuBLAS GEMM (OP_T, OP_N), followed by optional bias add.
• Kernel registration guarded by !defined(DISABLE_FLOAT8_TYPES).

Tests

Adds matmul_block_scaled_fp8_test.cc with 4 GPU tests: FP16 weight-only GEMM, BF16 GEMM with scales + bias, multi-block GEMV decode, and the W8A8 activation-scale path. All 4 pass locally on an SM 12.0 GPU.

[Copilot]

Pull request overview

Adds a new CUDA contrib operator MatMulBlockQuantizedFp8Weight (Microsoft domain, opset 1) for weight-only block-scaled FP8 (E4M3) matmul, including schema registration, CUDA kernel implementation (GEMV fast path + dequant→cuBLAS GEMM fallback), tests, and documentation updates.

Changes:

• Introduces the MatMulBlockQuantizedFp8Weight schema (type/shape inference) and registers it in the MS opset.
• Implements the CUDA kernel (optional activation quant/dequant path, GEMV fast path for small M, otherwise dequantize weights + cuBLAS GEMM + optional bias).
• Adds C++ CUDA tests and updates contrib operator docs/kernel tables.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
onnxruntime/test/contrib_ops/matmul_block_scaled_fp8_test.cc	Adds CUDA tests for the new FP8 weight-only matmul operator.
onnxruntime/core/graph/contrib_ops/ms_opset.h	Registers the new operator schema in the Microsoft opset v1 list.
onnxruntime/core/graph/contrib_ops/contrib_defs.cc	Defines the operator schema, docs, type constraints, and shape inference.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.h	Declares the CUDA kernel class and CUDA launcher helpers.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cc	Implements validation, scratch allocation, GEMV dispatch, and cuBLAS fallback.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cu	Implements CUDA kernels for dequant, bias add, activation qdq, and GEMV fast path.
onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc	Registers the CUDA kernel with the contrib kernel registry (guarded on float8 types).
docs/OperatorKernels.md	Adds the operator to the kernel availability table.
docs/ContribOperators.md	Adds the operator to the contrib operator documentation index and section list.
docs/contrib_ops/cuda/matmul_block_scaled_fp8.md	Adds CUDA EP documentation (but currently references non-existent source files).
docs/contrib_ops/cuda/matmul_block_scaled_fp8_experiments.md	Adds experiment notes/benchmarks (contains a copy/paste command typo).

Comments suppressed due to low confidence (1)

onnxruntime/test/contrib_ops/matmul_block_scaled_fp8_test.cc:192

• If you add an inner #if !defined(DISABLE_FLOAT8_TYPES) && defined(ENABLE_FP8) guard for the FP8-only tests, it also needs a matching #endif before the existing #endif // USE_CUDA to keep preprocessor blocks balanced.

#endif  // USE_CUDA

[Justin Chu (justinchuby)]

Will this op be used directly in the model graph? I would personally prefer that we don't expose a specialized op like this in the model (and instead keep it an implementation detail) , unless it is strictly necessary?

[Justin Chu (justinchuby)]

Is there a model that needs this, out of curiosity?

[Tianlei Wu (tianleiwu)]

Is there a model that needs this, out of curiosity?

https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4