Upgrade cudnn_frontend to 1.24 and enable cuDNN SDPA for MHA/GQA

[tianleiwu]

Description

Upgrades the cudnn_frontend dependency from 1.12.0 → 1.24.0 and wires the updated cuDNN SDPA (scaled dot-product attention) kernels into the CUDA MultiHeadAttention and GroupQueryAttention operators. On SM≥90 (Hopper/Blackwell), cuDNN SDPA is auto-preferred for FP16/BF16 ahead of Flash Attention / cutlass FMHA, which significantly improves GQA prefill throughput.

Key Changes

Area	Change
Dependency	cmake/deps.txt: cudnn_frontend 1.12.0 → 1.24.0.
Build	cmake/external/cudnn_frontend.cmake: mark cudnn_frontend headers as SYSTEM includes so v1.24's unused static helper does not trip -Werror=unused-function.
SDPA wrapper	cudnn_fmha/cudnn_flash_attention.cc: migrate to the v1.24 API — set_generate_stats(false) (replaces deprecated set_is_inference), diagonal-band causal masking (set_diagonal_alignment + set_diagonal_band_right_bound/set_diagonal_band_left_bound), and synthesize the missing seq_len_q/seq_len_kv side that v1.24 now requires when a padding mask is used.
MHA	multihead_attention.{cc,h}: enable cuDNN SDPA for FP16 and BF16; compute cuDNN eligibility before Flash and prefer it on SM≥90 unless the user pinned a kernel.
GQA	group_query_attention.{cc,h}, group_query_attention_impl.cu, attention_data.h: add a cuDNN SDPA path (non-quantized FP16/BF16, no softcap/smooth-softmax/head-sink/local-window, BNSH KV cache), dispatched after XQA and before Flash/MEA/unfused.
Kernel selection	attention_kernel_options.{cc,h}: track explicit sdpa_kernel selection and honor an explicit ORT_ENABLE_CUDNN_FLASH_ATTENTION=0 so it disables the SM≥90 auto path.

Kernel Priority

• SM≥90, FP16/BF16: cuDNN SDPA is auto-preferred unless the user explicitly selects a kernel via the sdpa_kernel provider option or sets ORT_ENABLE_CUDNN_FLASH_ATTENTION=0.
• GQA decode: XQA remains highest priority where eligible; cuDNN SDPA outranks Flash/MEA/unfused for the remaining eligible cases.
• ORT_ENABLE_CUDNN_FLASH_ATTENTION=0 disables cuDNN entirely (including the auto path); =1 force-enables it; the sdpa_kernel provider option overrides env vars.

Benchmark Results

Measured with onnxruntime/test/python/transformers/benchmark_gqa.py on NVIDIA H200 (SM 9.0), CUDA 13.0 / cuDNN 9.19, Llama3-8B-shaped GQA (b1, 32 query heads, 8 KV heads, head size 128, FP16).

• Baseline = Flash Attention (ORT_ENABLE_CUDNN_FLASH_ATTENTION=0)
• This PR = cuDNN SDPA (default on SM≥90)

The prefill (prompt) phase shows the largest gains for the dense variants:

ORT-GQA-Dense — prompt latency (ms, lower is better)

seq_len	Baseline (Flash)	This PR (cuDNN)	Speedup
64	0.082	0.054	1.52×
128	0.186	0.056	3.33×
256	0.184	0.065	2.83×
512	0.247	0.074	3.32×
1024	0.295	0.122	2.42×
2048	0.681	0.250	2.72×
4096	1.294	0.699	1.85×
8192	3.864	1.256	3.08×

ORT-GQA-Dense-PackedQKV — prompt latency (ms, lower is better)

seq_len	Baseline (Flash)	This PR (cuDNN)	Speedup
64	0.197	0.053	3.69×
128	0.160	0.055	2.93×
256	0.213	0.060	3.53×
512	0.226	0.073	3.09×
1024	0.333	0.291	1.15×
2048	0.595	0.252	2.36×
4096	1.312	0.697	1.88×
8192	5.014	1.259	3.98×

Prefill is ~1.5×–4× faster across sequence lengths for both dense variants. Decode (token) latency is unchanged within run-to-run noise.

Testing

• onnxruntime_provider_test --gtest_filter='GroupQueryAttentionTest.*:MultiHeadAttentionTest.*' — GQA 44/44, MHA 18/18 pass on H200.
• Broader attention regression (AttentionTest.*:PackedMultiHeadAttentionTest.*:DecoderMaskedMultiHeadAttentionTest.*) — 143/143 pass.
• Verified the new path on cuDNN 9.8 and 9.19 (decode s_q==1 causal-mask edge case handled for cuDNN ≤ 9.9).
• Verified ORT_ENABLE_CUDNN_FLASH_ATTENTION=0 disables the SM≥90 auto path (0 cuDNN selections) while the default run selects cuDNN.

[Copilot]

Pull request overview

This PR upgrades the cudnn_frontend dependency and integrates the newer cuDNN SDPA backend into the CUDA contrib MultiHeadAttention (MHA) and GroupQueryAttention (GQA) operators, including an SM≥90 auto-preference path to improve Hopper/Blackwell attention throughput.

Changes:

• Bump cudnn_frontend from 1.12.0 → 1.24.0 and adjust build to treat its headers as SYSTEM includes.
• Extend MHA to enable cuDNN SDPA for FP16/BF16 and optionally auto-prefer it on SM≥90 ahead of Flash Attention.
• Add a cuDNN SDPA execution path for GQA (non-quantized FP16/BF16, constrained feature set) and update kernel option handling to track explicit sdpa_kernel selection / disable SM≥90 auto when env-var disables cuDNN.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
onnxruntime/contrib_ops/cuda/bert/multihead_attention.h	Adds state to support SM≥90 auto-preference of cuDNN SDPA when not kernel-pinned.
onnxruntime/contrib_ops/cuda/bert/multihead_attention.cc	Computes cuDNN SDPA eligibility earlier and updates selection priority (esp. SM≥90); enables BF16.
onnxruntime/contrib_ops/cuda/bert/group_query_attention.h	Adds flags for explicit vs auto cuDNN SDPA enablement.
onnxruntime/contrib_ops/cuda/bert/group_query_attention.cc	Introduces cuDNN SDPA path selection/dispatch and wires temp allocator + cuDNN handle.
onnxruntime/contrib_ops/cuda/bert/group_query_attention_impl.cu	Implements the cuDNN SDPA runner path for GQA and hooks it into the dispatch.
onnxruntime/contrib_ops/cuda/bert/cudnn_fmha/cudnn_flash_attention.cc	Migrates to cudnn_frontend v1.24 APIs; adds diagonal-band causal mask logic and mask synthesis.
onnxruntime/contrib_ops/cuda/bert/attention_kernel_options.h	Tracks explicit sdpa_kernel selection and adds a gate for SM≥90 cuDNN auto-preference.
onnxruntime/contrib_ops/cuda/bert/attention_kernel_options.cc	Implements explicit-kernel tracking and honors ORT_ENABLE_CUDNN_FLASH_ATTENTION=0 for disabling auto.
onnxruntime/contrib_ops/cuda/bert/attention_impl.cu	Enables BF16 flagging for the cuDNN SDPA wrapper.
onnxruntime/contrib_ops/cuda/bert/attention_data.h	Extends GQA data to carry cuDNN SDPA selection and required runtime handles.
cmake/onnxruntime_providers_cuda_plugin.cmake	Links CUDA::nvrtc for the CUDA plugin EP to satisfy cudnn_frontend NVRTC symbol needs.
cmake/external/cudnn_frontend.cmake	Marks cudnn_frontend headers as SYSTEM to avoid -Werror failures from upstream warnings.
cmake/deps.txt	Updates cudnn_frontend archive URL/hash to v1.24.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[titaiwangms]

Review summary — cuDNN SDPA for MHA/GQA (cudnn_frontend 1.24)

Reviewed by a 5-model team (readability, code, critical, deep-semantic, integration). Strong PR, well-reasoned. The causal-mask and GQA capacity/padding math was independently verified against the cudnn_frontend source and found mathematically correct. One bug to fix before merge.

🔴 Major — fix before merge

1. MHA auto-enable path bypasses is_stable() → silent wrong results on cuDNN 9.10.0/9.10.1
(flagged by 3 of 5 reviewers)

In multihead_attention.cc, cudnn_sdpa_supported checks is_supported() but never calls cudnn_sdpa::is_stable(). The explicit env path is safe (UseCudnnFlashAttention() is already cleared by is_stable() in attention_kernel_options.cc), but the new auto path (auto_enable_cudnn_flash_attention_ && sm>=90) is not covered. On Hopper/Blackwell with cuDNN 9.10.0/9.10.1 and no env var set, MHA will auto-select cuDNN SDPA — exactly the versions this PR documents as buggy. GQA does this correctly (group_query_attention.cc calls is_stable() in its chain); MHA is the inconsistency.
Fix: add && onnxruntime::cudnn_sdpa::is_stable() to MHA's cudnn_sdpa_supported.

🟢 Minor

2. Missing tests — no coverage for the new auto-priority, env override (ORT_ENABLE_CUDNN_FLASH_ATTENTION=0), explicit sdpa_kernel pin, or seq-len synthesis paths. Suggested homes: attention_kernel_options_test.cc, multihead_attention_op_test.cc.
3. cudnn_sdpa_bias_ok over-conservative for s_q==1 — decode drops causal, so bias is fine; the current guard needlessly disables cuDNN for biased single-token decode. Consider || sequence_length == 1. (No correctness impact.)
4. Fill launch errors unchecked in CreateConstantSeqLenBuffer — returns void; consider returning Status + checking cudaGetLastError().
5. set_generate_stats(false) under 1.24 — please confirm the 1.24 build doesn't dereference an empty is_inference optional (CI build success should confirm).

❓ Open question

6. GQA non-first-prompt with s_q>1: seq_len_q is synthesized as the full sequence_length. This is correct only if continuation queries are never per-batch right-padded. Holds for standard GQA continuation — please confirm no variable-length query-batching path reaches here with padded queries.

⚪ Nits (optional polish)

• void* cudnn_handle could be a typed cudnnHandle_t (header already available transitively).
• Double-negative field disable_auto_cudnn_flash_attention_ behind the positive AllowCudnnFlashAttentionAuto().
• Three names for one concept: use_cudnn_sdpa / use_cudnn_flash_attention / AttentionKernel_CudnnFlashAttention.
• Stale "past_present_share_buffer" comment above the new cuDNN block in group_query_attention.cc.
• kIsFp16OrBf16 duplicated in MHA + GQA; CreateConstantSeqLenBuffer "Constant" is misleading in a CUDA context.

✅ Verified correct (no action)

Causal-mask migration (byte-identical to the old convenience setters), sliding-window mapping (no off-by-one), the is_bf16 fix (now required for BF16), kernel mutual-exclusion, env tri-state semantics, multi-file plumbing, CUDA-graph-safe Fill, and CMake/integration (nvrtc+cuda_driver, SYSTEM includes) are all clean.

A structural concern about GQA using cache capacity as the cuDNN KV length was raised and then resolved: with a padding mask present, cudnn_frontend's bottom-right diagonal re-anchors to the actual per-batch lengths, so masking the capacity tail with total_seq_lens is sound.

[tianleiwu]

Thanks for the thorough multi-model review. Responses below.

🔴 Major #1 — MHA auto path bypasses is_stable()

Good catch, fixed. Added && onnxruntime::cudnn_sdpa::is_stable() to cudnn_sdpa_supported in multihead_attention.cc, so the auto path (auto_enable_cudnn_flash_attention_ && sm>=90) is now gated identically to GQA. On cuDNN 9.10.0/9.10.1 the auto path will no longer select cuDNN SDPA.

🟢 Minor

2. Tests — agreed; I'll add coverage for auto-priority, the ORT_ENABLE_CUDNN_FLASH_ATTENTION=0 override, explicit sdpa_kernel pinning, and seq-len synthesis in a follow-up so this PR stays focused on the upgrade + integration.
3. cudnn_sdpa_bias_ok for s_q==1 — correct that biased single-token decode is currently disabled needlessly. Since there's no correctness impact, I'll fold the || sequence_length == 1 relaxation into the decode-perf follow-up.
4. Fill error checking — run() is void and the wider GQA/MHA dispatch path doesn't currently thread Status through here; a launch failure surfaces on the next stream sync. I'll leave the signature as-is for this PR to avoid a broad refactor, but noted for the cleanup pass.
5. set_generate_stats(false) under 1.24 — confirmed building against 1.24; the inference is_inference optional is set via set_is_inference(true) before stats, so it is not dereferenced empty. CI build success confirms.

❓ Open question #6 — GQA s_q>1 continuation

Confirmed: the synthesized seq_len_q = full sequence_length is correct for our GQA continuation path. ORT's GQA does not right-pad continuation queries per batch — query batching is left-aligned/contiguous — so no variable-length padded-query path reaches here.

⚪ Nits

• Removed the stale past_present_share_buffer comment above the cuDNN block in group_query_attention.cc.
• Typed cudnnHandle_t vs void* cudnn_handle, de-duplicating kIsFp16OrBf16, and the naming consolidation (use_cudnn_sdpa / use_cudnn_flash_attention) are reasonable; I'll batch these into the follow-up cleanup to keep this diff reviewable.

Thanks again — the causal-mask / capacity-padding verification is much appreciated.

[tianleiwu]

Correction on minor #5 above — I misspoke: the code calls set_generate_stats(false) directly, there is no set_is_inference(true) call. In cudnn_frontend 1.24, set_generate_stats(bool) sets the generate_stats optional, and the SDPA validation prefers generate_stats when present, only falling back to the legacy is_inference optional when generate_stats is unset. Since we set generate_stats explicitly, the empty is_inference optional is never dereferenced. The successful 1.24 build/run path confirms this.

[titaiwangms]

Thanks!