Attention: onnx conformance skip-list follow-ups for mode-3 (CUDA NOT_IMPLEMENTED) + latent fully-masked gaps

[titaiwangms]

⚙️ Status update (2026-06-12) — partially actioned ahead of pin-bump in #28958

Two items below were brought forward into #28958 (rather than waiting for the onnx pin to advance):

• Section A (mode-3 skips) — DONE (active now). The skip entries are promoted from TODO to active, gated by this issue:
  • onnx_backend_test_series_filters.jsonc: active regex ^test_attention_(23|24)_.*qk_matmul_output_mode3 (global).
  • TestCase.cc: the 6 exact mode-3 names scoped to the provider == "cuda" block (CPU implements mode-3 =0 / T1, so CPU conformance stays covered). #28994 / #27712 refs kept in both.
  • These are no-ops on the current v1.21.0 pin and become live when the pin passes [Android] Move add header files into AAR to using Gradle #8068 — i.e. Section A is already wired, no pin-bump edit needed for the skip itself.
• Section B (b2) Flash fully-masked guard — VERIFIED by tests. Added 3 Flash regression gtests (A100, SM>=8.0) that drive a structurally-empty leading-row case through FA2 and assert leading rows are exactly 0, not NaN:
  • …FlashStructuralEmptyRows_Zero_{FP16,BF16}_CUDA lock the single-tile epilogue (softmax.h:182-186).
  • …FlashStructuralEmptyRows_Zero_SplitKV_FP16_CUDA (total_seq=257 → num_splits=2, SM-independent) locks the split-KV combine (flash_fwd_kernel.h:1125/1135/1153).
  • Conclusion: FA2 emits 0 for fully-masked rows on both epilogues — no extra host-side guard needed; b2 is closed as "verified, no guard required."

Still open: Section A's unskip at pin-bump (Section C), and Section B (b1) float-mask predicate divergence (document-only / latent). The original tracker text is preserved below.

---

Tracks follow-up items left open by #28958 (ONNX-domain Attention bottom-right is_causal + fully-masked-row→0, implementing onnx/onnx#8068). None of these block #28958; they are deferred conformance / EP-gap items to action when the onnx submodule pin advances. This issue is the durable tracker for the CUDA qk_matmul_output mode-3 (post-softmax) capture gap.

Section A — Pin-bump action (HIGH)

When cmake/external/onnx is bumped past a release containing onnx/onnx#8068, the new node tests for qk_matmul_output mode 3 (post-softmax) will appear and hard-fail on CUDA, because CUDA Attention returns NOT_IMPLEMENTED for any qk_matmul_output_mode beyond kNone / kQK (i.e. kPostSoftCap, kPostMaskBias, kPostSoftMax). This capture gap is tracked by this issue (#28994).

• Ground: onnxruntime/core/providers/cuda/llm/attention.cc:1477 — "Only kNone and kQK output modes are supported in unified unfused attention."
• Action: at pin-bump, add the mode-3 conformance tests to BOTH skip lists, referencing this issue (Attention: onnx conformance skip-list follow-ups for mode-3 (CUDA NOT_IMPLEMENTED) + latent fully-masked gaps #28994):
  • onnxruntime/test/testdata/onnx_backend_test_series_filters.jsonc (Python backend series)
  • onnxruntime/test/onnx/TestCase.cc::GetBrokenTests() (C++ onnx_test_runner)
• Copy-paste regex (jsonc):

  ^test_attention_(23|24)_.*qk_matmul_output_mode3
  

  The ^... prefix also captures the _expanded (function-body) and EP-suffixed variants. The C++ side (TestCase.cc) uses bare test-name keys, not regex, so enumerate the matching names there.
• Caveat: the exact generated test names cannot be enumerated on the current v1.21.0 onnx pin (they ship with [Android] Move add header files into AAR to using Gradle #8068); confirm the names/regex against the [Android] Move add header files into AAR to using Gradle #8068 test set at bump time.
• Related closed work: Support softcap, softmax_precision in Attention(CUDA) #27712 (CUDA Attention softcap / softmax_precision, now implemented) — historical context only; it is not the tracker for this mode-3 capture gap.

Section B — Latent gaps (LOW, document-only; no test currently exercises them)

• (b1) Float-mask fully-masked predicate divergence. The "is this key masked?" predicate differs across implementations for native float masks: CPU uses != lowest_float, CUDA uses > -1e30f, and the onnx reference uses isneginf. These agree for bool / structural (nonpad/causal) / -inf masks — the only cases any current test hits. No test currently drives a fully-masked row produced by a native finite-float mask (softcap_neginf_mask only masks keys 4–5, never a whole row), so the divergence is latent. Action: add a targeted test if/when native-float-mask full-row masking becomes a supported scenario; otherwise document the intended predicate.
• (b2) Flash-attention fully-masked guard unverified. LaunchZeroFullyMaskedRows is wired only on the MEA / CUTLASS path; it is not confirmed in RunFlashAttention. fp32 routes to MEA (Flash is disabled for fp32), so only fp16/bf16 + Flash + fully-masked rows are currently untested. Action: confirm whether Flash can emit a fully-masked row under the [Android] Move add header files into AAR to using Gradle #8068 frontier and, if so, add the guard + a fp16 Flash test.

Section C — Unskip reminder

The bottom-right conformance skips added in #28958's G3 commit (onnx_backend_test_series_filters.jsonc + TestCase.cc, the #28904-tagged block) are no-ops on the current v1.21.0 pin. They must be revisited / removed when the onnx pin advances past #8068 and each kernel path is confirmed conformant. Link back to this issue from those blocks.

Note (out of scope for #28958): a pre-existing comment in onnx_backend_test_series_filters.jsonc (~line 52, from commit 93d31cfd8d4) also references the now-closed #27712 as a live tracker for the beyond-kQK (modes 1/2, softcap/bias) qk_matmul_output skips. It could be cleaned up separately.

Cross-links

• Fix Attention is_causal causal-mask alignment + composed is_causal/attn_mask NaN robustness for external (static) KV cache (#8054) onnx/onnx#8068 — spec / errata + the new node tests
• Fix Attention is_causal bottom-right alignment for external KV cache (onnx#8068, #28904) #28958 — the kernel PR these follow-ups derive from
• Support softcap, softmax_precision in Attention(CUDA) #27712 — CUDA Attention softcap / softmax_precision (CLOSED, now implemented); historical context only, not the mode-3 tracker
• Fix Attention op softcap ordering: apply before mask/bias onnx/onnx#7867 — Attention softcap-ordering fix (softcap before mask/bias); referenced for context, no action

[microsoft-github-policy-service]

Applying stale label due to no activity in 30 days. Please react to the issue by assigning it or labeling it as 'contributions welcome', 'documentation', 'feature request', 'regression', or 'no stale'.