Add doxygen generated website for the project#6
Closed
raymondxyang wants to merge 2 commits into
Closed
Conversation
Author
|
Redundant.. |
raymondxyang
changed the title
tmccrmck
pushed a commit
to tmccrmck/onnxruntime
that referenced
this pull request
Aug 28, 2019
Enable CI build in Vienna environment
faxu
pushed a commit
that referenced
this pull request
Oct 12, 2020
* Test re-using page layout from current ONNX Runtime website for docs * Add content for documentation on website * Fixed most broken links * Copy just-the-docs theme sources into repo * Remove local theme files as this did not work with GitHub * Remove nojekyll file * Move image assets into single location * Add Contents to markdown files and ensure only one h1 * Update after review * Fix img links * Add trailing slash to main nav links * Fix broken links on main docs page * Re-fix broken links on main docs page * Fix broken links #3 * Fix broken links #4 * Fix broken links #5 * Fix broken links #6 * Fix paths to global assets * Add updates since fork * Update custom op docs * Fix link
Merged
This was referenced Nov 1, 2021
carsonswope
added a commit
to boris-fx/onnxruntime
that referenced
this pull request
Feb 14, 2025
the previous commits: implement custom DML operator to perform sampling portion of DeformConv2D implement bfx::warp_flow and second_order_deform_alignment make mask and offset custom ops implement bfx grid_sample and upscale grid generator, for basicvsrpp upscaling change make upscale grid sample layer use 'x/y start lq' instead of 'x/y dims' include script to generate assembly manifest for onnxruntime.dll & DirectML.dll fix build in debug mode initial dml impl of rle encode, decode, using common prefix-sum primitive increment manifest version for ort dml disable DML implementation of NonZero ONNX op fixs from v1.18.1 merge, bump manifest version add script to build on linux (steel) add DirectML.Debug.dll to bfx windows build output compile dml grid sample hlsl using cs62 instead of cs50 for float_float types add windows build to jenkins, #0 add linux build to jenkins, microsoft#1 allow linux build scripts to be executable add linux build to jenkins, microsoft#2 add linux build to jenkins, microsoft#3 add linux build to jenkins, microsoft#4 add linux build to jenkins, microsoft#5 add linux build to jenkins, microsoft#6 add win build to jenkins, microsoft#2 win build to jenkins, microsoft#3 add win build to jenkins, microsoft#4 (switch back to vs2019) add win build to jenkins, microsoft#5 (use ninja cmake generator) add win build to jenkins, microsoft#6 (fix build paths after ninja change) add win build to jenkins, microsoft#7 (fix vs2019 build errors) extend assembly manifest to apply to cpu and cuda EPs, and not just dml
gedoensmax
pushed a commit
to gedoensmax/onnxruntime
that referenced
this pull request
Apr 22, 2025
Add support for python bindings of NV TensorRT RTX EP
Closed
titaiwangms
added a commit
to titaiwangms/onnxruntime
that referenced
this pull request
May 6, 2026
Tianlei BLOCKER microsoft#1: New mode-1+softcap differentiating test (C++ + Python). With softcap > 0 active, qk_matmul_output_mode=1 (post-microsoft#7913 numbering = kPostSoftCap) snapshots softcap*tanh(scale*QK/softcap) with NO mask added. Without softcap, mode 1 aliases mode 0, so the swap is observationally indistinguishable — this test is what proves the 1<->2 swap actually changed semantics correctly. Tianlei BLOCKER microsoft#2: New softcap+nonpad_kv_seqlen leakage test (C++ + Python). Exercises the latent fix where the nonpad sentinel is now applied AFTER softcap (per onnx#7867 ordering). Pre-fix: tanh squashed the sentinel, leaking poison V at padded positions through softmax. Bot inline minors: - microsoft#3 (test_gqa.py): clarify fp16 docstring — CPU does support fp16; fp32 is the natural EP-native dtype for the canary. - microsoft#4 (attention_op_test.cc): regen comment now cites shared opset 23/24 ordering and notes RunTest4D builds at opset 23. - microsoft#5 (attention_parameters.h): typo defintion -> definition. - microsoft#6 (attention.cc): replace 'guaranteed -inf' with precise wording citing mask_filter_value<T>() = numeric_limits::lowest() / MLFloat16::MinValue sentinel and the MLAS softmax finite-input requirement (attention.h). R-2 microsoft#1 (attention_parameters.h): Spec-leading documentation block on the QKMatMulOutputMode enum noting that ORT now uses the post-onnx#7913 numbering, while the bundled cmake/external/onnx (v1.21.0) still reflects the old numbering. ORT leads the spec change pending the next bundled-ONNX bump. Plumbing: common.py attention_prompt_func gains an optional output_qk kwarg (default 0 / disabled). When > 0, returns a 4-tuple including the qk_matmul snapshot tensor; otherwise unchanged 3-tuple. No existing callers are affected. Test results: - AttentionTest.* — 60/60 PASS (was 58, +2 new). - TestONNXAttentionCPUSoftcapMaskOrdering — 4/4 PASS (was 2, +2 new). - lintrunner clean across all 5 touched files. Refs: lead-39245992/upstream-pr-status-recheck.md, pr1v2-review-{code,critical,readability,qa}.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
titaiwangms
added a commit
that referenced
this pull request
Jun 19, 2026
…(onnx#8068, #28904) (#28958) ## Summary Aligns the opset-24 ONNX-domain `Attention` kernels (CPU + CUDA) with the ONNX errata onnx/onnx#8068 (tracking RFC onnx/onnx#8054) for the **external static KV-cache** path — keyed by `nonpad_kv_seqlen` (input #6), no `past_key`. **Addresses #28904.** ## What changed 1. **Bottom-right `is_causal` alignment.** Per batch, `offset[b] = nonpad_kv_seqlen[b] − q_sequence_length`; a query at in-block index `i` attends key `j` iff `j <= i + offset[b]`. Applied on CPU and on the CUDA Flash / Memory-Efficient (MEA) / unfused paths. The MEA causal-alignment selector is now offset-aware (no unconditional top-left when an external cache is present); Flash's native bottom-right + per-batch `seqlens_k` is used where eligible. 2. **Fully-masked-row → 0 guard (Bug-2).** A query row with no allowed key now outputs a **zero** row instead of mean-of-V (the finite-sentinel softmax result). Detected with an exact per-key structural predicate (`isneginf`-equivalent) and zeroed with **select (not multiply)** before `P @ V`, so `0 @ V = 0`. Added on CPU and the CUDA MEA path. The Flash `is_causal` + `seqlens_k` path (`offset >= 0`) cannot produce a fully-masked row and is intentionally left unguarded. Bool-mask conversion was already select-not-multiply on both EPs (Bug-1 satisfied; no change needed). 3. **Reject removal.** Removes the CUDA `NOT_IMPLEMENTED` reject for `is_causal` + `nonpad_kv_seqlen` with `S_q != total_kv` and no `past_key` — the spec now *defines* this result, so the op computes it rather than rejecting. Full-prefill (`offset = 0`) and `past_key` decode paths remain **bit-identical**. Contrib `MultiHeadAttention` / `GroupQueryAttention` consume the shared FMHA kernels and are **unchanged** — only the ONNX-domain `Attention` dispatch is retargeted. ## Test coverage - **C++ `AttentionTest` gtests: 73/73 pass**, including new bottom-right-offset, structural-empty causal row → 0 (CPU + CUDA), and fp16 fully-masked-row goldens. - **Python `test_onnx_attention`: 277/0** — includes the updated `test_tensorscatter_attention.py` (stale negative-reject → positive bottom-right acceptance). - QA final gate: from-scratch Debug build green. ## Preemptive onnx#8068 node-test skips (de-skip TODO) This branch adds the new onnx/onnx#8068 `Attention` backend node tests to both skip lists so they don't fail before the onnx dependency is bumped: - `onnxruntime/test/testdata/onnx_backend_test_series_filters.jsonc` - `onnxruntime/test/onnx/TestCase.cc` (C++ `GetBrokenTests`) These are a **no-op on the current onnx pin (v1.21.0)**. **TODO (de-skip):** remove **both** skip lists once `cmake/external/onnx` is bumped to a release containing onnx#8068. ## Deferred follow-up `q_seq > 1` Python bottom-right **parity** coverage requires upgrading the `test_onnx_attention` suite's numpy/torch reference functions from **total-kv-relative** causal (`offset = kv_seq − q_seq`) to **nonpad-relative** bottom-right (`offset = nonpad_kv_seqlen − q_seq`); a naive `is_causal=1` flip on the current refs is a no-op or a false failure against the correct kernel. The `q_seq > 1` / `nonpad < q_seq` behavior (including structural-empty rows) is already locked by the C++ gtest goldens. Tracked as follow-up. ## References - onnx/onnx#8068 — spec + reference errata (bottom-right `is_causal` on the `nonpad_kv_seqlen`/no-`past_key` path + composed `is_causal` + `attn_mask` NaN robustness). Separately pushed, CI green, awaiting SIG review. - onnx/onnx#8054 — RFC: offset-aware causal masking for KV-cache decode / chunked prefill. --------- Signed-off-by: Ti-Tai Wang <titaiwang@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
qjia7
added a commit
to qjia7/onnxruntime
that referenced
this pull request
Jun 25, 2026
In graph-capture mode the host total_sequence_length scalar is 0 and the dispatch grid for the flash-attention pipeline is computed on the GPU. The three shaders that prepare or consume the indirect-dispatch buffer (CopyKVCache, SplitPackedQKVWithRotaryEmbeddingAndCopyKV, FlashAttentionDecodeQKV) previously sized the grid from seqlens_k[batch=0] + 1. For batched right-padded prefill, batch 0 is not guaranteed to hold the maximum KV span, so when the spread across the batch crosses a tile boundary other batches lose tiles and produce wrong output. Thread GQA's input microsoft#6 (total_sequence_length, GPU-resident exactly when graph capture is enabled) through ApplyFlashAttention into the three shaders and use it for the indirect-dispatch sizing only. Per-batch seqlens_k[batch] + 1 still drives causal masking and per-batch bounds inside the kernels. Also enforce in GroupQueryAttention that graph capture implies past_present_share_buffer_, so the use_indirect_dispatch predicate only needs to check seqlen_k, total_seqlen, and IsGraphCaptureEnabled. Address PR review: - Clamp attention_bias load to offset_base + stride_total_seq - 1u in both scalar and vec4 paths so the one-past-end fallback stays within the same row. - Reword the smooth_softmax test comment to reference the outer gating in GroupQueryAttention::ComputeInternal that routes through ApplyAttention. - Extend the indirect-dispatch fix to FlashAttentionDecodeQKV; the new use of use_indirect_dispatch_ also resolves the -Wunused-private-field Clang error on the wasm and arm64 builds. Add BatchedRightPaddedRotaryPrefillFlashAttentionLargeSpread_WebGPU with real_lens spread > tile_size so a future regression in the dispatch sizing surfaces in the WebGPU test suite (graph capture itself cannot be toggled from OpTester).
6 tasks
qjia7
added a commit
that referenced
this pull request
Jun 26, 2026
…ts (#29247) ## Summary Lift WebGPU FlashAttention's `batch_size == 1` restriction so batched GQA with right-padded prompts (the common GenAI batched-prefill shape) takes the fused FlashAttention path instead of falling back to `ApplyAttention`. - **Per-batch seqlens in FlashAttention shaders.** Prefill, decode split-reduce, CopyKVCache, and the fused rotary-and-copyKV template now read `seqlens_k[batch_idx]` instead of hardcoding `seqlens_k[0]`. All `past_X = total_X - new_X` subtractions are clamped to avoid u32 underflow when a short batch's per-batch total is less than the batch-wide `sequence_length`. - **Indirect-dispatch sizing uses GQA's `total_sequence_length` input.** `CopyKVCache`, `SplitPackedQKVWithRotaryEmbeddingAndCopyKV`, and `FlashAttentionDecodeQKV` now take a new `total_sequence_length_input` binding (GQA input #6, GPU-resident under graph capture) for the indirect-dispatch grid sizing. This is the global max KV span across the batch by construction, replacing the previous `seqlens_k[0] + 1u` that under-dispatched whenever batch 0 wasn't the longest. Per-batch `seqlens_k[batch] + 1` still drives causal masking and K/V bounds inside the kernels. GQA now enforces `graph_capture_enabled -> past_present_share_buffer_` so the host-side `use_indirect_dispatch` predicate stays simple. - **Decoupled attention_bias stride from per-batch OOB.** `attention_bias` is still allocated to the global max `total_sequence_length`; only the causal-mask / softmax tile loops are gated by the per-batch total. The one-past-end fallback was tightened to clamp inside the same row (`offset_base + stride_total_seq - 1u`). - **Decode workgroup grid stays at global max.** `decode_qkv` keeps a workgroup grid sized to the global max tile count to keep `workgroup_idx` slicing consistent across batches, with neutral `(-inf, 0)` early-exit for tiles beyond a short batch's per-batch total so the `VxReduce` online softmax rescaling is not skewed. - **New `use_seqlen_k` template parameter** (separate from `use_indirect_dispatch` which still requires graph capture). It is enabled whenever `seqlen_k` is provided and (`graph_capture || batch_size_ > 1`). - **Rotary fix prerequisite** (`webgpu: fix GQA batched right-padded prefill with do_rotary`, 591df5b): clamps `past_seqlen` to 0 in `RotaryEmbeddingProgram`, `FusedQKRotaryEmbeddingProgram`, and `split_packed_qkv_with_rotary_embedding`, which previously produced gibberish for the shorter batches. ## Motivation GenAI's batched prefill right-pads short prompts to the batch max and reports each batch's real length via `seqlens_k[b] = real_len[b] - 1`. The previous FlashAttention gate forced every batched call onto the slower `ApplyAttention` path, and the rotary shaders underflowed `u32` for any batch shorter than the batch-wide `sequence_length`, producing garbage Q/K positions and gibberish output text for the shorter batches. ## Test plan - [x] All `GroupQueryAttentionTest.WebGPU_*` op tests pass, including `BatchedRightPaddedRotaryPrefill` (FlashAttention path) and the new `BatchedRightPaddedRotaryPrefillFlashAttentionLargeSpread_WebGPU` covering a `real_lens` spread > tile_size - [x] phi4-prune three-prompt batched generation: coherent outputs on WebGPU matching CPU reference (3 prompts, 384 tokens, 173 tps) - [x] phi4-prune single-prompt generation regression: coherent - [x] phi4-graph-prune (graph capture enabled): `verify_model_correctness.py` 4/4 PASS; `verify_multi_gen.py` sequential + overlapping both PASS - [x] whisper-tiny-int4 transcription regression: 2/2 byte-exact with CPU - [x] Lintrunner clean on all changed files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.