[pull] main from pytorch:main#1632
Merged
Merged
Conversation
## Summary Update mergebot (`trymerge.py`) to recognize CRCR L3 out-of-tree CI failures as non-blocking, preventing them from halting merges. Depends on the Dr.CI + CRCR infrastructure in [pytorch/test-infra#8183](pytorch/test-infra#8183). ## Changes ### New classification function: `is_crcr_l3` Mirrors the existing `is_flaky` / `is_broken_trunk` / `is_unstable` pattern. Consults the `CRCR_L3` category returned by the Dr.CI API and matches on the standard `name` and `id` fields of the `RecentWorkflowsData` schema. ### Classification in `get_classifications` When a check matches the `CRCR_L3` category from Dr.CI, it is classified as `"CRCR_L3"` and excluded from the unclassified failures pool. ### Non-blocking categorization in `categorize_checks` `CRCR_L3` is added to the list of non-blocking classifications (alongside `BROKEN_TRUNK`, `FLAKY`, `UNSTABLE`). CRCR_L3 failures are intentionally excluded from the `flaky_or_broken_trunk` threshold logic — L3 failures are definitionally non-blocking regardless of count. ## Data Flow ``` CRCR check run completes (oot_workflow_job) ↓ Dr.CI fetches from ClickHouse oot_workflow_job ↓ Dr.CI classifies: L3 → CRCR_L3, L4 → FAILED ↓ Dr.CI API returns {"CRCR_L3": [...], "FAILED": [...]} ↓ trymerge.py reads Dr.CI API response ├── CRCR_L3 → non-blocking (treated like FLAKY/BROKEN_TRUNK) └── FAILED → blocking (L4 failures land here, naturally stopping merge) ``` ## Design Notes - **L4 failures stay blocking**: Dr.CI merges L4 failures into `failedJobs`, not `CRCR_L3`. They flow through the normal failure path in trymerge.py and block the merge. No extra L4-specific logic is needed on this side. - **The classification authority is Dr.CI**: trymerge.py does not query the allowlist or check GitHub check run names itself. It trusts the `CRCR_L3` category from Dr.CI. - **Errors fail closed**: If Dr.CI is unreachable or returns no classifications, checks remain unclassified and block the merge. This is the same behavior as the other classification categories. Pull Request resolved: #185612 Approved by: https://github.com/can-gaa-hou, https://github.com/subinz1, https://github.com/atalman
Pull Request resolved: #188004 Approved by: https://github.com/rtimpe, https://github.com/hameerabbasi
Pull Request resolved: #188638 Approved by: https://github.com/rtimpe ghstack dependencies: #188004
Pull Request resolved: #188639 Approved by: https://github.com/rtimpe ghstack dependencies: #188004, #188638
**Motivation and Example** Explictly close all open generators in compile_subgraph to ensure that all remaining finally blocks are executed. In CPython this is done by invoking the `tp_finalize` function of the generator object, which triggers `genclose`: https://github.com/python/cpython/blob/58a42dea97f4fa0df38ef4a95a2ede65e0549f71/Objects/genobject.c#L128-L134 ```python import gc def whoo(t): nonlocal z z = 0 try: z += 1 yield t.sin() except ValueError: z += 10 yield t.cos() except RuntimeError: z += 100 yield t.tan() finally: z += 1000 z += 10_000 gen = whoo(t) a = next(gen) b = gen.throw(RuntimeError) gc.collect() print(z) # 1101 ``` Pull Request resolved: #157149 Approved by: https://github.com/zou3519 ghstack dependencies: #188004, #188638, #188639
…rror/NameError) (#189024) Dynamo wrapped exceptions in a single ExceptionVariable that only tracked args/__context__/__cause__/__traceback__. Reading exception-specific attributes (StopIteration.value, AttributeError.name/.obj, NameError.name) or passing those attributes as keyword arguments to the constructor either returned the wrong value or raised a graph break ("keyword args passed to an exception constructor"). This adds per-exception VariableTrackers so those attributes round-trip through tracing, reconstruction, and real raises. Suggested review order: 1. variables/misc.py: StopIterationVariable already carried `.value`. Add a shared `_KwargAttrExceptionVariable` base that pops keyword-only attributes (default None) from init_kwargs, exposes them via getattr, and restores them on reconstruct via store_attr (mirroring how the base emits __context__ / __cause__). AttributeErrorVariable and NameErrorVariable are one-liners that list their attribute names. 2. variables/builtin.py + variables/__init__.py: route AttributeError and NameError construction to the new trackers and export them. 3. variables/user_defined.py: when a generic getattr fails, pass name=<attr> and obj=self to the synthesized AttributeError so a caught exception reports the missing attribute like eager does. 4. symbolic_convert.py: an undefined name previously graph-broke with a DYNAMO_BUG hint. Raise an observed NameError(name=<ident>) instead, matching CPython so `except NameError as e: e.name` works. The old graph break (GB0354) becomes dormant; the registry keeps it for id history. This PR was authored with the assistance of Claude (Anthropic). Pull Request resolved: #189024 Approved by: https://github.com/hameerabbasi, https://github.com/Skylion007 ghstack dependencies: #188004, #188638, #188639, #157149, #188824, #188834, #188825
#185473 changed AOTAutograd's partitioner to discover backward symbol bindings from unreplaced placeholder expressions (_free_symbols_without_replacements), fixing undefined symbols in backward runtime assertions (#155468). But sizevar codegen and FxGraphCache guard production use the ShapeEnv-replaced expressions instead. When a backward input's symbol is replaced -- e.g. an offsets tensor of size `s` where `s` is replaced to `b + 1` -- the partitioner saved the binding for the raw symbol but not for the replacement target `b`. Downstream, produce_guards then referenced `b`, which was never bound as a backward input, so ShapeGuardPrinter fell back to var_to_sources and emitted the original forward source name into the stored guards_expr. On a later FxGraphCache lookup, find_guarded_entry -> evaluate_guards_expression evaluated that expression with args named t0, t1, ... and raised KeyError on the unbound source name (KeyError: 'B_RO' in the reported model). Fix by discovering both the unreplaced and replaced free symbols when selecting backward symbol bindings, so both runtime assertions (raw) and sizevar/guard codegen (replaced) have their symbols bound. This is a strict superset of #185473: the raw symbol is still bound, so #155468 is not regressed; the replacement target is additionally bound. When no replacement is involved the two symbol sets are equal and nothing extra is bound. An alternative was to make find_guarded_entry treat a guard-eval KeyError as a cache miss rather than a hard crash, but that only masks the missing binding (and disables caching for the affected backward graph); fixing the binding in the partitioner keeps backward guards correct and cacheable. Test Plan: ``` python test/inductor/test_backward_symint_guards.py python test/dynamo/test_aot_autograd.py -k test_partitioner_saves_unreplaced_input_symbols_for_bw python test/inductor/test_unbacked_symints.py -k cpu python test/functorch/test_aotdispatch.py -k "partition or min_cut or dynamic" ``` Authored with Claude. Pull Request resolved: #189783 Approved by: https://github.com/bobrenjc93
…anch (#189802) Follow-up to #189717 addressing @albanD's review comment ("You can re-use the condition above") on `test/test_tensor_creation_ops.py`. In `test_float_to_int_conversion_finite`, the `torch.int8` and `torch.int16` CPU branches used identical `vals`. This folds them into a single `dtype in (torch.int8, torch.int16)` case instead of duplicating the branch. No behavior change: the set of test values for int8 and int16 is unchanged. ## Test Plan - `python test/test_tensor_creation_ops.py TestTensorCreationCPU.test_float_to_int_conversion_finite_cpu_int8` - `python test/test_tensor_creation_ops.py TestTensorCreationCPU.test_float_to_int_conversion_finite_cpu_int16` Pull Request resolved: #189802 Approved by: https://github.com/albanD
Use is_apple_family_or_newer helper Pull Request resolved: #189867 Approved by: https://github.com/malfet
…ilable (#189763) authored with codex Pull Request resolved: #189763 Approved by: https://github.com/Skylion007
…ps from data sheet. (#189819) ## Motivation Some Inductor UT can not pass on XPU due to lack of correct device tflops and dram gdbps. ## Solution - Add Intel GPU PVC device info which the XPU CI runs on. - Query dram gbps from the datasheet frist because the `triton.testing.get_dram_gbps` current not work correctly on XPU. ## Test - Enabled expected failure/skipped cases on XPU. Pull Request resolved: #189819 Approved by: https://github.com/eellison
## Related Issue Fixes #169188 ## Why `exec_tvm` filters runtime arguments by rank before binding them, so a rank-0 input is never passed to TVM and the graph executor silently uses its default zero value. A scalar `nn.Parameter` captured by Dynamo hits exactly this and produces wrong numerics instead of an error. Reworks #184997, which had the same fix but tested it against mocked TVM modules. ## How - Drop the rank filter and bind every argument TVM reports as an active input - Add `test_tvm_scalar_tensor_input`, gated only on `has_tvm()` so it covers both the relay and relax paths Verified against TVM v0.19.0 built from source, since relay was removed in TVM 0.20 and no pip wheel ships it. The output was off by exactly the parameter value before the fix and matches eager after; the new test fails without the fix and passes with it. The relax path already handled rank-0 inputs correctly. Pull Request resolved: #189699 Approved by: https://github.com/ezyang
Fixes: intel/torch-xpu-ops#3479 Updates the mapping between `libkineto::ChromeTraceLogger::handleActivity()` and `KinetoEvent::externalId()` after following change: pytorch/kineto#1390 Pull Request resolved: #184358 Approved by: https://github.com/Silv3S, https://github.com/gujinghui, https://github.com/aostrowski-hbn, https://github.com/ryanzhang22
Use round up utility Pull Request resolved: #189869 Approved by: https://github.com/malfet, https://github.com/Skylion007
Fixes #188987 Resolves #157778 ## Problem `test_optimizer_non_static_param` was marked as `@xfailIf(TEST_XPU)` (expected to fail on XPU) because `torch.accelerator.Graph` support was missing on XPU. This caused CI to report "unexpected success" once the underlying issue was fixed: ``` test_optimizer_non_static_param ... unexpected success FAILED (unexpected successes=1) ``` ## Root Cause XPU Graph support was added in PR #176421 (April 2026). The `@xfailIf(TEST_XPU)` decorator was never removed after the fix landed. **Timeline**: - 2025-07-08: Issue #157778 created (XPU Graph support missing) - 2025-07-11: `@xfailIf(TEST_XPU)` added as workaround - **2026-04-01**: XPU Graph support added (PR #176421) ← THE FIX - 2026-04-07: Issue #157778 closed - 2026-07-06: Issue #188987 opened — test passes but decorator causes "unexpected success" ## Fix Remove the outdated `@xfailIf(TEST_XPU)` decorator and its now-unused imports (`TEST_XPU`, `xfailIf`). Update `test_recompiles` expected line numbers to reflect the 2-line shift. ```diff - TEST_XPU, - xfailIf, ) ... - - File [file_path], line 201, in outmost_fn + - File [file_path], line 199, in outmost_fn ... - @xfailIf(TEST_XPU) # #157778 @make_logging_test(perf_hints=True) @requires_gpu def test_optimizer_non_static_param(self, records): ``` **Note on `test_recompiles`**: `test_recompiles` validates PyTorch recompile logging by checking exact stack trace line numbers of functions defined within the test file. Removing 2 import lines shifts all subsequent line numbers by -2, so the expected assertions must be updated accordingly (`line 201` → `line 199`, etc.). ## Verification **Environment**: - Hardware: Intel Data Center GPU Max 1550 (PVC) x8 - PyTorch: 2.14.0.dev20260707+xpu (nightly) - oneAPI: 2026.0.0 (fresh environment, no cache) **Post-fix**: ``` test_optimizer_non_static_param ... ok OK ``` ### CI Validation (test PR #189407) A test PR was created to validate CI behavior before updating this PR: | Shards | Result | |--------|--------| | Shard 4 (`test_recompiles` + `test_optimizer_non_static_param`) | ✅ PASS | | Shards 1–8, 11–12 | ✅ PASS | | Shards 9, 10 | ❌ `test_create_with_external_constants_xpu` SIGSEGV — pre-existing bug, unrelated to this PR | ### Test Plan ```bash python test/dynamo/test_logging.py LoggingTests.test_optimizer_non_static_param -v python test/dynamo/test_logging.py LoggingTests.test_recompiles -v ``` Pull Request resolved: #189344 Approved by: https://github.com/frgossen Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…timm training baselines (#189766) #187692 added mobilenetv2_100 to accuracy.skip.eager_not_deterministic in timm_models.yaml to skip the flaky eager-vs-eager training self-check (cuDNN conv/bn backward is non-deterministic). That skip is global and device-agnostic (benchmarks/dynamo/common.py records pass_due_to_skip for any timm training accuracy run), but #187692 only updated inductor_timm_training.csv. The other timm training baselines still expected plain "pass", so they now regress: mobilenetv2_100 FAIL: accuracy=pass_due_to_skip, expected=pass Update the remaining timm training CSVs to pass_due_to_skip, matching inductor_timm_training.csv: - dynamic_inductor_timm_training.csv (fixes the CUDA a10g dynamic job) - rocm/dynamic_inductor_timm_training.csv - rocm/inductor_timm_training.csv Inference is unaffected (the eager self-check is training-only), so those CSVs are left as-is. Failing job: inductor-periodic / periodic-dynamo-benchmarks-test / test (dynamic_inductor_timm, 1, 2, mt-l-x86aavx2-29-113-a10g) Test Plan: CI -- these CSVs are consumed by the periodic-dynamo-benchmarks jobs. Authored with Claude. Pull Request resolved: #189766 Approved by: https://github.com/williamwen42
…9883) _log_torch_version() (added in #178024) emits a global {"artifact": {"name": "torch_version", ...}} record once per process the first time the trace handler initializes. This record propagates to the StructuredTraceTest buffer handler. In OSS unittest runs the lazy trace handler is not active so it never fires, but internally trace logging is enabled and each test runs in isolation, so whichever test runs first picks up a torch_version line its expected inline output does not have. This left test_dump_file permanently failing and disabled. Filter the torch_version artifact out in StructuredTraceTestingFilter, alongside the existing "str" drop, so the once-per-process artifact never pollutes any test's golden output regardless of run context or ordering. Differential Revision: [D111928737](https://our.internmc.facebook.com/intern/diff/D111928737/) Pull Request resolved: #189883 Approved by: https://github.com/aorenste
…um now passes (#189764) microbench_unbacked_tolist_sum improved from fail_to_run to pass accuracy on the periodic aot_inductor_torchbench job, so check_accuracy reports it as an IMPROVED regression against the fail_to_run baseline and reds the job: microbench_unbacked_tolist_sum IMPROVED: accuracy=pass, expected=fail_to_run The improvement is stable (pass on the last several consecutive periodic runs, not a flake), so bump the expected status to pass. Job: inductor-periodic / periodic-dynamo-benchmarks-test / test (aot_inductor_torchbench, 2, 2, mt-l-x86aavx2-29-113-a10g) Test Plan: CI -- this CSV is consumed by the periodic-dynamo-benchmarks jobs. Authored with Claude. Pull Request resolved: #189764 Approved by: https://github.com/williamwen42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )