[Perf] Linux/x64: 4 Regressions on 6/26/2026 6:28:17 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	22c1ced1d71c05e96f9cca3c2d6171ba9e5f0a55
Compare	207ffd155f96b38be1749584b99f3ac0a505218c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndRemove<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
OrderedDictionary - Duration of single invocation ADX - Test Multi Config Graph	272.19 μs	326.71 μs	1.20	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove<Int32>*'

Details

System.Collections.CreateAddAndRemove<Int32>.OrderedDictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	22c1ced1d71c05e96f9cca3c2d6171ba9e5f0a55
Compare	207ffd155f96b38be1749584b99f3ac0a505218c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.SearchValuesCharTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
LastIndexOfAnyExcept - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	87.76 ns	131.93 ns	1.50	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.SearchValuesCharTests*'

Details

System.Buffers.Tests.SearchValuesCharTests.LastIndexOfAnyExcept(Values: "ßäöüÄÖÜ")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	22c1ced1d71c05e96f9cca3c2d6171ba9e5f0a55
Compare	207ffd155f96b38be1749584b99f3ac0a505218c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Sort<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
List - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	3.34 μs	3.63 μs	1.08	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort<Int32>*'

Details

System.Collections.Sort<Int32>.List(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	22c1ced1d71c05e96f9cca3c2d6171ba9e5f0a55
Compare	207ffd155f96b38be1749584b99f3ac0a505218c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEach<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Dictionary - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	817.78 ns	923.82 ns	1.13	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach<String>*'

Details

System.Collections.IterateForEach<String>.Dictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in OrderedDictionary(Size: 512) — caused by commit 3681466 ("JIT: only skip loop inversion when the IV test is the bottom test (#129868)"). 4 tests bisected to related culprit commits. ⚠️ Note: Historical data shows bimodal behavior — bisection result may be unreliable.

Finding Confidence: 3/5 (analysis accuracy)
Regression Confidence: 2/5 (likelihood of true regression)

⚠️ Note: Low regression confidence - attribution uncertain. See rationale in full analysis.

Likely Cause: 368146691beb31c511384dfc08cffdeb52b54ff7

🤖 Proposed Next Actions: Uncertain — flag for re-triage — 1 done, 1 dry-run (rule R5_look_again — see Full Analysis for the table). · 🦆 Re-analysis: ⚠️ disputes (underconfident-change, inconsistent-noise-call)

📊 Full Analysis (click to expand)

Summary: All 4 reported regressions (+8–50%) are borderline or noise — compare values revert to prior historical levels after a transient step improvement in the baseline window. Z-scores are all below 1.0 (range 0.55–0.93) and the test histories show bimodal distributions that inflate adaptive thresholds. No commit in the 16-commit range directly modifies collections or buffers code. Finding confidence 3/5 (good data, bimodal distributions reduce discriminatory power), regression confidence 2/5 (weak signal, all within historical variance).

Issue Overview

• Issue: #75230 - [Perf] Linux/x64: 4 Regressions on 6/26/2026 6:28:17 PM +00:00
• Date: 2026-06-26 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: linux/x64
• Commits: 22c1ced1d71c → 207ffd155f96 (16 commits)
• Labels: untriaged, perf-regression, arch-x64, os-linux, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Runtime Stack: Desktop + CoreCLR (implicit)
• Historical Data: Available (285–299 runs per test). Note: all four tests exhibit bimodal before-series distributions (two distinct performance tiers), which elevates σ, inflates adaptive thresholds, and suppresses z-scores.

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
System.Collections.CreateAddAndRemove<Int32>.OrderedDictionary(Size: 512)	272.19 μs	326.71 μs	+20.03%	304,290	24,119	285	0.93	15.85%	Borderline	Compare (~327K) matches early historical tier; baseline was in recent lower tier. Reversion, not novel regression.
System.Buffers.Tests.SearchValuesCharTests.LastIndexOfAnyExcept(Values: "ßäöüÄÖÜ")	87.76 ns	131.93 ns	+50.33%	114.53	19.90	295	0.88	34.75%	Borderline	CV=17.4%. Before series stepped from ~132 to ~88; compare matches the earlier ~132 tier exactly. z < 1.
System.Collections.Sort<Int32>.List(Size: 512)	3.34 μs	3.63 μs	+8.50%	3,530.5	127.2	299	0.77	7.20%	Borderline	Marginal threshold exceedance (+1.3%). Before series drifted from ~3650 to ~3398; compare back to early levels.
System.Collections.IterateForEach<String>.Dictionary(Size: 512)	817.78 ns	923.82 ns	+12.97%	880.76	79.04	293	0.55	17.95%	Y (Noise)	Below adaptive threshold, z=0.55. CV=8.97%. Compare (~924) matches early historical mean (~923).

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 3/5 — Abundant historical data (285–299 runs), but bimodal distributions across all tests reduce the discriminatory power of z-score-based classification. Thresholds are inflated by the dual-tier distributions. No verification performed.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
368146691beb	src/coreclr/jit/optimizer.cpp	Relevant	JIT loop inversion fix; explicitly targets Dictionary MoveNext codegen. Should improve perf, not regress.
de11262bbe1a	src/coreclr/jit/emitarm64.cpp, src/coreclr/inc/corinfo.h	LessLikely	Arm64 R2R only; x64 not affected.
3c2aaa4b669c	src/coreclr/jit/lowerarmarch.cpp	LessLikely	arm64-only SIMD comparison changes.
d24159acdf71	src/coreclr/jit/codegenwasm.cpp, lowerwasm.cpp	LessLikely	WASM-only JIT codegen; not exercised on x64 CoreCLR.
5b733aa766fd	src/libraries/System.Reflection.Metadata/	Shared	BCL PE writer changes; no runtime hot-path impact.

Borderline Tests (No Confirmed Regressions)

All three borderline tests share the same pattern: bimodal historical distributions where the baseline captured a transiently improved (lower) performance tier and the compare reverts to the earlier (higher) tier. None represent novel regressions introduced by commits in this range.

System.Collections.CreateAddAndRemove<Int32>.OrderedDictionary(Size: 512) (+20.03%)

• Delta: 272.19 μs → 326.71 μs (+20.03%)
• Historical Context: μ=304,290 (σ=24,119, n=285); z-score=0.93; adaptive threshold=15.85%; CV=7.93%
• Assessment: Step change — bimodal distribution. The before series had two tiers (~326K then ~272K). The compare matches the earlier tier. The large delta reflects baseline capturing the lower tier, not a new regression.
• Finding Confidence: 3/5 — Good data volume but bimodal distribution reduces confidence in classification.
• Regression Confidence: 2/5 — z=0.93 is well below 2.0; compare is within historical norms. No commit in this range touches OrderedDictionary code.
• Likely Cause: No attribution — compare value matches pre-existing historical tier. No commit in the 16-commit range modifies collections creation/add/remove paths.
• Verification Status: Not verified — borderline classification with reversion pattern does not warrant verification.

System.Buffers.Tests.SearchValuesCharTests.LastIndexOfAnyExcept (+50.33%)

• Delta: 87.76 ns → 131.93 ns (+50.33%)
• Historical Context: μ=114.53 (σ=19.90, n=295); z-score=0.88; adaptive threshold=34.75%; CV=17.38%
• Assessment: Step change — extreme bimodal. Before series has clear step from ~132 to ~88 partway through. Compare (~132) matches the earlier tier exactly. Very high CV (17.4%) reflects dual-mode behavior. Despite the large raw delta, z < 1 and compare is within historical range.
• Finding Confidence: 3/5 — Adequate data but very high CV inflates threshold and makes classification uncertain.
• Regression Confidence: 2/5 — z=0.88, compare matches early historical levels, no SearchValues code changes in range.
• Likely Cause: No attribution — no commits in range touch System.Buffers or SearchValues code. The step improvement in the baseline window was likely transient.
• Verification Status: Not verified.

System.Collections.Sort<Int32>.List(Size: 512) (+8.50%)

• Delta: 3.34 μs → 3.63 μs (+8.50%)
• Historical Context: μ=3,530.5 (σ=127.2, n=299); z-score=0.77; adaptive threshold=7.20%; CV=3.60%
• Assessment: Step change — before series drifted from ~3650 down to ~3398. Compare (~3628) returns to early historical levels. Marginal threshold exceedance (8.5% vs 7.2%) with z well below significance.
• Finding Confidence: 3/5 — Best CV of the group (3.6%) but still bimodal.
• Regression Confidence: 2/5 — Marginal exceedance, z=0.77, reversion pattern. No sort-related code changes.
• Likely Cause: No attribution. The JIT loop inversion fix (368146691beb) re-enables inversion for ~1,600 methods and could theoretically affect sort loops, but the PR benchmarks show it improves Dictionary iteration performance (−2.4%), and the change restores pre-JIT: skip loop inversion for loops that are already bottom-tested #128459 codegen.
  • Candidates Considered:
    • 368146691beb - JIT loop inversion fix; could affect sort loops but expected to improve, not regress.
• Verification Status: Not verified.

Noise

System.Collections.IterateForEach<String>.Dictionary(Size: 512) (+12.97%)

• Delta: 817.78 ns → 923.82 ns (+12.97%)
• Historical Context: μ=880.76 (σ=79.04, n=293); z-score=0.55; adaptive threshold=17.95%; CV=8.97%
• Assessment: Noise — delta is below adaptive threshold (12.97% < 17.95%) and z=0.55 is far below 2. Compare (~924) matches early historical mean (~923). The before series had a step improvement to ~820 range, making the baseline atypically low.
• Note: Commit 368146691beb explicitly targets Dictionary MoveNext codegen and shows −2.4% improvement in its own BDN tests. This suggests it helps rather than hurts Dictionary iteration.

Open Questions

• Historical Data: Bimodal distributions across all 4 tests inflate σ and adaptive thresholds. The dual-tier pattern (step improvement followed by reversion) makes it difficult to distinguish new regressions from natural variance using z-score methods.
• Root Cause of Step Improvements: The step-down improvements visible in the before series for all tests share a common time window. Understanding what caused those improvements (JIT change? build configuration? infrastructure change?) would clarify whether the compare represents a true regression or expected reversion.
• Confidence Limitations: All z-scores < 1.0; no commit in the range directly modifies the affected test areas (collections, buffers). The only JIT change (368146691beb) is expected to improve performance.
• Bisection Status: Not performed — historical analysis strongly indicates reversion to prior levels rather than novel regression, making bisection unlikely to yield actionable results.

Recommended Actions

1. Monitor next runs — Confirm whether the compare-level values persist in subsequent runs or if the tests return to the lower tier, which would confirm transient behavior.
2. Investigate prior step improvement — Identify the commit/change that caused the step-down from ~326K to ~272K for OrderedDictionary (and similar for the other tests). If that improvement was intentional and has been lost, it may warrant a separate investigation.
3. Re-baseline consideration — Consider re-baselining these tests using the most recent stable window rather than the full bimodal history, to improve future detection accuracy.

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
OrderedDictionary(Size: 512)	✅ Success	368146691beb	✅ RELATED	272.2→326.7 μs (+20%)	277.3→326.5 μs (+18%)	2803s
LastIndexOfAnyExcept(Values: "ßäöüÄÖÜ")	✅ Success	368146691beb	✅ RELATED	87.8→131.9 ns (+50%)	87.6→132.0 ns (+51%)	2352s
List(Size: 512)	✅ Success	368146691beb	✅ RELATED	3.3→3.6 μs (+8%)	3.4→3.6 μs (+5%)	1291s
Dictionary(Size: 512)	✅ Success	368146691beb	✅ RELATED	817.8→923.8 ns (+13%)	819.3→930.3 ns (+14%)	1073s

Total bisection time: 7519s
Success rate: 4/4 tests

System.Collections.CreateAddAndRemove.OrderedDictionary(Size: 512)

• Culprit: 368146691beb
• Commit Message: JIT: only skip loop inversion when the IV test is the bottom test (JIT: only skip loop inversion when the IV test is the bottom test #129868)
• Relevance: RELATED
• Explanation: This JIT commit modifies loop inversion heuristics in optimizer.cpp, re-enabling inversion for ~1,600 methods including Dictionary-family enumerator patterns. The OrderedDictionary benchmark exercises collection add/remove operations that involve internal loops whose codegen is directly affected by these JIT loop optimization changes. The commit explicitly discusses Dictionary performance impact and code size/PerfScore changes.

Helix jobs

Job ID	Submitted by bisection	Last status
dce36f90-7f3e-46f3-bb90-f64369d76b7a	✅	submitted
ee2abe1a-4740-46f2-b409-aa049faa412d	✅	submitted

Per-commit measurements

Commit	Position	Measurements
3c2aaa4b669c	between q1-mid	277.343 μs
562ff6630e21	between q1-mid	277.272 μs
5ae5630ff3f4	q1 bisect	277.314 μs
d24159acdf71	3q bisect	327.378 μs
368146691beb	mid bisect	326.452 μs

Bisection details

…[earlier output omitted]

Now let me compile the results:

Measurements summary:

• 5ae5630 (Q1): 277,314 ns ✓
• 3c2aaa4: 277,343 ns ✓
• 562ff66: 277,272 ns ✓ (last good)
• 3681466: 326,452 ns ✗ (first bad — culprit)
• d24159a: 327,378 ns ✗

BISECTION RESULT:
- Culprit Commit: 368146691beb31c511384dfc08cffdeb52b54ff7
- Commit Message: JIT: only skip loop inversion when the IV test is the bottom test (#129868)
- Commits Tested: 6
- Confidence: 3
- Commit Relevance: RELATED
- Relevance Explanation: This JIT commit modifies loop inversion heuristics in optimizer.cpp, re-enabling inversion for ~1,600 methods including Dictionary-family enumerator patterns. The OrderedDictionary benchmark exercises collection add/remove operations that involve internal loops whose codegen is directly affected by these JIT loop optimization changes. The commit explicitly discusses Dictionary performance impact and code size/PerfScore changes.
- Baseline Value: 277272.23 ns
- Culprit Value: 326452.48 ns
- Summary: Bisected 16 commits between 22c1ced1d71c and 207ffd155f96 using double bisection. The regression (+18%) appears at commit 3681466 (JIT loop inversion change), with the preceding commit 562ff66 showing baseline-level performance. Note: the historical noise floor (CV 7.93%) is significant but the observed step change (~49,000 ns) exceeds 2σ (~48,239), providing moderate confidence. Single-sample measurements were used due to the clear step pattern across multiple commits.

HELIX JOBS:
- dce36f90-7f3e-46f3-bb90-f64369d76b7a
- ee2abe1a-4740-46f2-b409-aa049faa412d

MEASUREMENTS:
- 5ae5630ff3f4 (Q1 bisect): 277313.64 ns
- 3c2aaa4b669c (between Q1-mid): 277343.33 ns
- 562ff6630e21 (between Q1-mid): 277272.23 ns
- 368146691beb (mid bisect): 326452.48 ns
- d24159acdf71 (3Q bisect): 327377.77 ns

System.Buffers.Tests.SearchValuesCharTests.LastIndexOfAnyExcept(Values: "ßäöüÄÖÜ")

• Culprit: 368146691beb
• Commit Message: JIT: only skip loop inversion when the IV test is the bottom test (JIT: only skip loop inversion when the IV test is the bottom test #129868)
• Relevance: RELATED
• Explanation: This commit modifies the JIT loop inversion heuristic in src/coreclr/jit/optimizer.cpp, re-enabling loop inversion for ~1,600 methods. The SearchValuesCharTests.LastIndexOfAnyExcept benchmark exercises tight scanning loops over character buffers via SearchValues, which are directly affected by loop optimization decisions. The changed codegen (loop inversion being re-enabled) plausibly alters the hot loop's instruction layout and branch behavior, causing the observed regression.

Helix jobs

Job ID	Submitted by bisection	Last status
233bccc0-a65c-4a5d-b048-8c2e8e40ef94	✅	completed
43354e3c-b663-4543-a1f4-161b6b1fe374	✅	submitted
7b1e8f46-2e32-4944-b0a4-72a4260b1e76	✅	submitted
4aad409c-f796-4941-bac1-5e0c3166fb11	✅	submitted
de6db09b-34e8-464d-95e3-8e3cd0a1b0e1	✅	submitted

Per-commit measurements

Commit	Position	Measurements
22c1ced1d71c	baseline	87.680 ns
5ae5630ff3f4	q1 bisect	87.590 ns
3c2aaa4b669c	q2-start	87.650 ns
d24159acdf71	q3 bisect	131.780 ns
207ffd155f96	head	131.800 ns
368146691beb	culprit	131.850, 132.030 ns
562ff6630e21	last-good	87.540, 87.630 ns

Bisection details

…[earlier output omitted]

• 562ff6630e21: 87.63 ns (consistent with first run's 87.54 ns)
• 368146691beb: 132.03 ns (consistent with first run's 131.85 ns)

The results are very consistent across runs, confirming 368146691beb as the culprit despite the high historical noise floor.

BISECTION RESULT:
- Culprit Commit: 368146691beb31c511384dfc08cffdeb52b54ff7
- Commit Message: JIT: only skip loop inversion when the IV test is the bottom test (#129868)
- Commits Tested: 8
- Confidence: 3
- Commit Relevance: RELATED
- Relevance Explanation: This commit modifies the JIT loop inversion heuristic in `src/coreclr/jit/optimizer.cpp`, re-enabling loop inversion for ~1,600 methods. The `SearchValuesCharTests.LastIndexOfAnyExcept` benchmark exercises tight scanning loops over character buffers via `SearchValues`, which are directly affected by loop optimization decisions. The changed codegen (loop inversion being re-enabled) plausibly alters the hot loop's instruction layout and branch behavior, causing the observed regression.
- Baseline Value: 87.63 ns
- Culprit Value: 132.03 ns
- Summary: Bisected 16 commits using double bisection. The regression transitions cleanly between 562ff6630e21 (87.63 ns) and 368146691beb (132.03 ns), confirmed by a second Helix run. Confidence capped at 3/5 due to the test's high historical noise (CV=17.38%, 2σ≈45% of baseline), though measurements were very consistent across all runs.

HELIX JOBS:
- 233bccc0-a65c-4a5d-b048-8c2e8e40ef94
- 43354e3c-b663-4543-a1f4-161b6b1fe374
- 7b1e8f46-2e32-4944-b0a4-72a4260b1e76
- 4aad409c-f796-4941-bac1-5e0c3166fb11
- de6db09b-34e8-464d-95e3-8e3cd0a1b0e1

MEASUREMENTS:
- 22c1ced1d71c (baseline): 87.68 ns
- 5ae5630ff3f4 (Q1 bisect): 87.59 ns
- 3c2aaa4b669c (Q2-start): 87.65 ns
- 562ff6630e21 (last-good): 87.54 ns
- 562ff6630e21 (confirm): 87.63 ns
- 368146691beb (culprit): 131.85 ns
- 368146691beb (confirm): 132.03 ns
- d24159acdf71 (Q3 bisect): 131.78 ns
- 207ffd155f96 (head): 131.80 ns

System.Collections.Sort.List(Size: 512)

• Culprit: 368146691beb
• Commit Message: JIT: only skip loop inversion when the IV test is the bottom test (JIT: only skip loop inversion when the IV test is the bottom test #129868)
• Relevance: RELATED
• Explanation: This commit modifies JIT loop inversion heuristics in src/coreclr/jit/optimizer.cpp, re-enabling loop inversion for ~1,600 methods. The Sort.List benchmark exercises sorting loops that are directly affected by whether the JIT inverts top-tested loops to bottom-tested form. The change alters code generation for collection-related hot paths, providing a plausible mechanism for the ~8% regression in sorting performance.

Helix jobs

Job ID	Submitted by bisection	Last status
5b396146-fcc4-45bf-b55f-ba187c59c87b	✅	submitted
ebb8a732-b0ef-4276-9d22-4774c381333f	✅	submitted
77b620ca-3d43-44a0-8dfa-fc804a055068	✅	submitted

Per-commit measurements

Commit	Position	Measurements
22c1ced1d71c	baseline	3.458 μs
5ae5630ff3f4	q1	3.403 μs
d24159acdf71	q3	3.655 μs
3c2aaa4b669c	index 5	3.404, 3.404 μs
562ff6630e21	index 7 (run 1, 2)	3.386, 3.386, 3.450 μs
368146691beb	idx8 (run 1, 2)	3.688, 3.634, 3.688 μs

Bisection details

…[earlier output omitted]

-------|-------|--------|
| 22c1ced (baseline) | 3457.81 ns | — | ✓ good |
| 5ae5630 (Q1) | 3402.82 ns | — | ✓ good |
| 3c2aaa4 (idx 5) | 3404.12 ns | — | ✓ good |
| 562ff66 (idx 7) | 3385.84 ns | 3449.77 ns | ✓ good |
| 3681466 (idx 8) | 3687.68 ns | 3634.05 ns | ✗ regressed |
| d24159a (Q3) | 3655.03 ns | — | ✗ regressed |

BISECTION RESULT:
- Culprit Commit: 368146691beb31c511384dfc08cffdeb52b54ff7
- Commit Message: JIT: only skip loop inversion when the IV test is the bottom test (#129868)
- Commits Tested: 8
- Confidence: 3
- Commit Relevance: RELATED
- Relevance Explanation: This commit modifies JIT loop inversion heuristics in src/coreclr/jit/optimizer.cpp, re-enabling loop inversion for ~1,600 methods. The Sort<Int32>.List benchmark exercises sorting loops that are directly affected by whether the JIT inverts top-tested loops to bottom-tested form. The change alters code generation for collection-related hot paths, providing a plausible mechanism for the ~8% regression in sorting performance.
- Baseline Value: 3449.77 ns
- Culprit Value: 3634.05 ns
- Summary: Bisected 16 commits across 3 Helix jobs. The regression consistently appears at commit 368146691beb (JIT loop inversion change) with all prior commits measuring ~3400 ns and all subsequent commits measuring ~3650 ns. Confidence capped at 3/5 due to the test's historical noise floor (2σ ≈ 254 ns) being comparable to the regression delta (~200 ns), though the step was confirmed across two independent runs.

HELIX JOBS:
- 5b396146-fcc4-45bf-b55f-ba187c59c87b
- ebb8a732-b0ef-4276-9d22-4774c381333f
- 77b620ca-3d43-44a0-8dfa-fc804a055068

MEASUREMENTS:
- 22c1ced1d71c (baseline): 3457.81 ns
- 5ae5630ff3f4 (Q1): 3402.82 ns
- 3c2aaa4b669c (idx5): 3404.12 ns
- 562ff6630e21 (idx7, run1): 3385.84 ns
- 562ff6630e21 (idx7, run2): 3449.77 ns
- 368146691beb (idx8, run1): 3687.68 ns
- 368146691beb (idx8, run2): 3634.05 ns
- d24159acdf71 (Q3): 3655.03 ns
- 207ffd155f96 (compare): N/A

System.Collections.IterateForEach.Dictionary(Size: 512)

• Culprit: 368146691beb
• Commit Message: JIT: only skip loop inversion when the IV test is the bottom test (JIT: only skip loop inversion when the IV test is the bottom test #129868)
• Relevance: RELATED
• Explanation: The commit modifies JIT loop inversion heuristics in src/coreclr/jit/optimizer.cpp, re-enabling inversion for ~1,600 methods including Dictionary enumerator patterns. The commit explicitly discusses IterateForEach.Dictionary performance and fixes Dictionary+Enumerator.MoveNext codegen, but the String variant tested here may be affected differently by the re-enabled inversion, causing a regression instead of an improvement.

Helix jobs

Job ID	Submitted by bisection	Last status
a227dfff-f9bf-4079-9be0-28b4329bb3b6	✅	submitted
23d094b7-baab-409f-81cd-8894ad1a825f	✅	submitted
1b955840-3bdf-4a7a-b41e-99f72a389ebf	✅	submitted

Per-commit measurements

Commit	Position	Measurements
(baseline)	baseline	819.310 ns
5ae5630ff3f4	q1 bisect	818.660 ns
d24159acdf71	q3 bisect	920.040 ns
(culprit)	culprit	930.300 ns
368146691beb	mid bisect	934.180, 926.420 ns
3c2aaa4b669c	narrowing	820.050 ns
562ff6630e21	narrowing	819.880, 818.740 ns

Bisection details

…[earlier output omitted]

Actually, looking at the SPMI data in the commit, it shows code size increases for some collections. The loop inversion being re-enabled for more loops could cause regressions for some specific patterns while fixing others. This commit is clearly RELATED to the test.

Now let me compile the final results. Averaging the two runs:

• 562ff663 baseline: (819.88 + 818.74) / 2 = 819.31
• 36814669 regressed: (934.18 + 926.42) / 2 = 930.30

BISECTION RESULT:
- Culprit Commit: 368146691beb31c511384dfc08cffdeb52b54ff7
- Commit Message: JIT: only skip loop inversion when the IV test is the bottom test (#129868)
- Commits Tested: 8
- Confidence: 3
- Commit Relevance: RELATED
- Relevance Explanation: The commit modifies JIT loop inversion heuristics in src/coreclr/jit/optimizer.cpp, re-enabling inversion for ~1,600 methods including Dictionary enumerator patterns. The commit explicitly discusses IterateForEach<Int32>.Dictionary performance and fixes Dictionary+Enumerator.MoveNext codegen, but the String variant tested here may be affected differently by the re-enabled inversion, causing a regression instead of an improvement.
- Baseline Value: 819.31 ns
- Culprit Value: 930.30 ns
- Summary: Bisected 16 commits in 3 Helix jobs. The regression appears at commit 368146691beb which changes JIT loop inversion logic. Both confirmation runs consistently show a ~110 ns step change, though this is within the test's high noise floor (2σ=158 ns, CV=8.97%), so confidence is capped at 3/5.

HELIX JOBS:
- a227dfff-f9bf-4079-9be0-28b4329bb3b6
- 23d094b7-baab-409f-81cd-8894ad1a825f
- 1b955840-3bdf-4a7a-b41e-99f72a389ebf

MEASUREMENTS:
- 5ae5630ff3f4 (Q1 bisect): 818.66 ns
- 368146691beb (mid bisect): 934.18 ns
- d24159acdf71 (Q3 bisect): 920.04 ns
- 3c2aaa4b669c (narrowing): 820.05 ns
- 562ff6630e21 (narrowing): 819.88 ns
- 562ff6630e21 (confirmation): 818.74 ns
- 368146691beb (confirmation): 926.42 ns

🤖 Proposed Next Actions

Rule: R5_look_again — Uncertain — flag for re-triage.

Finding confidence 3/5 is below the required 4/5. Revisit once more historical data is available.

#	Action	Status	Details
1	Add label	✅ Done	Look Again
2	Remove label	⏭️ Skipped (dry-run)	untriaged

🦆 Pre-bisection Re-analysis (advisory)

⚠️ Disputes the pre-bisection plan — advisory only, not applied

Flags: wasted-bisection-slot

Rationale: System.Collections.IterateForEach.Dictionary(Size: 512) is classified as noise=True with a change confidence of only 1/5, yet it is still queued for bisection — this is likely to burn ~60+ Helix minutes for a test the analysis itself considers noise. The remaining three candidates have modest change confidence (2/5) but are at least classified as actionable (noise=False), so they are reasonable to bisect given bimodal history.

Suggested drop from bisection: System.Collections.IterateForEach<String>.Dictionary(Size: 512) (advisory — not applied)

This critique is advisory in the current configuration and did not change which tests were bisected.

🦆 Re-analysis (advisory)

⚠️ Disputes the analysis — advisory only, not applied

Flags: underconfident-change, inconsistent-noise-call

Rationale: Four independent bisections all converge on the same commit with no alignment warnings, which is strong evidence that the change is real and the culprit is correct — this justifies raising change confidence from 2 to 3 despite the bimodal history flag. Additionally, the IterateForEach finding is marked noise=True yet its bisection also converges on the same culprit with status=success, which is inconsistent with a pure noise classification for that benchmark.

Suggested confidence: Change Confidence → 3/5 (advisory — not applied)

This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

---

Generated by PerfTriageAgent on 2026-06-30 16:35 UTC

[LoopedBard3]

Likely due to #129868, FYI @AndyAyersMS. For some reason the triage bot is fairly unconfident.

Related Regressions:

• [Perf] Linux/x64: 1 Regression on 6/26/2026 9:56:31 PM +00:00 perf-autofiling-issues#75231
• [Perf] Windows/x64: 4 Regressions on 6/26/2026 6:28:17 PM +00:00 perf-autofiling-issues#75183
• https://github.com/dotnet/perf-autofiling-issues/issues/75181 (Portion of them match here)
• [Perf] Linux/arm64: 31 Regressions on 6/26/2026 6:28:17 PM +00:00 perf-autofiling-issues#75492
• [Perf] Linux/arm64: 1 Regression on 6/26/2026 9:56:31 PM +00:00 perf-autofiling-issues#75481
• [Perf] Linux/arm64: 3 Regressions on 6/27/2026 6:02:44 AM +00:00 perf-autofiling-issues#75493 (Incorrect date on issue)
• [Perf] Linux/arm64: 1 Regression on 6/27/2026 4:35:02 AM +00:00 perf-autofiling-issues#75482 (Incorrect date)
• [Perf] Linux/arm64: 7 Regressions on 6/26/2026 6:28:17 PM +00:00 perf-autofiling-issues#75480

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.