Skip to content

[Perf] Linux/x64: 29 Regressions on 3/1/2023 5:53:50 PM #13755

Open
@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 605d05ba54ccea0ff5486926a7dc630cee5db8cd
Compare ef0241ed9316bc4343e715daca4d88c1ff90231b
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexerCheckLengthHoisting - Duration of single invocation 26.39 ns 41.83 ns 1.59 0.52 False
ToLower - Duration of single invocation 225.62 ns 353.43 ns 1.57 0.15 False
CtorCharCount - Duration of single invocation 59.70 ns 77.58 ns 1.30 0.32 False
PadLeft - Duration of single invocation 79.43 ns 103.09 ns 1.30 0.40 False
ToUpper - Duration of single invocation 224.64 ns 377.85 ns 1.68 0.16 False
ToUpperInvariant - Duration of single invocation 193.00 ns 306.12 ns 1.59 0.11 False
ToLowerInvariant - Duration of single invocation 205.16 ns 303.99 ns 1.48 0.10 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.IndexerCheckLengthHoisting


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.82894696760661 > 27.703327538697756.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -32.191395005720636 (T) = (0 -39.227952731190776) / Math.Sqrt((2.2987827611203038 / (29)) + (2.130444531773961 / (39))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (39) - 2, .025) and -0.42914333445426117 = (27.448578309446148 - 39.227952731190776) / 27.448578309446148 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 353.4337818960568 > 232.82118714013097.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -40.634085757075816 (T) = (0 -362.1090027188929) / Math.Sqrt((20.297529142740533 / (30)) + (420.02307397867094 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.6119985744079897 = (224.6335750339471 - 362.1090027188929) / 224.6335750339471 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.CtorCharCount(size: 10)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 77.57732561551073 > 66.44490945482254.
IsChangePoint: Marked as a change because one of 1/18/2023 3:58:17 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -15.416099279258859 (T) = (0 -76.09339517367445) / Math.Sqrt((5.6031448767946035 / (30)) + (16.326698069258953 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.18713272215230525 = (64.09847336674788 - 76.09339517367445) / 64.09847336674788 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.PadLeft(n: 18)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 103.08926884112797 > 85.43390280392919.
IsChangePoint: Marked as a change because one of 1/14/2023 10:43:56 AM, 3/1/2023 5:53:50 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -9.68525725823996 (T) = (0 -98.39761883754494) / Math.Sqrt((16.14965701158545 / (31)) + (69.64983621634943 / (38))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (38) - 2, .025) and -0.17787499411268176 = (83.538252640867 - 98.39761883754494) / 83.538252640867 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 377.84674911210425 > 237.63925886733796.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -39.156693044760175 (T) = (0 -364.3480862142918) / Math.Sqrt((17.79434553648585 / (30)) + (475.4163860480634 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.6240293682749661 = (224.34821274279054 - 364.3480862142918) / 224.34821274279054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 306.12151880255936 > 203.10880620425826.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -50.77168771921563 (T) = (0 -303.4604725821685) / Math.Sqrt((17.974454628641865 / (30)) + (160.15292965899772 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.5696978903175034 = (193.3241259060286 - 303.4604725821685) / 193.3241259060286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 303.9909950767776 > 200.74636420269337.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -53.66505143777063 (T) = (0 -305.4980945172726) / Math.Sqrt((20.166320219948606 / (30)) + (140.9072518715231 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.5714330257579686 = (194.40732726736346 - 305.4980945172726) / 194.40732726736346 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 605d05ba54ccea0ff5486926a7dc630cee5db8cd
Compare ef0241ed9316bc4343e715daca4d88c1ff90231b
Diff Diff

Regressions in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetByteCount - Duration of single invocation 38.61 ns 44.33 ns 1.15 0.63 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_Encoding.GetByteCount(size: 16, encName: "utf-8")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 44.33202723180661 > 36.7779864338579.
IsChangePoint: Marked as a change because one of 1/17/2023 8:36:05 PM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -17.565347295110854 (T) = (0 -49.46813840273708) / Math.Sqrt((7.920079201110455 / (30)) + (11.12025465017022 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.35709294277888853 = (36.45154789578545 - 49.46813840273708) / 36.45154789578545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 605d05ba54ccea0ff5486926a7dc630cee5db8cd
Compare ef0241ed9316bc4343e715daca4d88c1ff90231b
Diff Diff

Regressions in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Log2_uint - Duration of single invocation 563.73 ns 5.07 μs 8.99 0.12 False
TrailingZeroCount_ulong - Duration of single invocation 392.74 ns 985.58 ns 2.51 0.01 False
PopCount_uint - Duration of single invocation 393.95 ns 762.57 ns 1.94 0.00 False
TrailingZeroCount_uint - Duration of single invocation 396.29 ns 762.06 ns 1.92 0.01 False
Log2_ulong - Duration of single invocation 661.24 ns 5.76 μs 8.71 0.11 False
PopCount_ulong - Duration of single invocation 393.10 ns 991.15 ns 2.52 0.00 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BitOperations.Log2_uint


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.068070551373357 > 590.4776095649311.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -174.47162115911692 (T) = (0 -5134.6709385560725) / Math.Sqrt((2.129295899276432 / (30)) + (26099.15980276876 / (38))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (38) - 2, .025) and -8.13621135799764 = (562.0131515522891 - 5134.6709385560725) / 562.0131515522891 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 985.5805571480137 > 413.39812363644256.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -1780.7560343152702 (T) = (0 -986.650709538981) / Math.Sqrt((1.0358434123149214 / (30)) + (2.9825231910354155 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -1.5083019964203095 = (393.35403430171755 - 986.650709538981) / 393.35403430171755 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.PopCount_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 762.5707871256872 > 414.27920255393366.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -819.3642406221456 (T) = (0 -763.8301086861795) / Math.Sqrt((4.258602725129243 / (29)) + (2.132076404945987 / (39))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (39) - 2, .025) and -0.9288043859507215 = (396.0122209643785 - 763.8301086861795) / 396.0122209643785 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 762.059634245763 > 416.76240386667666.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -769.6917580990686 (T) = (0 -762.9330275182712) / Math.Sqrt((4.816246841893949 / (30)) + (2.5027707526853127 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.9165929101516619 = (398.06733264911196 - 762.9330275182712) / 398.06733264911196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.Log2_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.7620901760057475 > 694.1558815702053.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -115.0729870104527 (T) = (0 -5307.71323745104) / Math.Sqrt((2.3267998044654035 / (30)) + (63571.89594998186 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -7.021789668906295 = (661.6619802466977 - 5307.71323745104) / 661.6619802466977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.PopCount_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 991.1470007149413 > 412.91586513153925.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -1406.0311449078295 (T) = (0 -986.5106706700818) / Math.Sqrt((0.7114944230762575 / (29)) + (5.980295080109642 / (39))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (39) - 2, .025) and -1.5069297445718326 = (393.513489082069 - 986.5106706700818) / 393.513489082069 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 605d05ba54ccea0ff5486926a7dc630cee5db8cd
Compare ef0241ed9316bc4343e715daca4d88c1ff90231b
Diff Diff

Regressions in SIMD.RayTracerBench

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Bench - Duration of single invocation 264.54 ms 278.23 ms 1.05 0.12 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SIMD.RayTracerBench*'

Payloads

Baseline
Compare

Histogram

SIMD.RayTracerBench.Bench


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 278.23386653333336 > 274.60366500875.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -12.046987083462755 (T) = (0 -283216617.77246165) / Math.Sqrt((22437991164822.406 / (30)) + (67962349326024.69 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.07195918545318261 = (264204665.2669231 - 283216617.77246165) / 264204665.2669231 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 605d05ba54ccea0ff5486926a7dc630cee5db8cd
Compare ef0241ed9316bc4343e715daca4d88c1ff90231b
Diff Diff

Regressions in System.Collections.IndexerSetReverse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Span - Duration of single invocation 81.87 ns 158.14 ns 1.93 0.06 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSetReverse&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IndexerSetReverse<String>.Span(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 158.13722887851608 > 86.07958356953412.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -230.94519911821197 (T) = (0 -157.46604371176812) / Math.Sqrt((1.6813944950185855 / (29)) + (1.8704954154538767 / (39))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (39) - 2, .025) and -0.9133914642648328 = (82.2968256379622 - 157.46604371176812) / 82.2968256379622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 605d05ba54ccea0ff5486926a7dc630cee5db8cd
Compare ef0241ed9316bc4343e715daca4d88c1ff90231b
Diff Diff

Regressions in System.Net.Primitives.Tests.IPAddressPerformanceTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetAddressBytes - Duration of single invocation 32.81 ns 85.81 ns 2.62 0.47 False
Ctor_Span - Duration of single invocation 27.88 ns 106.76 ns 3.83 0.62 False
TryWriteBytes - Duration of single invocation 30.17 ns 83.28 ns 2.76 0.59 False
Ctor_Bytes - Duration of single invocation 35.10 ns 99.33 ns 2.83 0.54 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*'

Payloads

Baseline
Compare

Histogram

System.Net.Primitives.Tests.IPAddressPerformanceTests.GetAddressBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 85.80709392409577 > 37.17487266650631.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -54.34721839420134 (T) = (0 -89.51410323062314) / Math.Sqrt((5.68208174265619 / (30)) + (30.466003459880707 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -1.4884219235465466 = (35.97223701640031 - 89.51410323062314) / 35.97223701640031 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [16, 32, 48, 64, 80, ...])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.75993455005616 > 28.909951048556067.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -63.22911949403745 (T) = (0 -94.17354292883925) / Math.Sqrt((4.847315333432165 / (30)) + (36.66677318412625 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -2.386864095511788 = (27.805527553832572 - 94.17354292883925) / 27.805527553832572 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.TryWriteBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.28338875001727 > 26.574624959354292.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -73.88415004942134 (T) = (0 -82.28313738745834) / Math.Sqrt((5.709251910682509 / (30)) + (16.354092329816343 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -2.345576298810557 = (24.594607935473544 - 82.28313738745834) / 24.594607935473544 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Bytes(address: [16, 32, 48, 64, 80, ...])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.32686038016432 > 33.51097992913993.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -81.3548538018689 (T) = (0 -97.93560255792029) / Math.Sqrt((3.450421306542092 / (30)) + (20.53673333631725 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -1.9885229795121284 = (32.77057035509498 - 97.93560255792029) / 32.77057035509498 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions