Skip to content

[Perf] Linux/x64: 22 Regressions on 1/19/2023 7:53:02 PM #12123

Open
@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IterateForEachArray - Duration of single invocation 122.34 ns 168.58 ns 1.38 0.48 False
IterateGetPositionTenSegments - Duration of single invocation 567.79 ns 649.16 ns 1.14 0.30 False
SliceTenSegments - Duration of single invocation 116.74 ns 123.93 ns 1.06 0.34 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateForEachArray


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 168.57730147382318 > 129.4312299220948.
IsChangePoint: Marked as a change because one of 11/11/2022 3:15:52 PM, 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -7.991065657844798 (T) = (0 -150.12832464926575) / Math.Sqrt((39.48952697299523 / (32)) + (125.02324060651331 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.16277782883613565 = (129.11178810446907 - 150.12832464926575) / 129.11178810446907 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;.IterateGetPositionTenSegments

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 649.1550815292176 > 590.0567236408356.
IsChangePoint: Marked as a change because one of 1/18/2023 3:58:17 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -8.399963335329797 (T) = (0 -611.4948611668126) / Math.Sqrt((221.78149977044868 / (31)) + (367.3784059836182 / (23))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (23) - 2, .025) and -0.07073393605606147 = (571.0987954852642 - 611.4948611668126) / 571.0987954852642 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;.SliceTenSegments

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 123.93399282784121 > 123.78641176051869.
IsChangePoint: Marked as a change because one of 1/17/2023 2:19:41 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -5.511648651675112 (T) = (0 -131.6193788119163) / Math.Sqrt((34.47185217413856 / (27)) + (68.24448449170157 / (27))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (27) - 2, .025) and -0.08894142571645364 = (120.86910801957892 - 131.6193788119163) / 120.86910801957892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EmptyStaticFunction - Duration of single invocation 1.86 ms 2.08 ms 1.12 0.20 False
GenericClassGenericInstanceMethod - Duration of single invocation 288.07 μs 399.86 μs 1.39 0.06 False
IntegerFormatting - Duration of single invocation 6.43 ms 16.00 ms 2.49 0.22 False
GenericClassGenericStaticMethod - Duration of single invocation 525.26 μs 654.43 μs 1.25 0.13 False
GenericGenericMethod - Duration of single invocation 282.59 μs 407.45 μs 1.44 0.11 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.EmptyStaticFunction


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0777625026041666 > 1.9564735608333337.
IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -8.33904228229637 (T) = (0 -2092664.665958521) / Math.Sqrt((13224642035.058285 / (32)) + (4610432699.730079 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.11043203818290226 = (1884549.9715434476 - 2092664.665958521) / 1884549.9715434476 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.GenericClassGenericInstanceMethod

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 399.85781274038453 > 302.60327825.
IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -32.49655378639152 (T) = (0 -385545.59940948687) / Math.Sqrt((3722473.27127428 / (32)) + (193594183.73892084 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.33632609820108267 = (288511.61399039906 - 385545.59940948687) / 288511.61399039906 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.IntegerFormatting

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.999866671428572 > 6.750741233177083.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -85.51900747711608 (T) = (0 -15665665.741158683) / Math.Sqrt((88809789056.23221 / (14)) + (190736111824.69205 / (40))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (40) - 2, .025) and -1.3554463402301093 = (6650826.840584391 - 15665665.741158683) / 6650826.840584391 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 654.4288209635417 > 551.8575060757212.
IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -12.904246876630882 (T) = (0 -678746.1346516752) / Math.Sqrt((14108803.420922305 / (32)) + (3050603211.164123 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.28904395828650026 = (526550.0298018685 - 678746.1346516752) / 526550.0298018685 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.GenericGenericMethod

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 407.4453983516483 > 296.680255409727.
IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -27.281880825488344 (T) = (0 -421779.54325366556) / Math.Sqrt((26778.137048615015 / (32)) + (572677012.4046113 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.49258026122844734 = (282584.1626141604 - 421779.54325366556) / 282584.1626141604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Reverse - Duration of single invocation 24.07 ns 29.50 ns 1.23 0.33 False
Reverse - Duration of single invocation 13.92 ns 16.34 ns 1.17 0.49 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.Reverse(Size: 33)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.496475509301526 > 25.27195062677713.
IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -6.369546646014871 (T) = (0 -27.27771003379253) / Math.Sqrt((2.5862794141959067 / (32)) + (2.5748570077647295 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.11590645135749313 = (24.44444155745257 - 27.27771003379253) / 24.44444155745257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.Reverse(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.335283971111732 > 14.595210687065808.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/11/2023 4:46:06 PM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -7.246150159413201 (T) = (0 -16.257344985231487) / Math.Sqrt((0.6651465984984736 / (32)) + (1.7153993588341885 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.16288530149559166 = (13.98017926989261 - 16.257344985231487) / 13.98017926989261 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Collections.CtorDefaultSizeNonGeneric

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Hashtable - Duration of single invocation 33.20 ns 35.76 ns 1.08 0.55 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorDefaultSizeNonGeneric*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorDefaultSizeNonGeneric.Hashtable


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.75823674828893 > 34.60016184970855.
IsChangePoint: Marked as a change because one of 12/5/2022 8:20:45 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -7.092898541732629 (T) = (0 -37.7738977748661) / Math.Sqrt((2.704561306230093 / (32)) + (5.5736301002327835 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1225169092093533 = (33.65107239361963 - 37.7738977748661) / 33.65107239361963 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Max - Duration of single invocation 2.20 μs 2.55 μs 1.16 0.24 False
Min - Duration of single invocation 2.16 μs 2.53 μs 1.17 0.22 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5542139839806897 > 2.2965649845619986.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -15.545053296030265 (T) = (0 -2599.3393038626728) / Math.Sqrt((5964.80182199081 / (32)) + (8670.259667415621 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1683460289006842 = (2224.802618029552 - 2599.3393038626728) / 2224.802618029552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.52573214617615 > 2.2845308787465597.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -13.94699586344723 (T) = (0 -2592.7625435435334) / Math.Sqrt((4919.536191229149 / (32)) + (14450.998808596494 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.18084862529589368 = (2195.6773188382604 - 2592.7625435435334) / 2195.6773188382604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Collections.Concurrent.IsEmpty<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Bag - Duration of single invocation 46.02 ns 55.31 ns 1.20 0.65 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Concurrent.IsEmpty&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Concurrent.IsEmpty<Int32>.Bag(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.31339062338936 > 50.430543768541376.
IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -4.010250998350441 (T) = (0 -56.00902489524304) / Math.Sqrt((20.5846307653969 / (32)) + (29.086145886795244 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1115770780520157 = (50.38699160061496 - 56.00902489524304) / 50.38699160061496 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Next_int_unseeded - Duration of single invocation 31.54 ns 36.32 ns 1.15 0.68 False
Next_int - Duration of single invocation 31.08 ns 39.53 ns 1.27 0.67 False
Next_long - Duration of single invocation 92.72 ns 101.65 ns 1.10 0.34 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Random*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Random.Next_int_unseeded


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.31676164278097 > 32.450065407653106.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -7.761320066892196 (T) = (0 -36.77698501590825) / Math.Sqrt((4.1244260360284954 / (32)) + (8.120823073221315 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.17499110542221658 = (31.299798650554855 - 36.77698501590825) / 31.299798650554855 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Random.Next_int

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.527610167892746 > 34.15863407003011.
IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -8.133106537691837 (T) = (0 -35.36393237750524) / Math.Sqrt((5.645423626696244 / (32)) + (6.777199158960745 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1905864814717677 = (29.702951383917444 - 35.36393237750524) / 29.702951383917444 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Random.Next_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 101.64717876401252 > 97.65172483793113.
IsChangePoint: Marked as a change because one of 1/18/2023 3:58:17 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -7.849730294836891 (T) = (0 -102.9104190573666) / Math.Sqrt((9.193992187457193 / (31)) + (15.929321628870577 / (23))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (23) - 2, .025) and -0.08209034779825782 = (95.10335182894816 - 102.9104190573666) / 95.10335182894816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in PerfLabTests.DictionaryExpansion

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExpandDictionaries - Duration of single invocation 1.54 μs 1.78 μs 1.16 0.06 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.DictionaryExpansion*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.DictionaryExpansion.ExpandDictionaries


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7846032902529425 > 1.619365508920234.
IsChangePoint: Marked as a change because one of 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -19.62660129151795 (T) = (0 -1764.8116436097232) / Math.Sqrt((671.913963459265 / (32)) + (2087.395903005932 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.1359958826298745 = (1553.5370071272757 - 1764.8116436097232) / 1553.5370071272757 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 19fc478d67c2fc55e62b50676935f903a77f5a4d
Compare 8300e381e41bdf013c5dc697e3e9c15f68e8f249
Diff Diff

Regressions in System.Tests.Perf_HashCode

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Add - Duration of single invocation 7.47 μs 10.35 μs 1.38 0.20 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_HashCode*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_HashCode.Add


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.349680475694125 > 7.89002947445516.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/18/2023 9:06:44 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -31.23939466955221 (T) = (0 -9753.146522303574) / Math.Sqrt((121458.9151308833 / (32)) + (33038.97876432847 / (22))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (22) - 2, .025) and -0.30399245155123705 = (7479.450138458374 - 9753.146522303574) / 7479.450138458374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions