Skip to content

[Perf] Linux/x64: 16 Regressions on 9/22/2022 3:16:12 AM #77900

Closed
dotnet/performance
#2703

Description

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 34c522b7e9eaa9bddb6d93ba471cf1c7d4ff3266
Compare b5b2af07114a7e94c82676ddd23ff51af1bed0d5
Diff Diff

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 969.31 μs 1.10 ms 1.13 0.07 False
ImmutableArray - Duration of single invocation 996.73 μs 1.11 ms 1.12 0.05 False
ICollection - Duration of single invocation 967.59 μs 1.06 ms 1.10 0.05 False
List - Duration of single invocation 954.73 μs 1.06 ms 1.11 0.06 False
Array - Duration of single invocation 974.99 μs 1.07 ms 1.10 0.06 False
Stack - Duration of single invocation 1.00 ms 1.24 ms 1.23 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.ContainsTrue<Int32>.Queue(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.095966200892857 > 1010.798569984976.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -26.205242060142314 (T) = (0 -1082176.5961467223) / Math.Sqrt((134557800.542629 / (33)) + (378753722.89546967 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.11677096832909976 = (969022.8586134029 - 1082176.5961467223) / 969022.8586134029 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1139794651785715 > 1.060484347207031.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -27.791974606900073 (T) = (0 -1115382.3246755877) / Math.Sqrt((180256514.8127183 / (33)) + (235012253.979763 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.10483220306009597 = (1009549.0714212264 - 1115382.3246755877) / 1009549.0714212264 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0617160416666667 > 1018.8446187011718.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -16.056018293701424 (T) = (0 -1087710.4426824627) / Math.Sqrt((185033592.69745567 / (33)) + (1162193806.342326 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.11694145311763579 = (973829.4157195234 - 1087710.4426824627) / 973829.4157195234 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0611418856944443 > 1018.2846668750002.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -35.482093782068404 (T) = (0 -1074002.2283184354) / Math.Sqrt((69967824.32398583 / (33)) + (179800464.4794185 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.11025724102303357 = (967345.3940537317 - 1074002.2283184354) / 967345.3940537317 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.Array(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0702191419642857 > 1023.6990125195313.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -30.120564854575687 (T) = (0 -1086000.919079096) / Math.Sqrt((291634961.6495431 / (33)) + (86090478.21488371 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.10701630982952783 = (981016.1868765348 - 1086000.919079096) / 981016.1868765348 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.Stack(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2352883976648354 > 1024.4168104882813.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -60.858087292552526 (T) = (0 -1236941.1502558445) / Math.Sqrt((337610251.7351471 / (33)) + (203005793.13338426 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.2641636960719056 = (978465.9645735368 - 1236941.1502558445) / 978465.9645735368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 34c522b7e9eaa9bddb6d93ba471cf1c7d4ff3266
Compare b5b2af07114a7e94c82676ddd23ff51af1bed0d5
Diff Diff

Regressions in System.Memory.SequenceReader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryReadTo - Duration of single invocation 4.81 μs 5.38 μs 1.12 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.SequenceReader*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Memory.SequenceReader.TryReadTo


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.378923124141483 > 5.053520181621476.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -51.38447919574725 (T) = (0 -5367.488921550735) / Math.Sqrt((2937.7007674825045 / (32)) + (472.35950356068855 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.11159539603981007 = (4828.635437563927 - 5367.488921550735) / 4828.635437563927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 34c522b7e9eaa9bddb6d93ba471cf1c7d4ff3266
Compare b5b2af07114a7e94c82676ddd23ff51af1bed0d5
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 1.89 ms 2.38 ms 1.26 0.06 False
ImmutableArray - Duration of single invocation 1.93 ms 2.14 ms 1.11 0.03 False
Array - Duration of single invocation 1.90 ms 2.11 ms 1.11 0.03 False
List - Duration of single invocation 1.89 ms 2.10 ms 1.11 0.05 False
Queue - Duration of single invocation 1.89 ms 2.23 ms 1.18 0.06 False
ICollection - Duration of single invocation 1.87 ms 2.10 ms 1.12 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.ContainsFalse<Int32>.Stack(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3842154708333334 > 1.9866159997796478.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -63.97195320199993 (T) = (0 -2394094.2410446103) / Math.Sqrt((1695724547.8362103 / (33)) + (190835740.57041073 / (25))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (25) - 2, .025) and -0.2583031159565782 = (1902637.1394022887 - 2394094.2410446103) / 1902637.1394022887 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.1394566171875 > 2.0256352184960935.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -31.077029944721016 (T) = (0 -2151332.1590478206) / Math.Sqrt((360162614.46956015 / (33)) + (997867569.4428535 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.11286796194600314 = (1933142.3247066252 - 2151332.1590478206) / 1933142.3247066252 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Array(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.1087881843750003 > 1.9809466521310766.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -45.259498036275815 (T) = (0 -2113783.74757238) / Math.Sqrt((353745729.22009766 / (33)) + (319361059.55712956 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.11444507736727372 = (1896714.1499390076 - 2113783.74757238) / 1896714.1499390076 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.1026052706473215 > 1.9608499316145835.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -68.80938666641624 (T) = (0 -2099614.6035013716) / Math.Sqrt((211620946.8627083 / (33)) + (92498111.32462098 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.11542629051873408 = (1882342.7611025164 - 2099614.6035013716) / 1882342.7611025164 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2304079871651785 > 2.008788108359375.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -23.59913129326589 (T) = (0 -2147578.562214543) / Math.Sqrt((1116474284.4023762 / (33)) + (1587306697.1320746 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.11986933064211758 = (1917704.5959309835 - 2147578.562214543) / 1917704.5959309835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0979382890625 > 1.9806727393359376.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -42.85148744252998 (T) = (0 -2103699.1899045617) / Math.Sqrt((347858606.2090312 / (33)) + (396049417.258107 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.11533974964320046 = (1886151.005178054 - 2103699.1899045617) / 1886151.005178054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 34c522b7e9eaa9bddb6d93ba471cf1c7d4ff3266
Compare b5b2af07114a7e94c82676ddd23ff51af1bed0d5
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Contains_ElementNotFound - Duration of single invocation 832.17 ns 895.38 ns 1.08 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Linq.Tests.Perf_Enumerable.Contains_ElementNotFound(input: ICollection)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 895.3778104156972 > 849.9629668568364.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -29.21081282954959 (T) = (0 -892.9335262280066) / Math.Sqrt((168.6694102042165 / (33)) + (82.54061160444941 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.10395462812604744 = (808.8498417219853 - 892.9335262280066) / 808.8498417219853 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 34c522b7e9eaa9bddb6d93ba471cf1c7d4ff3266
Compare b5b2af07114a7e94c82676ddd23ff51af1bed0d5
Diff Diff

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOfValue - Duration of single invocation 1.87 μs 2.49 μs 1.33 0.06 False
IndexOfValue - Duration of single invocation 1.98 μs 2.11 μs 1.07 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Memory.Span<Int32>.LastIndexOfValue(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.486075723145284 > 1.961551330630418.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -32.74534469892047 (T) = (0 -2378.1927078714825) / Math.Sqrt((666.1354551089895 / (33)) + (5810.079235983002 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.2737630503998014 = (1867.0605236390152 - 2378.1927078714825) / 1867.0605236390152 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Int32&gt;.IndexOfValue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.1059525350629564 > 1.9711226404728943.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/21/2022 9:54:19 PM, 9/27/2022 7:01:57 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 7:01:57 AM.
IsRegressionStdDev: Marked as regression because -26.75366215584882 (T) = (0 -2108.271736496125) / Math.Sqrt((824.5871390603775 / (33)) + (1462.8924148464291 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.1291609019154783 = (1867.1136530849672 - 2108.271736496125) / 1867.1136530849672 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions