Description
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | f70d8a9663d3d674c453e26d405dc13baa79fd16 |
Compare | 4c25992c7b25c1ab21dc47e8dcba9f491b5fcb80 |
Diff | Diff |
Regressions in System.Xml.Linq.Perf_XName
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
EmptyNameSpaceToString - Duration of single invocation | 43.32 ns | 47.13 ns | 1.09 | 0.18 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Xml.Linq.Perf_XName*'
Payloads
Histogram
System.Xml.Linq.Perf_XName.EmptyNameSpaceToString
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 47.12856610035319 > 45.305691024157575.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -11.29444062655892 (T) = (0 -48.74874085802872) / Math.Sqrt((1.0533304790869424 / (29)) + (4.2016003052945825 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.10479703876804816 = (44.124612166220246 - 48.74874085802872) / 44.124612166220246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | f70d8a9663d3d674c453e26d405dc13baa79fd16 |
Compare | 4c25992c7b25c1ab21dc47e8dcba9f491b5fcb80 |
Diff | Diff |
Regressions in Benchstone.BenchF.InProd
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Test - Duration of single invocation | 18.61 secs | 26.21 secs | 1.41 | 0.06 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchF.InProd*'
Payloads
Histogram
Benchstone.BenchF.InProd.Test
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.205195733066667 > 19.305948829965004.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -98.85894439391814 (T) = (0 -26606051681.260426) / Math.Sqrt((79270009519463810 / (28)) + (1.3830588599464738E+17 / (32))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (32) - 2, .025) and -0.45827020401135216 = (18244939523.603752 - 26606051681.260426) / 18244939523.603752 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | f70d8a9663d3d674c453e26d405dc13baa79fd16 |
Compare | 4c25992c7b25c1ab21dc47e8dcba9f491b5fcb80 |
Diff | Diff |
Regressions in System.Collections.Tests.Perf_BitArray
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
BitArrayOr - Duration of single invocation | 95.14 ns | 103.63 ns | 1.09 | 0.15 | False | |||||
BitArrayOr - Duration of single invocation | 1.80 μs | 1.93 μs | 1.07 | 0.06 | False | |||||
BitArrayXor - Duration of single invocation | 1.81 μs | 2.00 μs | 1.11 | 0.10 | False | |||||
BitArrayXor - Duration of single invocation | 95.92 ns | 101.78 ns | 1.06 | 0.13 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'
Payloads
Histogram
System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 4)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 103.62949515046421 > 100.07630528756027.
IsChangePoint: Marked as a change because one of 12/9/2022 5:51:37 PM, 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -12.014747119256125 (T) = (0 -102.37460327165117) / Math.Sqrt((2.184662355404025 / (29)) + (6.14196699776061 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.06459226822931405 = (96.16320381691857 - 102.37460327165117) / 96.16320381691857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9290554918015517 > 1.8975577560223353.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -20.42802626838268 (T) = (0 -1939.143159026297) / Math.Sqrt((435.05694358306727 / (29)) + (829.9598587576104 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.07227482792012163 = (1808.4385724018434 - 1939.143159026297) / 1808.4385724018434 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.000241374022836 > 1.8938687484486685.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -10.202588483403126 (T) = (0 -1960.1590722328583) / Math.Sqrt((2533.248316028486 / (29)) + (3138.579466608735 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.07628535383584165 = (1821.2261880614867 - 1960.1590722328583) / 1821.2261880614867 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 4)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 101.77632354872301 > 101.0245137145482.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -5.334123898569263 (T) = (0 -103.99761606544608) / Math.Sqrt((14.331189496113264 / (28)) + (37.50876469712356 / (32))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (32) - 2, .025) and -0.07130516751859066 = (97.07562253837573 - 103.99761606544608) / 97.07562253837573 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | f70d8a9663d3d674c453e26d405dc13baa79fd16 |
Compare | 4c25992c7b25c1ab21dc47e8dcba9f491b5fcb80 |
Diff | Diff |
Regressions in Benchstone.BenchI.Puzzle
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Test - Duration of single invocation | 6.03 secs | 11.30 secs | 1.87 | 0.05 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.Puzzle*'
Payloads
Histogram
Benchstone.BenchI.Puzzle.Test
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.29619129417647 > 6.3421423501500005.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -109.78694762377619 (T) = (0 -11015936087.42787) / Math.Sqrt((8385305480076015 / (29)) + (55974746814263450 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.8180350933464877 = (6059253821.7459 - 11015936087.42787) / 6059253821.7459 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | f70d8a9663d3d674c453e26d405dc13baa79fd16 |
Compare | 4c25992c7b25c1ab21dc47e8dcba9f491b5fcb80 |
Diff | Diff |
Regressions in Benchstone.BenchI.BubbleSort2
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Test - Duration of single invocation | 443.09 ms | 535.81 ms | 1.21 | 0.10 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.BubbleSort2*'
Payloads
Histogram
Benchstone.BenchI.BubbleSort2.Test
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 535.8078668 > 465.49965864500007.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -28.213373271552154 (T) = (0 -544264616.5287261) / Math.Sqrt((60042265154224.39 / (29)) + (337508556266910.3 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.2256906502175292 = (444047294.01511943 - 544264616.5287261) / 444047294.01511943 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | f70d8a9663d3d674c453e26d405dc13baa79fd16 |
Compare | 4c25992c7b25c1ab21dc47e8dcba9f491b5fcb80 |
Diff | Diff |
Regressions in System.MathBenchmarks.Single
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Min - Duration of single invocation | 110.45 μs | 129.06 μs | 1.17 | 0.06 | False | |||||
Exp2 - Duration of single invocation | 137.63 μs | 147.61 μs | 1.07 | 0.08 | False | |||||
Sin - Duration of single invocation | 86.22 μs | 91.85 μs | 1.07 | 0.10 | False | |||||
Max - Duration of single invocation | 110.45 μs | 128.57 μs | 1.16 | 0.08 | False | |||||
Log2P1 - Duration of single invocation | 106.47 μs | 113.69 μs | 1.07 | 0.06 | False | |||||
Tan - Duration of single invocation | 90.58 μs | 120.33 μs | 1.33 | 0.03 | False | |||||
Cos - Duration of single invocation | 85.60 μs | 99.38 μs | 1.16 | 0.07 | False | |||||
ScaleB - Duration of single invocation | 74.65 μs | 79.24 μs | 1.06 | 0.08 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'
Payloads
Histogram
System.MathBenchmarks.Single.Min
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.0599173553719 > 115.97761081527626.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -35.62830717838659 (T) = (0 -131017.1516623779) / Math.Sqrt((1187981.5652213139 / (29)) + (9080547.290670648 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.18338091162978007 = (110714.26822487613 - 131017.1516623779) / 110714.26822487613 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.MathBenchmarks.Single.Exp2
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.6123035377358 > 144.9073773005757.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -10.240157716071355 (T) = (0 -147072.60157280142) / Math.Sqrt((12384330.808553386 / (29)) + (5734946.536912625 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.057320789505229446 = (139099.31879957044 - 147072.60157280142) / 139099.31879957044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.MathBenchmarks.Single.Sin
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 91.85321671539961 > 90.32558836710166.
IsChangePoint: Marked as a change because one of 12/6/2022 5:44:36 AM, 12/6/2022 6:34:01 PM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -13.91668536779783 (T) = (0 -93175.56215613037) / Math.Sqrt((251569.76357284622 / (29)) + (8216477.5893470645 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.08336662382377677 = (86005.56829715159 - 93175.56215613037) / 86005.56829715159 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.MathBenchmarks.Single.Max
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 128.5740997877635 > 115.94916123899647.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -38.318184388149554 (T) = (0 -131306.5915591391) / Math.Sqrt((65203.113795019584 / (29)) + (9368574.3362931 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.18836039537219082 = (110493.91419512453 - 131306.5915591391) / 110493.91419512453 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.MathBenchmarks.Single.Log2P1
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 113.6920294384058 > 112.03917543681796.
IsChangePoint: Marked as a change because one of 1/11/2023 8:19:46 AM, 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -3.3822815193564724 (T) = (0 -115535.39155212583) / Math.Sqrt((25153.149158311913 / (29)) + (219235723.60336426 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.08299032181273389 = (106681.83198418621 - 115535.39155212583) / 106681.83198418621 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.MathBenchmarks.Single.Tan
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 120.33003201923077 > 94.83769554479674.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -60.32723732608465 (T) = (0 -117009.29575659323) / Math.Sqrt((793621.1906637271 / (28)) + (5171392.163162273 / (32))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (32) - 2, .025) and -0.2898312243901014 = (90716.74924905098 - 117009.29575659323) / 90716.74924905098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.MathBenchmarks.Single.Cos
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.37526376582281 > 90.02081162909836.
IsChangePoint: Marked as a change because one of 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -27.984776883482695 (T) = (0 -100741.71289911905) / Math.Sqrt((1189897.623023747 / (28)) + (6793155.895700418 / (32))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (32) - 2, .025) and -0.16308269506754633 = (86616.12224680933 - 100741.71289911905) / 86616.12224680933 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.MathBenchmarks.Single.ScaleB
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.2446113140332 > 78.41893136904761.
IsChangePoint: Marked as a change because one of 1/9/2023 3:58:12 PM, 1/10/2023 7:48:18 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -6.677061858928303 (T) = (0 -80407.85267025419) / Math.Sqrt((6208538.547997926 / (29)) + (12194889.0000175 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.06844849361567135 = (75256.64844933318 - 80407.85267025419) / 75256.64844933318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | f70d8a9663d3d674c453e26d405dc13baa79fd16 |
Compare | 4c25992c7b25c1ab21dc47e8dcba9f491b5fcb80 |
Diff | Diff |
Regressions in System.Collections.ContainsFalse<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
ImmutableSortedSet - Duration of single invocation | 827.74 μs | 893.49 μs | 1.08 | 0.06 | False | |||||
ImmutableList - Duration of single invocation | 44.52 ms | 50.07 ms | 1.12 | 0.05 | False | |||||
LinkedList - Duration of single invocation | 27.98 ms | 33.33 ms | 1.19 | 0.05 | False | |||||
SortedSet - Duration of single invocation | 622.65 μs | 674.99 μs | 1.08 | 0.07 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse<Int32>*'
Payloads
Histogram
System.Collections.ContainsFalse<Int32>.ImmutableSortedSet(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 893.4863608630951 > 870.490178582996.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -14.320945430167825 (T) = (0 -899792.0947261108) / Math.Sqrt((211506859.7459526 / (29)) + (357964331.8439371 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.07344406241391402 = (838229.1413515277 - 899792.0947261108) / 838229.1413515277 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsFalse<Int32>.ImmutableList(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.072538442307696 > 47.08204081583334.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -19.560573979402264 (T) = (0 -50343736.79809124) / Math.Sqrt((1054678442820.3978 / (29)) + (1048003617175.7091 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.11377015710181639 = (45201190.27887458 - 50343736.79809124) / 45201190.27887458 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsFalse<Int32>.LinkedList(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.32608568571429 > 29.46462736463675.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -34.487731545476066 (T) = (0 -32234575.169746693) / Math.Sqrt((153684958498.49475 / (28)) + (247336356285.17484 / (32))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (32) - 2, .025) and -0.14025844074155863 = (28269534.359932628 - 32234575.169746693) / 28269534.359932628 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsFalse<Int32>.SortedSet(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 674.9908859747025 > 654.682306175.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -13.258362811517882 (T) = (0 -676666.5838976173) / Math.Sqrt((285919041.0273229 / (29)) + (78795918.97171909 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.07385780139094102 = (630126.8035871678 - 676666.5838976173) / 630126.8035871678 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository