Description
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 605d05ba54ccea0ff5486926a7dc630cee5db8cd |
Compare | ef0241ed9316bc4343e715daca4d88c1ff90231b |
Diff | Diff |
Regressions in System.Tests.Perf_String
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
IndexerCheckLengthHoisting - Duration of single invocation | 26.39 ns | 41.83 ns | 1.59 | 0.52 | False | |||||
ToLower - Duration of single invocation | 225.62 ns | 353.43 ns | 1.57 | 0.15 | False | |||||
CtorCharCount - Duration of single invocation | 59.70 ns | 77.58 ns | 1.30 | 0.32 | False | |||||
PadLeft - Duration of single invocation | 79.43 ns | 103.09 ns | 1.30 | 0.40 | False | |||||
ToUpper - Duration of single invocation | 224.64 ns | 377.85 ns | 1.68 | 0.16 | False | |||||
ToUpperInvariant - Duration of single invocation | 193.00 ns | 306.12 ns | 1.59 | 0.11 | False | |||||
ToLowerInvariant - Duration of single invocation | 205.16 ns | 303.99 ns | 1.48 | 0.10 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'
Payloads
Histogram
System.Tests.Perf_String.IndexerCheckLengthHoisting
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.82894696760661 > 27.703327538697756.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -32.191395005720636 (T) = (0 -39.227952731190776) / Math.Sqrt((2.2987827611203038 / (29)) + (2.130444531773961 / (39))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (39) - 2, .025) and -0.42914333445426117 = (27.448578309446148 - 39.227952731190776) / 27.448578309446148 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 353.4337818960568 > 232.82118714013097.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -40.634085757075816 (T) = (0 -362.1090027188929) / Math.Sqrt((20.297529142740533 / (30)) + (420.02307397867094 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.6119985744079897 = (224.6335750339471 - 362.1090027188929) / 224.6335750339471 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.CtorCharCount(size: 10)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 77.57732561551073 > 66.44490945482254.
IsChangePoint: Marked as a change because one of 1/18/2023 3:58:17 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -15.416099279258859 (T) = (0 -76.09339517367445) / Math.Sqrt((5.6031448767946035 / (30)) + (16.326698069258953 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.18713272215230525 = (64.09847336674788 - 76.09339517367445) / 64.09847336674788 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.PadLeft(n: 18)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 103.08926884112797 > 85.43390280392919.
IsChangePoint: Marked as a change because one of 1/14/2023 10:43:56 AM, 3/1/2023 5:53:50 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -9.68525725823996 (T) = (0 -98.39761883754494) / Math.Sqrt((16.14965701158545 / (31)) + (69.64983621634943 / (38))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (38) - 2, .025) and -0.17787499411268176 = (83.538252640867 - 98.39761883754494) / 83.538252640867 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 377.84674911210425 > 237.63925886733796.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -39.156693044760175 (T) = (0 -364.3480862142918) / Math.Sqrt((17.79434553648585 / (30)) + (475.4163860480634 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.6240293682749661 = (224.34821274279054 - 364.3480862142918) / 224.34821274279054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 306.12151880255936 > 203.10880620425826.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -50.77168771921563 (T) = (0 -303.4604725821685) / Math.Sqrt((17.974454628641865 / (30)) + (160.15292965899772 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.5696978903175034 = (193.3241259060286 - 303.4604725821685) / 193.3241259060286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 303.9909950767776 > 200.74636420269337.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -53.66505143777063 (T) = (0 -305.4980945172726) / Math.Sqrt((20.166320219948606 / (30)) + (140.9072518715231 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.5714330257579686 = (194.40732726736346 - 305.4980945172726) / 194.40732726736346 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 605d05ba54ccea0ff5486926a7dc630cee5db8cd |
Compare | ef0241ed9316bc4343e715daca4d88c1ff90231b |
Diff | Diff |
Regressions in System.Text.Tests.Perf_Encoding
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
GetByteCount - Duration of single invocation | 38.61 ns | 44.33 ns | 1.15 | 0.63 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*'
Payloads
Histogram
System.Text.Tests.Perf_Encoding.GetByteCount(size: 16, encName: "utf-8")
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 44.33202723180661 > 36.7779864338579.
IsChangePoint: Marked as a change because one of 1/17/2023 8:36:05 PM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -17.565347295110854 (T) = (0 -49.46813840273708) / Math.Sqrt((7.920079201110455 / (30)) + (11.12025465017022 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.35709294277888853 = (36.45154789578545 - 49.46813840273708) / 36.45154789578545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 605d05ba54ccea0ff5486926a7dc630cee5db8cd |
Compare | ef0241ed9316bc4343e715daca4d88c1ff90231b |
Diff | Diff |
Regressions in System.Numerics.Tests.Perf_BitOperations
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Log2_uint - Duration of single invocation | 563.73 ns | 5.07 μs | 8.99 | 0.12 | False | |||||
TrailingZeroCount_ulong - Duration of single invocation | 392.74 ns | 985.58 ns | 2.51 | 0.01 | False | |||||
PopCount_uint - Duration of single invocation | 393.95 ns | 762.57 ns | 1.94 | 0.00 | False | |||||
TrailingZeroCount_uint - Duration of single invocation | 396.29 ns | 762.06 ns | 1.92 | 0.01 | False | |||||
Log2_ulong - Duration of single invocation | 661.24 ns | 5.76 μs | 8.71 | 0.11 | False | |||||
PopCount_ulong - Duration of single invocation | 393.10 ns | 991.15 ns | 2.52 | 0.00 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'
Payloads
Histogram
System.Numerics.Tests.Perf_BitOperations.Log2_uint
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.068070551373357 > 590.4776095649311.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -174.47162115911692 (T) = (0 -5134.6709385560725) / Math.Sqrt((2.129295899276432 / (30)) + (26099.15980276876 / (38))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (38) - 2, .025) and -8.13621135799764 = (562.0131515522891 - 5134.6709385560725) / 562.0131515522891 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_ulong
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 985.5805571480137 > 413.39812363644256.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -1780.7560343152702 (T) = (0 -986.650709538981) / Math.Sqrt((1.0358434123149214 / (30)) + (2.9825231910354155 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -1.5083019964203095 = (393.35403430171755 - 986.650709538981) / 393.35403430171755 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_BitOperations.PopCount_uint
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 762.5707871256872 > 414.27920255393366.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -819.3642406221456 (T) = (0 -763.8301086861795) / Math.Sqrt((4.258602725129243 / (29)) + (2.132076404945987 / (39))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (39) - 2, .025) and -0.9288043859507215 = (396.0122209643785 - 763.8301086861795) / 396.0122209643785 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_BitOperations.TrailingZeroCount_uint
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 762.059634245763 > 416.76240386667666.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -769.6917580990686 (T) = (0 -762.9330275182712) / Math.Sqrt((4.816246841893949 / (30)) + (2.5027707526853127 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.9165929101516619 = (398.06733264911196 - 762.9330275182712) / 398.06733264911196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_BitOperations.Log2_ulong
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.7620901760057475 > 694.1558815702053.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -115.0729870104527 (T) = (0 -5307.71323745104) / Math.Sqrt((2.3267998044654035 / (30)) + (63571.89594998186 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -7.021789668906295 = (661.6619802466977 - 5307.71323745104) / 661.6619802466977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Numerics.Tests.Perf_BitOperations.PopCount_ulong
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 991.1470007149413 > 412.91586513153925.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -1406.0311449078295 (T) = (0 -986.5106706700818) / Math.Sqrt((0.7114944230762575 / (29)) + (5.980295080109642 / (39))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (39) - 2, .025) and -1.5069297445718326 = (393.513489082069 - 986.5106706700818) / 393.513489082069 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 605d05ba54ccea0ff5486926a7dc630cee5db8cd |
Compare | ef0241ed9316bc4343e715daca4d88c1ff90231b |
Diff | Diff |
Regressions in SIMD.RayTracerBench
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Bench - Duration of single invocation | 264.54 ms | 278.23 ms | 1.05 | 0.12 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SIMD.RayTracerBench*'
Payloads
Histogram
SIMD.RayTracerBench.Bench
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 278.23386653333336 > 274.60366500875.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -12.046987083462755 (T) = (0 -283216617.77246165) / Math.Sqrt((22437991164822.406 / (30)) + (67962349326024.69 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -0.07195918545318261 = (264204665.2669231 - 283216617.77246165) / 264204665.2669231 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 605d05ba54ccea0ff5486926a7dc630cee5db8cd |
Compare | ef0241ed9316bc4343e715daca4d88c1ff90231b |
Diff | Diff |
Regressions in System.Collections.IndexerSetReverse<String>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Span - Duration of single invocation | 81.87 ns | 158.14 ns | 1.93 | 0.06 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSetReverse<String>*'
Payloads
Histogram
System.Collections.IndexerSetReverse<String>.Span(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 158.13722887851608 > 86.07958356953412.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -230.94519911821197 (T) = (0 -157.46604371176812) / Math.Sqrt((1.6813944950185855 / (29)) + (1.8704954154538767 / (39))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (39) - 2, .025) and -0.9133914642648328 = (82.2968256379622 - 157.46604371176812) / 82.2968256379622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 605d05ba54ccea0ff5486926a7dc630cee5db8cd |
Compare | ef0241ed9316bc4343e715daca4d88c1ff90231b |
Diff | Diff |
Regressions in System.Net.Primitives.Tests.IPAddressPerformanceTests
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
GetAddressBytes - Duration of single invocation | 32.81 ns | 85.81 ns | 2.62 | 0.47 | False | |||||
Ctor_Span - Duration of single invocation | 27.88 ns | 106.76 ns | 3.83 | 0.62 | False | |||||
TryWriteBytes - Duration of single invocation | 30.17 ns | 83.28 ns | 2.76 | 0.59 | False | |||||
Ctor_Bytes - Duration of single invocation | 35.10 ns | 99.33 ns | 2.83 | 0.54 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*'
Payloads
Histogram
System.Net.Primitives.Tests.IPAddressPerformanceTests.GetAddressBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 85.80709392409577 > 37.17487266650631.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -54.34721839420134 (T) = (0 -89.51410323062314) / Math.Sqrt((5.68208174265619 / (30)) + (30.466003459880707 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -1.4884219235465466 = (35.97223701640031 - 89.51410323062314) / 35.97223701640031 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [16, 32, 48, 64, 80, ...])
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.75993455005616 > 28.909951048556067.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -63.22911949403745 (T) = (0 -94.17354292883925) / Math.Sqrt((4.847315333432165 / (30)) + (36.66677318412625 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -2.386864095511788 = (27.805527553832572 - 94.17354292883925) / 27.805527553832572 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.TryWriteBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.28338875001727 > 26.574624959354292.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -73.88415004942134 (T) = (0 -82.28313738745834) / Math.Sqrt((5.709251910682509 / (30)) + (16.354092329816343 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -2.345576298810557 = (24.594607935473544 - 82.28313738745834) / 24.594607935473544 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Bytes(address: [16, 32, 48, 64, 80, ...])
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.32686038016432 > 33.51097992913993.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsRegressionStdDev: Marked as regression because -81.3548538018689 (T) = (0 -97.93560255792029) / Math.Sqrt((3.450421306542092 / (30)) + (20.53673333631725 / (39))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (39) - 2, .025) and -1.9885229795121284 = (32.77057035509498 - 97.93560255792029) / 32.77057035509498 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository