[Perf] Linux/x64: 3 Regressions on 9/1/2022 12:26:41 AM #8317
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 51191575ccced7759ab2b82c7348a488f56d6f52 |
| Compare | d684343d7fc6e050d5c7fa11298dc1b6a5ee7a0e |
| Diff | Diff |
Regressions in System.Net.Primitives.Tests.IPAddressPerformanceTests
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryFormat - Duration of single invocation | 112.02 ns | 119.94 ns | 1.07 | 0.02 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*'Payloads
Histogram
System.Net.Primitives.Tests.IPAddressPerformanceTests.TryFormat(address: 1020:3040:5060:7080:9010:1112:1314:1516)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 119.93998396079188 > 117.82630695007674.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/6/2022 3:55:24 AM, 7/12/2022 12:16:14 PM, 8/18/2022 11:13:04 AM, 8/31/2022 7:50:47 PM, 9/5/2022 11:32:52 PM falls between 8/28/2022 2:29:57 AM and 9/5/2022 11:32:52 PM.
IsRegressionStdDev: Marked as regression because -22.945115419504816 (T) = (0 -119.75370479929217) / Math.Sqrt((0.9409967420817432 / (36)) + (1.5932731666297995 / (24))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (24) - 2, .025) and -0.06188853321854291 = (112.77427060665524 - 119.75370479929217) / 112.77427060665524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 51191575ccced7759ab2b82c7348a488f56d6f52 |
| Compare | d684343d7fc6e050d5c7fa11298dc1b6a5ee7a0e |
| Diff | Diff |
Regressions in System.Text.Tests.Perf_StringBuilder
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Append_Char - Duration of single invocation | 105.07 μs | 183.01 μs | 1.74 | 0.01 | False | |||||
| Append_Char - Duration of single invocation | 175.32 ns | 260.28 ns | 1.48 | 0.03 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Tests.Perf_StringBuilder*'Payloads
Histogram
System.Text.Tests.Perf_StringBuilder.Append_Char(length: 100000)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 183.01028106831396 > 110.20548391450818.
IsChangePoint: Marked as a change because one of 7/6/2022 7:05:00 PM, 8/31/2022 7:50:47 PM, 9/5/2022 11:32:52 PM falls between 8/28/2022 2:29:57 AM and 9/5/2022 11:32:52 PM.
IsRegressionStdDev: Marked as regression because -507.1242249385458 (T) = (0 -182739.4332902559) / Math.Sqrt((636707.7649472884 / (35)) + (119437.4865538021 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -0.73723363959139 = (105189.90026766782 - 182739.4332902559) / 105189.90026766782 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Tests.Perf_StringBuilder.Append_Char(length: 100)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 260.2849208108184 > 183.2933804805217.
IsChangePoint: Marked as a change because one of 7/6/2022 7:05:00 PM, 8/31/2022 7:50:47 PM, 9/5/2022 11:32:52 PM falls between 8/28/2022 2:29:57 AM and 9/5/2022 11:32:52 PM.
IsRegressionStdDev: Marked as regression because -88.80643570635883 (T) = (0 -265.47340888979835) / Math.Sqrt((26.025530996279194 / (35)) + (6.697065425369016 / (24))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (24) - 2, .025) and -0.5112248832952091 = (175.6677062588703 - 265.47340888979835) / 175.6677062588703 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository