Closed
Description
Run Information
Architecture | arm64 |
---|---|
OS | ubuntu 20.04 |
Baseline | c9f80fdde29c71ac87971383c77a1a5e665fadb9 |
Compare | a38b50c3a2f828458d15e68c78460db3ec7e5cda |
Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Deep
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
WriteDeepUtf16 - Duration of single invocation | 4.93 ms | 5.29 ms | 1.07 | 0.00 | False | |||||
WriteDeepUtf8 - Duration of single invocation | 4.97 ms | 5.26 ms | 1.06 | 0.00 | False | |||||
WriteDeepUtf16 - Duration of single invocation | 5.25 ms | 5.55 ms | 1.06 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Deep*'
Payloads
Histogram
System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.2924019456521725 > 5.195116339699999.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -62.883214080844944 (T) = (0 -5271577.21509731) / Math.Sqrt((185315176.39245772 / (32)) + (235544487.6445184 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.0639908960754851 = (4954532.25637686 - 5271577.21509731) / 4954532.25637686 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.25883959710145 > 5.171207155182354.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -58.01295726028566 (T) = (0 -5242644.245942542) / Math.Sqrt((197904450.00905737 / (32)) + (297749108.7652593 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.06565223672943832 = (4919657.713132181 - 5242644.245942542) / 4919657.713132181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.546248321212121 > 5.525902827765958.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -38.37097812747013 (T) = (0 -5570782.904317904) / Math.Sqrt((269717487.2190087 / (32)) + (695229046.3110473 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.059447521740249476 = (5258196.173008486 - 5570782.904317904) / 5258196.173008486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Architecture | arm64 |
---|---|
OS | ubuntu 20.04 |
Baseline | c9f80fdde29c71ac87971383c77a1a5e665fadb9 |
Compare | a38b50c3a2f828458d15e68c78460db3ec7e5cda |
Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Basic
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
WriteBasicUtf16 - Duration of single invocation | 2.13 ms | 2.41 ms | 1.13 | 0.00 | False | |||||
WriteBasicUtf8 - Duration of single invocation | 2.12 ms | 2.39 ms | 1.12 | 0.00 | False | |||||
WriteBasicUtf8 - Duration of single invocation | 2.84 ms | 3.09 ms | 1.09 | 0.00 | False | |||||
WriteBasicUtf16 - Duration of single invocation | 3.01 ms | 3.28 ms | 1.09 | 0.00 | False | |||||
WriteBasicUtf16 - Duration of single invocation | 2.28 ms | 2.55 ms | 1.12 | 0.00 | False | |||||
WriteBasicUtf16 - Duration of single invocation | 2.85 ms | 3.12 ms | 1.09 | 0.00 | False | |||||
WriteBasicUtf8 - Duration of single invocation | 3.03 ms | 3.27 ms | 1.08 | 0.00 | False | |||||
WriteBasicUtf8 - Duration of single invocation | 2.27 ms | 2.56 ms | 1.12 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'
Payloads
Histogram
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.413046492905153 > 2.2323625869395554.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -345.39885295672553 (T) = (0 -2415661.9224883527) / Math.Sqrt((11466036.577396711 / (32)) + (4055508.360646513 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.13547246025008064 = (2127450.913213975 - 2415661.9224883527) / 2127450.913213975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3869956682692304 > 2.2289094366944817.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -152.0413470626112 (T) = (0 -2394699.9200306633) / Math.Sqrt((14084111.144689558 / (32)) + (32800381.06979755 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.1275283995121029 = (2123848.87251344 - 2394699.9200306633) / 2123848.87251344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0888318115226343 > 2.9820909594602276.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -165.20487949464604 (T) = (0 -3102876.479503347) / Math.Sqrt((9167201.45048848 / (32)) + (26419804.084898308 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.09168351763717046 = (2842285.7260125936 - 3102876.479503347) / 2842285.7260125936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.278109338596491 > 3.162077486470761.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -263.1311054031104 (T) = (0 -3276587.661000043) / Math.Sqrt((1732394.4003758235 / (32)) + (11484271.472290196 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.08784726663213838 = (3011992.364648777 - 3276587.661000043) / 3011992.364648777 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5517615633284234 > 2.394371170527523.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -401.51257075080696 (T) = (0 -2553628.3245032714) / Math.Sqrt((3139674.47373653 / (32)) + (4375365.40976178 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.1197654962107207 = (2280502.77771974 - 2553628.3245032714) / 2280502.77771974 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.1181703383333335 > 2.9931869454022983.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -143.7329587356498 (T) = (0 -3112665.3868698566) / Math.Sqrt((5652288.301119157 / (32)) + (37752435.64427367 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.09190806637328912 = (2850666.1711991914 - 3112665.3868698566) / 2850666.1711991914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.2734030062222224 > 3.180161731956315.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -65.035228030537 (T) = (0 -3281967.6176546924) / Math.Sqrt((5025270.405031393 / (32)) + (178131450.1298167 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.08313041922842368 = (3030076.119543044 - 3281967.6176546924) / 3030076.119543044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.555302539027982 > 2.38507844843007.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -351.0938854483454 (T) = (0 -2556497.161926386) / Math.Sqrt((14760943.328055177 / (32)) + (2310805.7294822093 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.12492179784800715 = (2272599.8970035114 - 2556497.161926386) / 2272599.8970035114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | arm64 |
---|---|
OS | ubuntu 20.04 |
Baseline | c9f80fdde29c71ac87971383c77a1a5e665fadb9 |
Compare | a38b50c3a2f828458d15e68c78460db3ec7e5cda |
Diff | Diff |
Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
MultipleSerial - Duration of single invocation | 351.77 ns | 378.77 ns | 1.08 | 0.05 | False | |||||
SingleParallel - Duration of single invocation | 7.32 μs | 7.97 μs | 1.09 | 0.64 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests<Object>*'
Payloads
Histogram
System.Buffers.Tests.RentReturnArrayPoolTests<Object>.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 378.76595033333336 > 369.10307163333334.
IsChangePoint: Marked as a change because one of 6/19/2022 10:30:09 AM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -35.54303301900895 (T) = (0 -378.6799848611111) / Math.Sqrt((15.14520057775964 / (32)) + (0.9888827785107522 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.07523194857242699 = (352.18446156094984 - 378.6799848611111) / 352.18446156094984 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Tests.RentReturnArrayPoolTests<Object>.SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.971872746499999 > 6.724874629200001.
IsChangePoint: Marked as a change because one of 5/30/2022 4:33:23 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -8.521013743535269 (T) = (0 -8200.837295708334) / Math.Sqrt((235835.19010466183 / (32)) + (449876.24444322934 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.28216603074986774 = (6396.0806159496515 - 8200.837295708334) / 6396.0806159496515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | arm64 |
---|---|
OS | ubuntu 20.04 |
Baseline | c9f80fdde29c71ac87971383c77a1a5e665fadb9 |
Compare | a38b50c3a2f828458d15e68c78460db3ec7e5cda |
Diff | Diff |
Regressions in System.Buffers.Text.Tests.Utf8FormatterTests
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
FormatterInt32 - Duration of single invocation | 9.82 ns | 15.97 ns | 1.63 | 0.16 | False | |||||
FormatterInt64 - Duration of single invocation | 10.42 ns | 16.10 ns | 1.55 | 0.20 | False | |||||
FormatterInt32 - Duration of single invocation | 3.01 ns | 6.25 ns | 2.08 | 0.45 | False | |||||
FormatterInt32 - Duration of single invocation | 23.21 ns | 28.53 ns | 1.23 | 0.02 | False | |||||
FormatterInt64 - Duration of single invocation | 48.79 ns | 54.49 ns | 1.12 | 0.00 | False | |||||
FormatterInt32 - Duration of single invocation | 21.64 ns | 27.80 ns | 1.28 | 0.02 | False | |||||
FormatterInt64 - Duration of single invocation | 51.93 ns | 55.85 ns | 1.08 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'
Payloads
Histogram
System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.972661986380423 > 10.892674528760839.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -54.720373930164925 (T) = (0 -15.819675463730695) / Math.Sqrt((0.23147261689839174 / (32)) + (0.041780717070086536 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5577714757301261 = (10.155324904967866 - 15.819675463730695) / 10.155324904967866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.09793687781935 > 10.824888950439226.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -85.17981573969624 (T) = (0 -16.084501663821538) / Math.Sqrt((0.09366564672038877 / (32)) + (0.019441132352970018 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5554691210964263 = (10.340611360053114 - 16.084501663821538) / 10.340611360053114 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.250267075893282 > 3.352759540047291.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -40.262601231643046 (T) = (0 -6.38887748599731) / Math.Sqrt((0.05214754665397373 / (32)) + (0.05028820581828033 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.9259796783391032 = (3.3172091885760975 - 6.38887748599731) / 3.3172091885760975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.534069930021154 > 24.45641570850646.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -74.28006930773032 (T) = (0 -28.327018439232784) / Math.Sqrt((0.0551718473136399 / (32)) + (0.03327214186978605 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.21335989886260198 = (23.345932617178466 - 28.327018439232784) / 23.345932617178466 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.49095898922284 > 51.227091925254726.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -42.96161504376527 (T) = (0 -54.068247923961614) / Math.Sqrt((0.13240447345404585 / (32)) + (0.14339452112148282 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.11207604377403953 = (48.61920030258977 - 54.068247923961614) / 48.61920030258977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.795311175006713 > 22.797346430260795.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -110.16261371835587 (T) = (0 -27.615813537490684) / Math.Sqrt((0.02893923834467118 / (32)) + (0.024010447309944224 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.27390750635811145 = (21.67803659187132 - 27.615813537490684) / 21.67803659187132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 9223372036854775807)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.848231463165604 > 54.11531003142006.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -30.30745666185831 (T) = (0 -56.216933586070546) / Math.Sqrt((0.16993580402529118 / (32)) + (0.24770725649393255 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.0951113723664043 = (51.33444415301112 - 56.216933586070546) / 51.33444415301112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | arm64 |
---|---|
OS | ubuntu 20.04 |
Baseline | c9f80fdde29c71ac87971383c77a1a5e665fadb9 |
Compare | a38b50c3a2f828458d15e68c78460db3ec7e5cda |
Diff | Diff |
Regressions in System.Collections.Concurrent.Count<String>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Queue - Duration of single invocation | 7.78 ns | 12.44 ns | 1.60 | 0.05 | False | |||||
Queue_EnqueueCountDequeue - Duration of single invocation | 33.47 ns | 42.69 ns | 1.28 | 0.11 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.Count<String>*'
Payloads
Histogram
System.Collections.Concurrent.Count<String>.Queue(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.440158083173902 > 8.179443964478883.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -146.98536372924585 (T) = (0 -12.371144724936286) / Math.Sqrt((0.014615805850382501 / (32)) + (0.0061045493574273715 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.5852219552120929 = (7.8040457894624 - 12.371144724936286) / 7.8040457894624 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.Concurrent.Count<String>.Queue_EnqueueCountDequeue(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.689222531441224 > 35.04501625853797.
IsChangePoint: Marked as a change because one of 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -76.91165895557451 (T) = (0 -42.74227963262908) / Math.Sqrt((0.4322036508108061 / (32)) + (0.005601279562968299 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.2701748915241361 = (33.650704259584934 - 42.74227963262908) / 33.650704259584934 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | arm64 |
---|---|
OS | ubuntu 20.04 |
Baseline | c9f80fdde29c71ac87971383c77a1a5e665fadb9 |
Compare | a38b50c3a2f828458d15e68c78460db3ec7e5cda |
Diff | Diff |
Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
FirstSpanTenSegments - Duration of single invocation | 6.66 ns | 8.73 ns | 1.31 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests<Char>*'
Payloads
Histogram
System.Buffers.Tests.ReadOnlySequenceTests<Char>.FirstSpanTenSegments
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.733886982660364 > 6.9937097877618255.
IsChangePoint: Marked as a change because one of 5/20/2022 10:18:01 PM, 6/27/2022 8:54:23 AM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -179.81577296993908 (T) = (0 -8.75725016440443) / Math.Sqrt((0.0021192516157491448 / (32)) + (0.0008248153498479827 / (12))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (12) - 2, .025) and -0.3132697641219422 = (6.668279742402785 - 8.75725016440443) / 6.668279742402785 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository