Closed
Description
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 5108757b997c59ab8ba1fc5309ab0d4e730e2b77 |
Compare | f4b715f1cfadcbf85486f369724528c051be5733 |
Diff | Diff |
Regressions in System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
SerializeToUtf8Bytes - Duration of single invocation | 38.33 μs | 41.19 μs | 1.07 | 0.00 | False | |||||
SerializeToWriter - Duration of single invocation | 37.63 μs | 40.99 μs | 1.09 | 0.00 | False | |||||
SerializeToStream - Duration of single invocation | 39.12 μs | 42.35 μs | 1.08 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>*'
Related Issues
Regressions
- [Perf] Windows/x64: 10 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Alpine/x64: 12 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Windows/x64: 6 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Linux/x64: 8 Regressions on 10/25/2022 12:27:43 PM
Improvements
Payloads
Histogram
Edge Detector Info
System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToUtf8Bytes
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.19229951077397 > 40.2988708713241.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -38.859507512178354 (T) = (0 -41405.27184342041) / Math.Sqrt((67903.13798543392 / (17)) + (49041.70475777477 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.07466581123050155 = (38528.5094312352 - 41405.27184342041) / 38528.5094312352 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToWriter
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.98579117716166 > 39.59248766643648.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -29.205029867439894 (T) = (0 -40824.94549951425) / Math.Sqrt((146698.3928160469 / (17)) + (38064.0425676531 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.07614357202646804 = (37936.337270163196 - 40824.94549951425) / 37936.337270163196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToStream
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 42.35071906920992 > 41.033094007646895.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -25.604568556776417 (T) = (0 -42099.32046705353) / Math.Sqrt((189495.98123079867 / (17)) + (39780.30731195595 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.07249558589599962 = (39253.60721357404 - 42099.32046705353) / 39253.60721357404 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 5108757b997c59ab8ba1fc5309ab0d4e730e2b77 |
Compare | f4b715f1cfadcbf85486f369724528c051be5733 |
Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Strings
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
WriteStringsUtf16 - Duration of single invocation | 42.48 ms | 52.23 ms | 1.23 | 0.00 | False | |||||
WriteStringsUtf16 - Duration of single invocation | 26.83 ms | 37.08 ms | 1.38 | 0.00 | False | |||||
WriteStringsUtf16 - Duration of single invocation | 124.27 ms | 216.55 ms | 1.74 | 0.18 | False | |||||
WriteStringsUtf16 - Duration of single invocation | 120.27 ms | 219.52 ms | 1.83 | 0.02 | False | |||||
WriteStringsUtf16 - Duration of single invocation | 43.16 ms | 51.91 ms | 1.20 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Strings*'
Related Issues
Regressions
- [Perf] Windows/x64: 10 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Alpine/x64: 12 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Windows/x64: 6 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Linux/x64: 8 Regressions on 10/25/2022 12:27:43 PM
Improvements
Payloads
Histogram
Edge Detector Info
System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: OneEscaped)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 52.23277989285714 > 44.635313475000004.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -199.1803998005975 (T) = (0 -52384990.36119156) / Math.Sqrt((21570267646.12087 / (17)) + (39462176651.104904 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.23268646027811166 = (42496605.624574445 - 52384990.36119156) / 42496605.624574445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: NoneEscaped)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 37.08365311428572 > 28.23125232788462.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -203.0113901928131 (T) = (0 -36924033.60399125) / Math.Sqrt((11006680080.33792 / (17)) + (58490573723.158844 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.37073417929040897 = (26937413.65893845 - 36924033.60399125) / 26937413.65893845 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: AllEscaped)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 216.54523714285713 > 128.8745705625.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -146.51231834752994 (T) = (0 -214934070.2859307) / Math.Sqrt((6022041992050.817 / (17)) + (1510696338901.8418 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.757869498963925 = (122269639.70454645 - 214934070.2859307) / 122269639.70454645 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: AllEscaped)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 219.5176467857143 > 126.260763825.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -271.6737862201362 (T) = (0 -218330712.1158119) / Math.Sqrt((552719372430.2446 / (16)) + (3159307017528.8384 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.8153180834681577 = (120271325.50714858 - 218330712.1158119) / 120271325.50714858 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: OneEscaped)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 51.908773 > 45.00764144041667.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -106.7442563290323 (T) = (0 -52051160.8832487) / Math.Sqrt((99784015775.82983 / (17)) + (56996714791.007225 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.2176462464688151 = (42747358.71292465 - 52051160.8832487) / 42747358.71292465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository