Closed
Description
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 5108757b997c59ab8ba1fc5309ab0d4e730e2b77 |
Compare | f4b715f1cfadcbf85486f369724528c051be5733 |
Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Basic
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
WriteBasicUtf16 - Duration of single invocation | 891.21 ns | 955.27 ns | 1.07 | 0.04 | False | |||||
WriteBasicUtf16 - Duration of single invocation | 734.44 ns | 775.72 ns | 1.06 | 0.06 | False | |||||
WriteBasicUtf16 - Duration of single invocation | 848.10 ns | 896.41 ns | 1.06 | 0.05 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'
Related Issues
Regressions
- [Perf] Windows/x64: 6 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Windows/x64: 10 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Alpine/x64: 12 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Linux/x64: 8 Regressions on 10/25/2022 12:27:43 PM
Improvements
Payloads
Histogram
Edge Detector Info
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 10)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 955.2699081163859 > 940.0068051150469.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -22.085948361088732 (T) = (0 -951.7876147927569) / Math.Sqrt((55.66860440376942 / (19)) + (115.18402382642793 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.06303181469333649 = (895.3519562039907 - 951.7876147927569) / 895.3519562039907 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 775.7183232626668 > 772.389699164534.
IsChangePoint: Marked as a change because one of 10/19/2022 7:27:15 PM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -16.47229823064067 (T) = (0 -781.8680253448002) / Math.Sqrt((33.9677200334984 / (19)) + (141.3269665454751 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.055382830576054046 = (740.838303118914 - 781.8680253448002) / 740.838303118914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 896.4063893312984 > 891.2485808182577.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -7.902503018180222 (T) = (0 -909.2863029466005) / Math.Sqrt((608.1230664348213 / (19)) + (398.4166670152741 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.06151187192539703 = (856.5955096642622 - 909.2863029466005) / 856.5955096642622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 5108757b997c59ab8ba1fc5309ab0d4e730e2b77 |
Compare | f4b715f1cfadcbf85486f369724528c051be5733 |
Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Deep
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
WriteDeepUtf16 - Duration of single invocation | 4.20 ms | 4.67 ms | 1.11 | 0.02 | False | |||||
WriteDeepUtf8 - Duration of single invocation | 4.49 ms | 5.63 ms | 1.25 | 0.41 | False | |||||
WriteDeepUtf16 - Duration of single invocation | 4.33 ms | 4.93 ms | 1.14 | 0.37 | False | |||||
WriteDeepUtf8 - Duration of single invocation | 4.26 ms | 5.12 ms | 1.20 | 0.14 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Deep*'
Related Issues
Regressions
- [Perf] Windows/x64: 6 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Windows/x64: 10 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Alpine/x64: 12 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Linux/x64: 8 Regressions on 10/25/2022 12:27:43 PM
Improvements
Payloads
Histogram
Edge Detector Info
System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.670000740740742 > 4.4003929685870276.
IsChangePoint: Marked as a change because one of 8/24/2022 2:39:26 AM, 9/3/2022 11:44:47 AM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -42.09518212679972 (T) = (0 -4678205.985948087) / Math.Sqrt((2379087833.5240636 / (19)) + (760275662.5638984 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.12337676684872374 = (4164414.045228393 - 4678205.985948087) / 4164414.045228393 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.63025365079365 > 4.595497236842106.
IsChangePoint: Marked as a change because one of 9/8/2022 11:29:08 PM, 10/10/2022 9:53:50 PM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -12.764533322379403 (T) = (0 -5423838.67701309) / Math.Sqrt((64261975070.34909 / (19)) + (80509674539.7906 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.22061453744558224 = (4443531.115370561 - 5423838.67701309) / 4443531.115370561 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.92659725490196 > 4.537332057017544.
IsChangePoint: Marked as a change because one of 9/9/2022 3:12:19 AM, 9/13/2022 12:23:36 PM, 9/29/2022 7:42:19 AM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -17.74439566938396 (T) = (0 -5125143.069514541) / Math.Sqrt((890545861.9605262 / (19)) + (60895354234.34023 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.1804728599557754 = (4341601.779567 - 5125143.069514541) / 4341601.779567 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.117812222222222 > 4.383590147540984.
IsChangePoint: Marked as a change because one of 9/12/2022 12:30:55 AM, 9/12/2022 4:32:08 PM, 9/29/2022 7:42:19 AM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -16.524497825454777 (T) = (0 -4748141.721915257) / Math.Sqrt((4578301164.950006 / (19)) + (34646336812.04457 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.14497343693508505 = (4146944.8711625054 - 4748141.721915257) / 4146944.8711625054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 5108757b997c59ab8ba1fc5309ab0d4e730e2b77 |
Compare | f4b715f1cfadcbf85486f369724528c051be5733 |
Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Strings
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
WriteStringsUtf16 - Duration of single invocation | 12.04 ms | 13.18 ms | 1.09 | 0.11 | False | |||||
WriteStringsUtf16 - Duration of single invocation | 6.41 ms | 7.34 ms | 1.15 | 0.02 | False | |||||
WriteStringsUtf16 - Duration of single invocation | 6.82 ms | 7.31 ms | 1.07 | 0.16 | False | |||||
WriteStringsUtf16 - Duration of single invocation | 7.41 ms | 8.06 ms | 1.09 | 0.07 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Strings*'
Related Issues
Regressions
- [Perf] Alpine/x64: 11 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Windows/x64: 6 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Windows/x64: 10 Regressions on 10/25/2022 12:27:43 PM
- [Perf] Linux/x64: 8 Regressions on 10/25/2022 12:27:43 PM
Improvements
Payloads
Histogram
Edge Detector Info
System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: OneEscaped)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 13.182028333333333 > 12.604363625000001.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -7.517760058566849 (T) = (0 -14021786.150173608) / Math.Sqrt((37982335964.816666 / (19)) + (1974472526058.98 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.1564961198233643 = (12124369.385964913 - 14021786.150173608) / 12124369.385964913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: True, Escaped: NoneEscaped)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 7.338444343891402 > 6.814796546777546.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -27.73451463518777 (T) = (0 -7212034.915400775) / Math.Sqrt((7326427218.386213 / (19)) + (7157851924.128363 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.10487871351703947 = (6527444.892519916 - 7212034.915400775) / 6527444.892519916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: False, Escaped: NoneEscaped)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 7.311784117647059 > 7.119729858585859.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -22.540203732373996 (T) = (0 -7390676.456919307) / Math.Sqrt((11732914861.085526 / (19)) + (7517455652.618128 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.09774822810339516 = (6732578.80788234 - 7390676.456919307) / 6732578.80788234 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: NoneEscaped)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 8.06285261904762 > 7.770194070032909.
IsChangePoint: Marked as a change because one of 8/24/2022 2:39:26 AM, 8/27/2022 3:22:36 PM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -21.301081305962047 (T) = (0 -7952684.979971807) / Math.Sqrt((4453242220.426787 / (19)) + (10979510298.51758 / (32))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (32) - 2, .025) and -0.06879465610760273 = (7440797.850668854 - 7952684.979971807) / 7440797.850668854 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository