Skip to content

[Perf] Linux/x64: 8 Regressions on 10/25/2022 12:27:43 PM #9351

Closed
@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 5108757b997c59ab8ba1fc5309ab0d4e730e2b77
Compare f4b715f1cfadcbf85486f369724528c051be5733
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToUtf8Bytes - Duration of single invocation 38.33 μs 41.19 μs 1.07 0.00 False
SerializeToWriter - Duration of single invocation 37.63 μs 40.99 μs 1.09 0.00 False
SerializeToStream - Duration of single invocation 39.12 μs 42.35 μs 1.08 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.Json.Serialization.Tests.WriteJson<Dictionary<String, String>>.SerializeToUtf8Bytes


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.19229951077397 > 40.2988708713241.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -38.859507512178354 (T) = (0 -41405.27184342041) / Math.Sqrt((67903.13798543392 / (17)) + (49041.70475777477 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.07466581123050155 = (38528.5094312352 - 41405.27184342041) / 38528.5094312352 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;.SerializeToWriter

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.98579117716166 > 39.59248766643648.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -29.205029867439894 (T) = (0 -40824.94549951425) / Math.Sqrt((146698.3928160469 / (17)) + (38064.0425676531 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.07614357202646804 = (37936.337270163196 - 40824.94549951425) / 37936.337270163196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Dictionary&lt;String, String&gt;&gt;.SerializeToStream

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 42.35071906920992 > 41.033094007646895.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -25.604568556776417 (T) = (0 -42099.32046705353) / Math.Sqrt((189495.98123079867 / (17)) + (39780.30731195595 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.07249558589599962 = (39253.60721357404 - 42099.32046705353) / 39253.60721357404 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 5108757b997c59ab8ba1fc5309ab0d4e730e2b77
Compare f4b715f1cfadcbf85486f369724528c051be5733
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Strings

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteStringsUtf16 - Duration of single invocation 42.48 ms 52.23 ms 1.23 0.00 False
WriteStringsUtf16 - Duration of single invocation 26.83 ms 37.08 ms 1.38 0.00 False
WriteStringsUtf16 - Duration of single invocation 124.27 ms 216.55 ms 1.74 0.18 False
WriteStringsUtf16 - Duration of single invocation 120.27 ms 219.52 ms 1.83 0.02 False
WriteStringsUtf16 - Duration of single invocation 43.16 ms 51.91 ms 1.20 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Strings*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: OneEscaped)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 52.23277989285714 > 44.635313475000004.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -199.1803998005975 (T) = (0 -52384990.36119156) / Math.Sqrt((21570267646.12087 / (17)) + (39462176651.104904 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.23268646027811166 = (42496605.624574445 - 52384990.36119156) / 42496605.624574445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: NoneEscaped)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 37.08365311428572 > 28.23125232788462.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -203.0113901928131 (T) = (0 -36924033.60399125) / Math.Sqrt((11006680080.33792 / (17)) + (58490573723.158844 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.37073417929040897 = (26937413.65893845 - 36924033.60399125) / 26937413.65893845 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: AllEscaped)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 216.54523714285713 > 128.8745705625.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -146.51231834752994 (T) = (0 -214934070.2859307) / Math.Sqrt((6022041992050.817 / (17)) + (1510696338901.8418 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.757869498963925 = (122269639.70454645 - 214934070.2859307) / 122269639.70454645 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: AllEscaped)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 219.5176467857143 > 126.260763825.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -271.6737862201362 (T) = (0 -218330712.1158119) / Math.Sqrt((552719372430.2446 / (16)) + (3159307017528.8384 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.8153180834681577 = (120271325.50714858 - 218330712.1158119) / 120271325.50714858 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: OneEscaped)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 51.908773 > 45.00764144041667.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -106.7442563290323 (T) = (0 -52051160.8832487) / Math.Sqrt((99784015775.82983 / (17)) + (56996714791.007225 / (33))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (33) - 2, .025) and -0.2176462464688151 = (42747358.71292465 - 52051160.8832487) / 42747358.71292465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions