Skip to content

[Perf] Linux/x64: 7 Regressions on 10/25/2022 12:27:43 PM #9369

Closed
@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 5108757b997c59ab8ba1fc5309ab0d4e730e2b77
Compare f4b715f1cfadcbf85486f369724528c051be5733
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf16 - Duration of single invocation 11.65 ms 12.81 ms 1.10 0.09 False
WriteDeepUtf8 - Duration of single invocation 11.71 ms 12.72 ms 1.09 0.12 False
WriteDeepUtf16 - Duration of single invocation 11.30 ms 12.57 ms 1.11 0.12 False
WriteDeepUtf8 - Duration of single invocation 11.28 ms 13.44 ms 1.19 0.13 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 12.807059527777778 > 12.577279040480768.
IsChangePoint: Marked as a change because one of 9/23/2022 4:51:48 AM, 9/29/2022 2:35:34 PM, 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -15.60465220598909 (T) = (0 -12891813.298965862) / Math.Sqrt((48206077420.79716 / (18)) + (48245271341.293365 / (30))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (30) - 2, .025) and -0.08606718755044086 = (11870180.26761546 - 12891813.298965862) / 11870180.26761546 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 12.715969921052633 > 12.28625825175.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -14.836978613525337 (T) = (0 -12903827.474388717) / Math.Sqrt((56569313334.44945 / (19)) + (77450997388.35188 / (31))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (31) - 2, .025) and -0.09299672414284273 = (11805915.964211369 - 12903827.474388717) / 11805915.964211369 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 12.571327927125504 > 11.882048450833334.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -10.16660406120826 (T) = (0 -12679529.375793314) / Math.Sqrt((161481946020.28162 / (18)) + (149369758629.06366 / (30))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (30) - 2, .025) and -0.10460971677522353 = (11478741.480574414 - 12679529.375793314) / 11478741.480574414 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 13.439125922222223 > 11.94821418359375.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -11.588391969252594 (T) = (0 -12590869.55033526) / Math.Sqrt((93756075510.60684 / (18)) + (106505128955.79842 / (27))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (27) - 2, .025) and -0.09655794426199818 = (11482174.395087827 - 12590869.55033526) / 11482174.395087827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 5108757b997c59ab8ba1fc5309ab0d4e730e2b77
Compare f4b715f1cfadcbf85486f369724528c051be5733
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf16 - Duration of single invocation 2.18 μs 2.35 μs 1.08 0.10 False
WriteBasicUtf16 - Duration of single invocation 1.91 μs 2.05 μs 1.07 0.08 False
WriteBasicUtf16 - Duration of single invocation 2.13 μs 2.27 μs 1.06 0.05 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 10)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.3470457769333537 > 2.3006363588000185.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -8.98485125716051 (T) = (0 -2349.664023670746) / Math.Sqrt((2488.650926598435 / (17)) + (1180.8929824583415 / (31))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (31) - 2, .025) and -0.05478332452480708 = (2227.6271998605007 - 2349.664023670746) / 2227.6271998605007 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.048472369784775 > 2.0047261577822555.
IsChangePoint: Marked as a change because one of 10/25/2022 2:56:59 AM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -13.405038021732336 (T) = (0 -2061.4292284936373) / Math.Sqrt((1077.6122243919144 / (15)) + (1609.7869389272123 / (29))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (29) - 2, .025) and -0.07919548535783179 = (1910.1536806467677 - 2061.4292284936373) / 1910.1536806467677 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.265119184527259 > 2.241521056202956.
IsChangePoint: Marked as a change because one of 10/24/2022 7:39:25 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -10.172598185686985 (T) = (0 -2282.928184589658) / Math.Sqrt((1574.831246786393 / (16)) + (1161.7978626864585 / (34))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (34) - 2, .025) and -0.05408579709470785 = (2165.789721180107 - 2282.928184589658) / 2165.789721180107 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions