Regressions in System.Net.Security.Tests.SslStreamTests

[performanceautofiler]

Run Information

Architecture	x64
OS	alpine 3.12
Baseline	b4e258a377ccfd8fe636f51b50e8f9204d292d8f
Compare	0e35b8d33851efde52e4e7aaadac239f3c2e97d5
Diff	Diff

Regressions in System.Net.Security.Tests.SslStreamTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
HandshakeECDSA256CertAsync - Duration of single invocation	2.14 ms	2.75 ms	1.28	0.01	True
HandshakeRSA2048CertAsync - Duration of single invocation	2.42 ms	2.96 ms	1.23	0.00	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Security.Tests.SslStreamTests*'

Payloads

Baseline
Compare

Histogram

System.Net.Security.Tests.SslStreamTests.HandshakeECDSA256CertAsync(protocol: Tls13)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7451470734265735 > 2.2503181640281595.
IsChangePoint: Marked as a change because one of 3/24/2022 7:54:35 AM, 3/31/2022 1:45:25 PM falls between 3/22/2022 11:55:12 PM and 3/31/2022 1:45:25 PM.
IsRegressionStdDev: Marked as regression because -57.985291775661906 (T) = (0 -2737000.43546216) / Math.Sqrt((1622080925.6075373 / (22)) + (1036970535.6862763 / (31))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (31) - 2, .025) and -0.2809551936244573 = (2136687.0981004643 - 2737000.43546216) / 2136687.0981004643 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Net.Security.Tests.SslStreamTests.HandshakeRSA2048CertAsync(protocol: Tls13)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9622326522435896 > 2.558988685171287.
IsChangePoint: Marked as a change because one of 3/24/2022 7:54:35 AM, 3/31/2022 1:45:25 PM falls between 3/22/2022 11:55:12 PM and 3/31/2022 1:45:25 PM.
IsRegressionStdDev: Marked as regression because -65.73768931290239 (T) = (0 -2961577.750771263) / Math.Sqrt((424628717.59201896 / (22)) + (1391852437.844686 / (31))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (31) - 2, .025) and -0.21632557941456396 = (2434856.095188523 - 2961577.750771263) / 2434856.095188523 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[kunalspathak]

Regression introduced in #66334

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x64
OS	alpine 3.12
Baseline	b4e258a377ccfd8fe636f51b50e8f9204d292d8f
Compare	0e35b8d33851efde52e4e7aaadac239f3c2e97d5
Diff	Diff

Regressions in System.Net.Security.Tests.SslStreamTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
HandshakeECDSA256CertAsync - Duration of single invocation	2.14 ms	2.75 ms	1.28	0.01	True
HandshakeRSA2048CertAsync - Duration of single invocation	2.42 ms	2.96 ms	1.23	0.00	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Security.Tests.SslStreamTests*'

Payloads

Baseline
Compare

Histogram

System.Net.Security.Tests.SslStreamTests.HandshakeECDSA256CertAsync(protocol: Tls13)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7451470734265735 > 2.2503181640281595.
IsChangePoint: Marked as a change because one of 3/24/2022 7:54:35 AM, 3/31/2022 1:45:25 PM falls between 3/22/2022 11:55:12 PM and 3/31/2022 1:45:25 PM.
IsRegressionStdDev: Marked as regression because -57.985291775661906 (T) = (0 -2737000.43546216) / Math.Sqrt((1622080925.6075373 / (22)) + (1036970535.6862763 / (31))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (31) - 2, .025) and -0.2809551936244573 = (2136687.0981004643 - 2737000.43546216) / 2136687.0981004643 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Net.Security.Tests.SslStreamTests.HandshakeRSA2048CertAsync(protocol: Tls13)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9622326522435896 > 2.558988685171287.
IsChangePoint: Marked as a change because one of 3/24/2022 7:54:35 AM, 3/31/2022 1:45:25 PM falls between 3/22/2022 11:55:12 PM and 3/31/2022 1:45:25 PM.
IsRegressionStdDev: Marked as regression because -65.73768931290239 (T) = (0 -2961577.750771263) / Math.Sqrt((424628717.59201896 / (22)) + (1391852437.844686 / (31))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (31) - 2, .025) and -0.21632557941456396 = (2434856.095188523 - 2961577.750771263) / 2434856.095188523 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Net.Security, untriaged, refs/heads/main, RunKind=micro, PGO, Regression, CoreClr, x64, alpine 3.12
Milestone:	-

[kunalspathak]

@rzikm, @wfurt

[wfurt]

I actually experienced something similar when looking at memory allocations. #66334 made processing of expired certificates way more expensive as on each round they would be considered expired and we would rebuild the chain over and over.

This is the RS2048 certificate from test.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            9b:91:c6:45:a4:ba:77:e6
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = localhost
        Validity
            Not Before: Feb  1 00:23:54 2020 GMT
            Not After : Jan 31 00:23:54 2021 GMT
        Subject: CN = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)

I'm not sure how much we care about that, but something to consider. The expiration did not matter before is it is form of regression. I think we can get perf back by re-generating the certificates with updated time.