Skip to content

[Perf] Windows/x86: 6 Regressions on 9/22/2022 4:52:57 PM #8700

Open
@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e013f9dfc31c49c34428ea3400eb0a0a7a8f1e24
Compare 011b9496d03aff0120161f0609f4c2a850f39a40
Diff Diff

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Get_EnumerateHeaders_Unvalidated - Duration of single invocation 36.75 μs 51.52 μs 1.40 0.44 False
Get - Duration of single invocation 55.05 μs 73.33 μs 1.33 0.27 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: False, chunkedResponse: False, responseLength: 1)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.52245506535948 > 38.363666913216655.
IsChangePoint: Marked as a change because one of 9/2/2022 11:09:40 AM, 9/13/2022 12:23:36 PM, 9/22/2022 3:11:11 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -8.96045248747586 (T) = (0 -48887.48441540962) / Math.Sqrt((2253962.57402785 / (35)) + (40030642.70745127 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -0.32667784985179654 = (36849.55200003592 - 48887.48441540962) / 36849.55200003592 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: True, chunkedResponse: True, responseLength: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.33093996840442 > 57.7645757995595.
IsChangePoint: Marked as a change because one of 8/13/2022 7:54:18 PM, 9/2/2022 11:09:40 AM, 9/22/2022 3:11:11 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -39.89770487921154 (T) = (0 -73622.40186663314) / Math.Sqrt((3112491.070022479 / (35)) + (2405533.7651674664 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -0.3130166466317612 = (56071.18695371783 - 73622.40186663314) / 56071.18695371783 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x86
OS Windows 10.0.18362
Baseline e013f9dfc31c49c34428ea3400eb0a0a7a8f1e24
Compare 011b9496d03aff0120161f0609f4c2a850f39a40
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Span - Duration of single invocation 21.88 μs 26.03 μs 1.19 0.06 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.ContainsFalse<Int32>.Span(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.030177777777777 > 22.890186888415403.
IsChangePoint: Marked as a change because one of 8/17/2022 2:11:12 PM, 9/13/2022 6:36:50 PM, 9/22/2022 3:11:11 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -18.47173386739818 (T) = (0 -26053.181093988405) / Math.Sqrt((1520485.061950293 / (35)) + (1969.3783705020223 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -0.17360089970626924 = (22199.353375162747 - 26053.181093988405) / 22199.353375162747 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e013f9dfc31c49c34428ea3400eb0a0a7a8f1e24
Compare 011b9496d03aff0120161f0609f4c2a850f39a40
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArraySetAll - Duration of single invocation 9.86 ns 12.22 ns 1.24 0.29 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.223696591978399 > 9.779698143501854.
IsChangePoint: Marked as a change because one of 9/22/2022 3:11:11 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -8.083244116459822 (T) = (0 -11.95869735018357) / Math.Sqrt((1.3963728310277783 / (35)) + (0.6438112191327675 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -0.21376311557685285 = (9.852579302098896 - 11.95869735018357) / 9.852579302098896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e013f9dfc31c49c34428ea3400eb0a0a7a8f1e24
Compare 011b9496d03aff0120161f0609f4c2a850f39a40
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DotBenchmark - Duration of single invocation 32.73 ns 35.43 ns 1.08 0.09 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int64>.DotBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.43270041553871 > 34.31409092453629.
IsChangePoint: Marked as a change because one of 8/26/2022 6:25:06 PM, 9/13/2022 12:23:36 PM, 9/22/2022 3:11:11 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -12.45357883145386 (T) = (0 -35.64003329631755) / Math.Sqrt((0.5275154685890452 / (35)) + (0.7395035494308355 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -0.08217421293960321 = (32.93372995786459 - 35.64003329631755) / 32.93372995786459 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e013f9dfc31c49c34428ea3400eb0a0a7a8f1e24
Compare 011b9496d03aff0120161f0609f4c2a850f39a40
Diff Diff

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseSpan - Duration of single invocation 0.41 ns 90.97 ns 219.53 0.50 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_Int64.ParseSpan(value: "9223372036854775807")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 90.97053517886009 > 0.34195371954454246.
IsChangePoint: Marked as a change because one of 8/13/2022 2:59:44 AM, 9/13/2022 12:23:36 PM, 9/22/2022 3:11:11 PM, 9/27/2022 3:32:36 AM falls between 9/18/2022 2:31:46 PM and 9/27/2022 3:32:36 AM.
IsRegressionStdDev: Marked as regression because -397.67415731456595 (T) = (0 -90.5828921460029) / Math.Sqrt((1.622910537503068 / (35)) + (0.10782617365513492 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -123.89375901500135 = (0.7252795724974755 - 90.5828921460029) / 0.7252795724974755 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions