Skip to content

[Perf] Windows/arm64: 23 Regressions on 12/21/2022 4:32:37 PM #11194

Closed

Description

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 8d66372138fca4c9c73eef07c1632f13a0887d8e
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Join_Enumerable - Duration of single invocation 296.07 ns 334.66 ns 1.13 0.06 False
ToUpperInvariant - Duration of single invocation 19.83 ns 25.41 ns 1.28 0.31 False
ToLowerInvariant - Duration of single invocation 6.32 ns 9.04 ns 1.43 0.34 False
ToLower - Duration of single invocation 17.23 ns 19.51 ns 1.13 0.22 False
ToUpper - Duration of single invocation 29.71 ns 33.00 ns 1.11 0.05 False
ToLower - Duration of single invocation 30.42 ns 33.42 ns 1.10 0.03 False
ToUpperInvariant - Duration of single invocation 21.16 ns 24.29 ns 1.15 0.32 False
ToLowerInvariant - Duration of single invocation 19.80 ns 25.28 ns 1.28 0.24 False
ToLower - Duration of single invocation 30.85 ns 33.68 ns 1.09 0.04 False
ToLowerInvariant - Duration of single invocation 19.48 ns 23.06 ns 1.18 0.31 False
ToUpper - Duration of single invocation 29.30 ns 32.33 ns 1.10 0.04 False
ToUpper - Duration of single invocation 14.76 ns 18.94 ns 1.28 0.24 False
ToUpperInvariant - Duration of single invocation 5.65 ns 9.16 ns 1.62 0.40 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Join_Enumerable


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 334.66333073105886 > 310.1232281009754.
IsChangePoint: Marked as a change because one of 9/30/2022 1:24:23 AM, 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -35.56753947678202 (T) = (0 -338.1024718087037) / Math.Sqrt((12.24625632510734 / (13)) + (10.160627020295303 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.1450514697595924 = (295.27272855227153 - 338.1024718087037) / 295.27272855227153 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpperInvariant(s: "test")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.414427047072177 > 20.597503598316973.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -22.061815479173696 (T) = (0 -23.994170338302084) / Math.Sqrt((0.12916561174130706 / (13)) + (0.602321354753627 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.22550875253312355 = (19.578946530333784 - 23.994170338302084) / 19.578946530333784 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLowerInvariant(s: "test")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.036497944769593 > 6.627082228036932.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -22.50111154160463 (T) = (0 -9.171730434726742) / Math.Sqrt((0.20891689611225828 / (13)) + (0.024832697954338973 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.476660305783434 = (6.2111308869108734 - 9.171730434726742) / 6.2111308869108734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLower(s: "test")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.51009981849415 > 17.985569481689346.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -19.179701037557003 (T) = (0 -19.597615576673654) / Math.Sqrt((0.14031857224319072 / (13)) + (0.18774109945438738 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.16148029786276744 = (16.872964279062764 - 19.597615576673654) / 16.872964279062764 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpper(s: "TeSt")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.000620663713114 > 31.055951448408138.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -19.43446867925207 (T) = (0 -33.52746689619404) / Math.Sqrt((0.12438905590055883 / (13)) + (0.7008816314786561 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.13951471056601336 = (29.422583653650655 - 33.52746689619404) / 29.422583653650655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLower(s: "TEST")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.417354494004385 > 31.84558462930623.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -13.395060246232482 (T) = (0 -34.13213888290169) / Math.Sqrt((0.2832731455885848 / (13)) + (1.3289671016411684 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.13195973790609833 = (30.153138614310965 - 34.13213888290169) / 30.153138614310965 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpperInvariant(s: "TeSt")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.290574363224817 > 20.791499677158964.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -19.518210726867927 (T) = (0 -23.98887070096966) / Math.Sqrt((0.3014290855452062 / (13)) + (0.432212644573708 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.2080350723349983 = (19.857760134895607 - 23.98887070096966) / 19.857760134895607 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLowerInvariant(s: "TEST")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.27565528156059 > 20.77821741123003.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -19.773528258430883 (T) = (0 -23.9518979242687) / Math.Sqrt((0.4383164479841955 / (13)) + (0.3000113879439696 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.2228161388680815 = (19.587489208672157 - 23.9518979242687) / 19.587489208672157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLower(s: "TeSt")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.67716767418449 > 32.01994829408373.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -17.14699364826293 (T) = (0 -33.84112233688593) / Math.Sqrt((0.26412430448800156 / (13)) + (0.44141920804607765 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.11646963280450641 = (30.310830982369858 - 33.84112233688593) / 30.310830982369858 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLowerInvariant(s: "TeSt")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.063296915172234 > 20.505618850660184.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -33.090668274750065 (T) = (0 -23.911675199947346) / Math.Sqrt((0.08327327970194669 / (13)) + (0.21458909181712643 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.22122514391528453 = (19.580071143381307 - 23.911675199947346) / 19.580071143381307 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpper(s: "test")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.332151632345 > 30.804338657715615.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -26.337359233834636 (T) = (0 -33.62495721567597) / Math.Sqrt((0.10092346314378493 / (13)) + (0.3675918892346647 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.14500962102741463 = (29.36652810437031 - 33.62495721567597) / 29.36652810437031 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpper(s: "TEST")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.942335073449645 > 16.817424168362383.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -14.595238090343788 (T) = (0 -18.545804408767488) / Math.Sqrt((0.2629325554235216 / (13)) + (0.26753598986410165 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.16858133008526954 = (15.8703582979665 - 18.545804408767488) / 15.8703582979665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpperInvariant(s: "TEST")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.155163945008864 > 6.44893053461952.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -25.68438599814117 (T) = (0 -9.30690673955127) / Math.Sqrt((0.1734794542635362 / (13)) + (0.02865013532801322 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.5048307453179597 = (6.184686728728943 - 9.30690673955127) / 6.184686728728943 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS Windows 10.0.19041
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 8d66372138fca4c9c73eef07c1632f13a0887d8e
Diff Diff

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetByteCount - Duration of single invocation 17.92 μs 34.17 μs 1.91 0.01 False
GetByteCount - Duration of single invocation 23.79 μs 40.12 μs 1.69 0.00 False
GetByteCount - Duration of single invocation 51.81 μs 86.77 μs 1.67 0.00 False
GetBytes - Duration of single invocation 123.69 μs 141.71 μs 1.15 0.00 False
GetBytes - Duration of single invocation 197.98 μs 215.88 μs 1.09 0.00 False
GetBytes - Duration of single invocation 146.39 μs 181.08 μs 1.24 0.00 False
GetByteCount - Duration of single invocation 25.96 μs 43.57 μs 1.68 0.00 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.16599150031192 > 18.81367854071101.
IsChangePoint: Marked as a change because one of 10/14/2022 9:06:25 AM, 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -8568.995089522856 (T) = (0 -34164.00743610237) / Math.Sqrt((30.758538662959406 / (13)) + (24.571060388198948 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.9067190563681012 = (17917.69339169328 - 34164.00743610237) / 17917.69339169328 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.1208949704142 > 24.9784349552892.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -6727.035352665285 (T) = (0 -40134.01645034987) / Math.Sqrt((27.045655028749024 / (13)) + (76.45384947871726 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.6870280536832469 = (23789.774190610675 - 40134.01645034987) / 23789.774190610675 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishMostlyAscii)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.76546221389108 > 54.466236225961545.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -915.7129729032915 (T) = (0 -86769.46212714349) / Math.Sqrt((18520.589505035812 / (13)) + (505.19599847967487 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.6718229719017443 = (51901.10650797007 - 86769.46212714349) / 51901.10650797007 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: Cyrillic)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 141.70833333333337 > 129.93362967519684.
IsChangePoint: Marked as a change because one of 11/30/2022 2:05:04 AM, 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -154.90269403697508 (T) = (0 -141518.9849586993) / Math.Sqrt((29511.042669360813 / (13)) + (215669.12084546566 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.14293172819960412 = (123821.03100911037 - 141518.9849586993) / 123821.03100911037 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: Greek)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 215.87748842592592 > 208.3405915033674.
IsChangePoint: Marked as a change because one of 11/30/2022 2:05:04 AM, 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -52.80859080276642 (T) = (0 -217844.13584237377) / Math.Sqrt((842773.0640384926 / (13)) + (1498820.1294783743 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.09966069955928718 = (198101.22879691847 - 217844.13584237377) / 198101.22879691847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishMostlyAscii)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 181.0798782771536 > 154.0526226635514.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -301.52435481089407 (T) = (0 -180895.914878383) / Math.Sqrt((49613.73812359402 / (13)) + (178722.17948667676 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.23187820694107752 = (146845.61660326173 - 180895.914878383) / 146845.61660326173 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: Greek)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 43.57316968430826 > 27.2476458519678.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -1860.5546188179449 (T) = (0 -43550.95653844046) / Math.Sqrt((700.9654185582541 / (13)) + (710.035346761315 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.6778156796942642 = (25956.93738323892 - 43550.95653844046) / 25956.93738323892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 8d66372138fca4c9c73eef07c1632f13a0887d8e
Diff Diff

Regressions in System.ComponentModel.Tests.Perf_TypeDescriptorTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetConverter - Duration of single invocation 173.99 ns 194.21 ns 1.12 0.03 False
GetConverter - Duration of single invocation 116.55 ns 129.12 ns 1.11 0.03 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.ComponentModel.Tests.Perf_TypeDescriptorTests*'

Payloads

Baseline
Compare

Histogram

System.ComponentModel.Tests.Perf_TypeDescriptorTests.GetConverter(typeToConvert: typeof(System.Guid))


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 194.21294329564145 > 181.62521817889.
IsChangePoint: Marked as a change because one of 10/6/2022 2:13:23 PM, 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -39.33973176724806 (T) = (0 -194.72908774562407) / Math.Sqrt((2.2746909683452006 / (13)) + (2.0519011197646444 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.11911380118318617 = (174.00293655546577 - 194.72908774562407) / 174.00293655546577 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.ComponentModel.Tests.Perf_TypeDescriptorTests.GetConverter(typeToConvert: typeof(string))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.11781944754324 > 122.06454926686476.
IsChangePoint: Marked as a change because one of 9/22/2022 3:11:11 PM, 9/27/2022 9:53:04 AM, 10/6/2022 2:13:23 PM, 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -31.88883289065751 (T) = (0 -127.99747529264737) / Math.Sqrt((0.4536949040804344 / (13)) + (1.9810635166438846 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.10033133067123312 = (116.32630256430606 - 127.99747529264737) / 116.32630256430606 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 8d66372138fca4c9c73eef07c1632f13a0887d8e
Diff Diff

Regressions in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetByteCount - Duration of single invocation 10.60 ns 11.90 ns 1.12 0.11 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_Encoding.GetByteCount(size: 16, encName: "utf-8")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.90207453945245 > 11.175408462193328.
IsChangePoint: Marked as a change because one of 9/13/2022 2:36:07 PM, 11/21/2022 10:59:14 PM, 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -13.338569703714693 (T) = (0 -12.233820410584944) / Math.Sqrt((0.04504923734971284 / (13)) + (0.1980317719117278 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.14423778044593555 = (10.691676694871187 - 12.233820410584944) / 10.691676694871187 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions