Skip to content

[Perf] Linux/arm64: 12 Regressions on 2/2/2023 8:18:56 AM #12773

Closed
@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 3c47b2c75dfca5a82a59db238f103cdbda902826
Compare 873775b42431bf27a9ae69633864e5290443c29b
Diff Diff

Regressions in System.Numerics.Tests.Perf_Quaternion

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MultiplyByQuaternionOperatorBenchmark - Duration of single invocation 2.22 ns 7.94 ns 3.57 0.36 False
LerpBenchmark - Duration of single invocation 14.14 ns 36.98 ns 2.62 0.01 False
SlerpBenchmark - Duration of single invocation 14.12 ns 37.19 ns 2.63 0.03 False
CreateFromRotationMatrixBenchmark - Duration of single invocation 15.51 ns 17.74 ns 1.14 0.05 False
DivisionOperatorBenchmark - Duration of single invocation 8.90 ns 15.65 ns 1.76 0.01 False
ConcatenateBenchmark - Duration of single invocation 2.20 ns 8.08 ns 3.68 0.39 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Quaternion*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Quaternion.MultiplyByQuaternionOperatorBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.941581479533306 > 2.3212239364452296.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -167.45954392890064 (T) = (0 -8.105748291406591) / Math.Sqrt((0.009554647809051419 / (24)) + (0.013660284774689476 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -2.7170171732207136 = (2.180713166945941 - 8.105748291406591) / 2.180713166945941 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.LerpBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.983729453513575 > 14.878670792138404.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -655.3871681404464 (T) = (0 -37.14380633354826) / Math.Sqrt((0.014716382798237213 / (24)) + (0.009806419615714195 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -1.6166620942734653 = (14.195110027709367 - 37.14380633354826) / 14.195110027709367 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.SlerpBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.194904381975675 > 14.831858502405032.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -681.314249430754 (T) = (0 -37.12840737317261) / Math.Sqrt((0.01145128982304304 / (24)) + (0.010573844010070088 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -1.6248887137268486 = (14.144754853419004 - 37.12840737317261) / 14.144754853419004 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.CreateFromRotationMatrixBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.74071543547166 > 16.523233185803527.
IsChangePoint: Marked as a change because one of 1/6/2023 8:18:17 PM, 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -66.16114680045021 (T) = (0 -17.685975274609746) / Math.Sqrt((0.006977108161075092 / (24)) + (0.009905508520053573 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -0.1271876354155651 = (15.69035599657672 - 17.685975274609746) / 15.69035599657672 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.DivisionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.652727299059364 > 9.351297146848694.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -179.15302155289265 (T) = (0 -15.597904040419898) / Math.Sqrt((0.010926303958308299 / (24)) + (0.015376183196756387 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -0.7613279646593847 = (8.85576357917891 - 15.597904040419898) / 8.85576357917891 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.ConcatenateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.082194213649098 > 2.3080874927400212.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -158.75556827975439 (T) = (0 -8.044385993022145) / Math.Sqrt((0.005161299902007397 / (23)) + (0.018279518290545582 / (16))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (16) - 2, .025) and -2.6985814767956176 = (2.174992235128926 - 8.044385993022145) / 2.174992235128926 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 3c47b2c75dfca5a82a59db238f103cdbda902826
Compare 873775b42431bf27a9ae69633864e5290443c29b
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformBenchmark - Duration of single invocation 24.31 ns 28.59 ns 1.18 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.TransformBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.58503990700628 > 24.11130244544981.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -24.07581338992146 (T) = (0 -28.66686478868555) / Math.Sqrt((1.2286013034574401 / (23)) + (0.014949985175678191 / (16))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (16) - 2, .025) and -0.24346838304287843 = (23.05395551637201 - 28.66686478868555) / 23.05395551637201 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 3c47b2c75dfca5a82a59db238f103cdbda902826
Compare 873775b42431bf27a9ae69633864e5290443c29b
Diff Diff

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformByQuaternionBenchmark - Duration of single invocation 0.79 ns 7.38 ns 9.36 0.33 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector3.TransformByQuaternionBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.3838777325075755 > 0.8277504860603464.
IsChangePoint: Marked as a change because one of 1/22/2023 2:59:30 AM, 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -126.99526607522674 (T) = (0 -7.35893009654001) / Math.Sqrt((0.05410990308401292 / (24)) + (0.00642522359262174 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -8.042009880592124 = (0.8138599928247484 - 7.35893009654001) / 0.8138599928247484 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 3c47b2c75dfca5a82a59db238f103cdbda902826
Compare 873775b42431bf27a9ae69633864e5290443c29b
Diff Diff

Regressions in System.Numerics.Tests.Perf_Vector2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformByQuaternionBenchmark - Duration of single invocation 0.04 ns 2.46 ns 55.28 0.78 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector2*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector2.TransformByQuaternionBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4592848641998617 > 0.003390660399691646.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -27.399889821767825 (T) = (0 -2.5765840624988376) / Math.Sqrt((0.1773866162467119 / (24)) + (0.011222966552333794 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -22.060985317555495 = (0.11172914023484405 - 2.5765840624988376) / 0.11172914023484405 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 3c47b2c75dfca5a82a59db238f103cdbda902826
Compare 873775b42431bf27a9ae69633864e5290443c29b
Diff Diff

Regressions in System.Numerics.Tests.Perf_Vector4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformByQuaternionBenchmark - Duration of single invocation 1.34 ns 8.04 ns 6.02 0.29 False
TransformVector2ByQuaternionBenchmark - Duration of single invocation 0.49 ns 5.80 ns 11.93 0.40 False
TransformVector3ByQuaternionBenchmark - Duration of single invocation 1.30 ns 8.04 ns 6.17 0.26 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector4.TransformByQuaternionBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.038862144980559 > 1.375150017228637.
IsChangePoint: Marked as a change because one of 1/22/2023 2:59:30 AM, 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -253.67002250969566 (T) = (0 -8.003021323808895) / Math.Sqrt((0.0031426920369748467 / (24)) + (0.00903622795642789 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -5.09920332347962 = (1.3121420781301547 - 8.003021323808895) / 1.3121420781301547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.TransformVector2ByQuaternionBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.804749575425005 > 0.504518725267644.
IsChangePoint: Marked as a change because one of 12/28/2022 7:19:51 AM, 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -168.98161044721124 (T) = (0 -5.964098047930127) / Math.Sqrt((0.011060216079014102 / (24)) + (0.009670498095240392 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -12.28745338259838 = (0.44885185115613474 - 5.964098047930127) / 0.44885185115613474 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.TransformVector3ByQuaternionBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.037643934677403 > 1.369175346531895.
IsChangePoint: Marked as a change because one of 1/22/2023 2:59:30 AM, 2/2/2023 2:17:49 AM, 2/7/2023 2:48:42 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 2:48:42 AM.
IsRegressionStdDev: Marked as regression because -196.70057792976218 (T) = (0 -7.9674953362402965) / Math.Sqrt((0.011146751982089 / (24)) + (0.01126782603079823 / (16))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (16) - 2, .025) and -5.409497985068502 = (1.2430763461976722 - 7.9674953362402965) / 1.2430763461976722 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions