[Perf] Linux/x64: 95 Improvements on 4/12/2023 4:47:47 AM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Vector4

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
DivideBenchmark - Duration of single invocation	12.93 ns	10.18 ns	0.79	0.02	True
AddOperatorBenchmark - Duration of single invocation	12.94 ns	9.85 ns	0.76	0.05	True
DivideOperatorBenchmark - Duration of single invocation	13.10 ns	10.92 ns	0.83	0.02	True
MinBenchmark - Duration of single invocation	16.90 ns	14.88 ns	0.88	0.05	False
DistanceSquaredBenchmark - Duration of single invocation	15.40 ns	13.49 ns	0.88	0.01	True
MultiplyFunctionBenchmark - Duration of single invocation	12.81 ns	10.43 ns	0.81	0.01	True
DistanceSquaredJitOptimizeCanaryBenchmark - Duration of single invocation	15.78 ns	13.25 ns	0.84	0.01	True
AddFunctionBenchmark - Duration of single invocation	12.91 ns	10.49 ns	0.81	0.02	True
MultiplyByScalarBenchmark - Duration of single invocation	23.05 ns	19.25 ns	0.84	0.01	True
DivideByScalarBenchmark - Duration of single invocation	25.87 ns	21.65 ns	0.84	0.01	True
NormalizeBenchmark - Duration of single invocation	32.50 ns	27.99 ns	0.86	0.05	False
SubtractFunctionBenchmark - Duration of single invocation	12.88 ns	10.41 ns	0.81	0.01	True
LerpBenchmark - Duration of single invocation	50.46 ns	42.71 ns	0.85	0.00	True
SubtractOperatorBenchmark - Duration of single invocation	12.94 ns	10.46 ns	0.81	0.00	True
MaxBenchmark - Duration of single invocation	16.94 ns	14.20 ns	0.84	0.05	False
ClampBenchmark - Duration of single invocation	36.20 ns	28.61 ns	0.79	0.06	False
MultiplyByScalarOperatorBenchmark - Duration of single invocation	22.82 ns	18.39 ns	0.81	0.00	True
MultiplyOperatorBenchmark - Duration of single invocation	12.91 ns	10.50 ns	0.81	0.02	True
DistanceBenchmark - Duration of single invocation	21.61 ns	19.40 ns	0.90	0.02	True
NegateBenchmark - Duration of single invocation	12.79 ns	10.61 ns	0.83	0.01	True
NegateOperatorBenchmark - Duration of single invocation	12.81 ns	10.36 ns	0.81	0.01	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector4* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector4* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector4.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.182891306853648 < 12.266888143481781.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 35.279464402385884 (T) = (0 -10.577235348322155) / Math.Sqrt((0.010393718209358381 / (299)) + (0.03994866136394515 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.18239645291236575 = (12.936875562733391 - 10.577235348322155) / 12.936875562733391 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.848419289608888 < 12.249234302595246.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 24.453975396126832 (T) = (0 -10.532692575491774) / Math.Sqrt((0.17648035376581664 / (299)) + (0.08007638282603499 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.18443629572110892 = (12.914616626796333 - 10.532692575491774) / 12.914616626796333 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.DivideOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.920097572933651 < 12.412556626636198.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 53.34304652574121 (T) = (0 -10.635285331942761) / Math.Sqrt((0.015030625810661183 / (299)) + (0.016346621285418413 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.17810162230165003 = (12.939903059215045 - 10.635285331942761) / 12.939903059215045 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.MinBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 14.882426879424619 < 16.016064265226568.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 28.382731666475262 (T) = (0 -14.453008963298549) / Math.Sqrt((0.10718454631685023 / (299)) + (0.06372017257042213 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.1519409864622566 = (17.04245663636863 - 14.453008963298549) / 17.04245663636863 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.DistanceSquaredBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 13.488597802686824 < 14.628205161122207.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 67.10803164443665 (T) = (0 -13.245601441589633) / Math.Sqrt((0.04112752786993269 / (299)) + (0.008532812983992244 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.143053803628429 = (15.456748040510998 - 13.245601441589633) / 15.456748040510998 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.MultiplyFunctionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.431950765036886 < 12.213006095390512.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 96.88548675983743 (T) = (0 -10.489332914226113) / Math.Sqrt((0.012804514785828678 / (299)) + (0.005013985731133594 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.184496599651359 = (12.862402424982841 - 10.489332914226113) / 12.862402424982841 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.DistanceSquaredJitOptimizeCanaryBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 13.245515403749392 < 14.65762107784137.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 64.87416819035406 (T) = (0 -13.282564715334399) / Math.Sqrt((0.012436044006028877 / (299)) + (0.009731793137109151 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.14064755787528796 = (15.456481024821233 - 13.282564715334399) / 15.456481024821233 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.AddFunctionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.487036900817767 < 12.198192750164212.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 83.36116577209474 (T) = (0 -10.478464671690237) / Math.Sqrt((0.16977085158288718 / (299)) + (0.0025056770728445602 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.1879303618609277 = (12.903406530137666 - 10.478464671690237) / 12.903406530137666 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.MultiplyByScalarBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.250798774183217 < 21.895578455220395.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 60.528574108722324 (T) = (0 -19.096007262862635) / Math.Sqrt((0.006862276088058057 / (299)) + (0.03772611732157051 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.1706613747900293 = (23.025585306638693 - 19.096007262862635) / 23.025585306638693 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.DivideByScalarBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 21.64937778405403 < 24.529674447296134.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 34.085051047339704 (T) = (0 -21.864415488089044) / Math.Sqrt((0.05816445983195346 / (299)) + (0.11936729491631892 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.15314963398070022 = (25.818510997243582 - 21.864415488089044) / 25.818510997243582 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.NormalizeBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 27.9866758404336 < 31.015659936348122.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 68.3528368314396 (T) = (0 -28.10381102827764) / Math.Sqrt((0.2966827228664183 / (299)) + (0.02611383701861819 / (7))) is greater than 1.9677981407170237 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (7) - 2, .975) and 0.14320775651462037 = (32.80119683851597 - 28.10381102827764) / 32.80119683851597 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.SubtractFunctionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.408311371327605 < 12.25978547374115.
IsChangePoint: Marked as a change because one of 4/11/2023 3:40:00 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 79.73831903173172 (T) = (0 -10.544485587539622) / Math.Sqrt((0.0056893218099886116 / (299)) + (0.008350849815870132 / (10))) is greater than 1.9677212881555053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (10) - 2, .975) and 0.18100157835021616 = (12.874854589217517 - 10.544485587539622) / 12.874854589217517 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.LerpBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 42.71310326543409 < 47.80637703236035.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 65.36730402885897 (T) = (0 -42.30208702795217) / Math.Sqrt((0.03196294142927517 / (299)) + (0.12033899107747821 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.1598001800625719 = (50.347650670887454 - 42.30208702795217) / 50.347650670887454 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.SubtractOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.464274262446692 < 12.205866606501312.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 193.57403568385627 (T) = (0 -10.525330041845189) / Math.Sqrt((0.005296143721374585 / (299)) + (0.0011657237379496485 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.18244703641414448 = (12.874187374577193 - 10.525330041845189) / 12.874187374577193 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.MaxBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 14.195427827851274 < 16.05089263042943.
IsChangePoint: Marked as a change because one of 3/6/2023 11:36:59 AM, 3/6/2023 9:56:15 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 14.952670426492869 (T) = (0 -14.397811712757072) / Math.Sqrt((0.4367710460690113 / (299)) + (0.23796186413102202 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.15502001824054268 = (17.039234092596214 - 14.397811712757072) / 17.039234092596214 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.ClampBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 28.60598868812678 < 33.84980625395032.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 63.423742633789416 (T) = (0 -28.793202561628092) / Math.Sqrt((0.4532542398583695 / (299)) + (0.081444813186276 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.18461466721968073 = (35.31238716724077 - 28.793202561628092) / 35.31238716724077 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.MultiplyByScalarOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 18.38660050776042 < 21.644955281832875.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 48.254708221838285 (T) = (0 -18.73085964365883) / Math.Sqrt((0.004478217097394046 / (299)) + (0.06376642793375475 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.17835980684550123 = (22.79691256551849 - 18.73085964365883) / 22.79691256551849 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.500124799243903 < 12.263171023884437.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 51.10017639167703 (T) = (0 -10.50864356579048) / Math.Sqrt((0.011359126517379494 / (299)) + (0.019117417316658756 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.18441202868118264 = (12.884745650181493 - 10.50864356579048) / 12.884745650181493 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.DistanceBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.401837072118145 < 20.554537263891497.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 128.38564823342477 (T) = (0 -19.374935407384164) / Math.Sqrt((0.0476559451421847 / (299)) + (0.0013651787600749228 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.10744563391957047 = (21.707288814761405 - 19.374935407384164) / 21.707288814761405 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.610696003152894 < 12.189173283967293.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 51.42624165751006 (T) = (0 -10.332956199071571) / Math.Sqrt((0.012858471852456404 / (299)) + (0.021277386225662612 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.19625868153358914 = (12.856071924717648 - 10.332956199071571) / 12.856071924717648 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector4.NegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.358195599673099 < 12.17170315876789.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 57.66744852074033 (T) = (0 -10.38567303788385) / Math.Sqrt((0.011921747690064233 / (299)) + (0.016475976369781858 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.19364468862150075 = (12.87977259073186 - 10.38567303788385) / 12.87977259073186 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Matrix4x4

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
MultiplyByScalarBenchmark - Duration of single invocation	151.79 ns	142.85 ns	0.94	0.04	False
CreateShadowBenchmark - Duration of single invocation	270.06 ns	207.74 ns	0.77	0.00	True
LerpBenchmark - Duration of single invocation	269.83 ns	234.16 ns	0.87	0.01	True
MultiplyByScalarOperatorBenchmark - Duration of single invocation	151.63 ns	135.54 ns	0.89	0.03	False
CreateScaleFromVectorBenchmark - Duration of single invocation	67.16 ns	63.37 ns	0.94	0.03	False
CreatePerspectiveOffCenterBenchmark - Duration of single invocation	71.89 ns	68.25 ns	0.95	0.08	False
CreateWorldBenchmark - Duration of single invocation	182.13 ns	147.51 ns	0.81	0.01	True
CreateReflectionBenchmark - Duration of single invocation	257.73 ns	202.00 ns	0.78	0.00	True
CreateFromQuaternionBenchmark - Duration of single invocation	75.69 ns	69.94 ns	0.92	0.02	False
CreateOrthographicOffCenterBenchmark - Duration of single invocation	64.91 ns	60.06 ns	0.93	0.03	False
TranslationBenchmark - Duration of single invocation	57.77 ns	54.54 ns	0.94	0.12	False
CreateScaleFromScalarWithCenterBenchmark - Duration of single invocation	92.88 ns	80.80 ns	0.87	0.04	True
SubtractBenchmark - Duration of single invocation	158.99 ns	148.27 ns	0.93	0.04	False
CreateRotationZBenchmark - Duration of single invocation	85.04 ns	79.10 ns	0.93	0.07	False
CreateScaleFromScalarXYZBenchmark - Duration of single invocation	92.92 ns	79.72 ns	0.86	0.06	False
CreateOrthographicBenchmark - Duration of single invocation	59.73 ns	55.61 ns	0.93	0.06	False
CreateScaleFromVectorWithCenterBenchmark - Duration of single invocation	94.28 ns	84.05 ns	0.89	0.01	True
CreateConstrainedBillboardBenchmark - Duration of single invocation	249.08 ns	202.75 ns	0.81	0.01	True
CreateScaleFromScalarXYZWithCenterBenchmark - Duration of single invocation	92.79 ns	79.85 ns	0.86	0.05	False
CreateTranslationFromVectorBenchmark - Duration of single invocation	74.83 ns	70.70 ns	0.94	0.05	False
MultiplyByMatrixBenchmark - Duration of single invocation	554.27 ns	475.29 ns	0.86	0.00	True
MultiplyByMatrixOperatorBenchmark - Duration of single invocation	560.22 ns	474.84 ns	0.85	0.01	True
CreateLookAtBenchmark - Duration of single invocation	184.62 ns	153.09 ns	0.83	0.01	True
CreateBillboardBenchmark - Duration of single invocation	190.26 ns	158.87 ns	0.83	0.01	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Matrix4x4* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Matrix4x4* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 142.84680847260535 < 144.03030464669175.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 14.865043851583337 (T) = (0 -136.53253837983843) / Math.Sqrt((4.985984603695461 / (299)) + (10.377091226774667 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.10534636301116995 = (152.60938170370738 - 136.53253837983843) / 152.60938170370738 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateShadowBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 207.7444762958147 < 256.9773288618938.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 340.8429471196193 (T) = (0 -207.54882548705513) / Math.Sqrt((4.503245811284911 / (299)) + (0.1568953318498382 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.228412438915568 = (268.9893356955554 - 207.54882548705513) / 268.9893356955554 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.LerpBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 234.15859439697834 < 254.39462754769733.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 34.04673546889976 (T) = (0 -235.59158343392969) / Math.Sqrt((6.657227577754024 / (299)) + (8.591302665143806 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.12498202200924663 = (269.2419920044423 - 235.59158343392969) / 269.2419920044423 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 135.54423115075014 < 144.0487804138372.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 19.4937981756403 (T) = (0 -135.25562782192904) / Math.Sqrt((3.527148132907299 / (299)) + (6.743744619112924 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.11169289056439385 = (152.26223722093692 - 135.25562782192904) / 152.26223722093692 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromVectorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 63.3729377945413 < 63.77389175878884.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 6.487172189451257 (T) = (0 -64.08861100849501) / Math.Sqrt((0.6318998160418889 / (299)) + (2.450309866702361 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.05322086346129659 = (67.69119484697805 - 64.08861100849501) / 67.69119484697805 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveOffCenterBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 68.24823339328925 < 68.37416109383132.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 39.20305385177162 (T) = (0 -67.82917910586502) / Math.Sqrt((1.3798962177193297 / (299)) + (0.07675795446827643 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.06214387920030758 = (72.32365135926004 - 67.82917910586502) / 72.32365135926004 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateWorldBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 147.51218324819826 < 172.22889640051045.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 190.31978225000904 (T) = (0 -147.97501135016003) / Math.Sqrt((1.7610135939928442 / (299)) + (0.22116766418814468 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.1833303782313235 = (181.1932358028548 - 147.97501135016003) / 181.1932358028548 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateReflectionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 201.9995874635129 < 244.33536543522865.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 160.18556437978864 (T) = (0 -202.2125825146057) / Math.Sqrt((4.663699834646537 / (299)) + (0.9143571735251741 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.21333251382462942 = (257.04962524601757 - 202.2125825146057) / 257.04962524601757 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateFromQuaternionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 69.94427401667582 < 70.04826795095862.
IsChangePoint: Marked as a change because one of 2/8/2023 7:09:09 PM, 4/12/2023 4:47:47 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 12.783816675758853 (T) = (0 -69.95185991565114) / Math.Sqrt((0.8164936668394559 / (299)) + (0.7972859534593351 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.055247334540846585 = (74.04251130813616 - 69.95185991565114) / 74.04251130813616 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateOrthographicOffCenterBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 60.057091043856836 < 61.199619924023885.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 69.85141000827086 (T) = (0 -60.09131670679046) / Math.Sqrt((0.7261582745388558 / (299)) + (0.014706494121343328 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.06898080930876031 = (64.54358546806685 - 60.09131670679046) / 64.54358546806685 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.TranslationBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 54.53824903610543 < 55.06361058216388.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 33.53745604974687 (T) = (0 -54.666111735445014) / Math.Sqrt((2.0290551964058774 / (299)) + (0.05258233615395204 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.06621333081274937 = (58.54239896466429 - 54.666111735445014) / 58.54239896466429 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarWithCenterBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 80.79728865986337 < 88.33046220862408.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 20.658870954971768 (T) = (0 -81.23114170657428) / Math.Sqrt((1.478275665987767 / (299)) + (2.7628386693144455 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.130830209079576 = (93.45831223672995 - 81.23114170657428) / 93.45831223672995 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 148.26580876831392 < 149.91613933957248.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 50.43675242634965 (T) = (0 -148.608024342666) / Math.Sqrt((6.0369747719359825 / (299)) + (0.18154256186149345 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.0656675406337785 = (159.0526186401253 - 148.608024342666) / 159.0526186401253 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateRotationZBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 79.09605067703296 < 79.72243227886992.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 37.30633897279971 (T) = (0 -79.36662166446642) / Math.Sqrt((2.9681308415585628 / (299)) + (0.04994399237905491 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.056400817292667174 = (84.11052395864867 - 79.36662166446642) / 84.11052395864867 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarXYZBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 79.721937813145 < 88.27393911084839.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 53.409219639537056 (T) = (0 -80.46420531793888) / Math.Sqrt((1.893035235503016 / (299)) + (0.466533413759774 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.1379958229629364 = (93.34549351549042 - 80.46420531793888) / 93.34549351549042 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateOrthographicBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 55.6121513588301 < 56.76507185683635.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 33.17360252972923 (T) = (0 -55.751622919543514) / Math.Sqrt((1.1661086733383252 / (299)) + (0.10019585998775073 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.07085414282279745 = (60.00309046086702 - 55.751622919543514) / 60.00309046086702 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromVectorWithCenterBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 84.04549530134202 < 89.78250582638113.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 39.615680000025705 (T) = (0 -81.8562270460849) / Math.Sqrt((0.6872893316882871 / (299)) + (0.8314758190615803 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.13624915365476187 = (94.76833208608777 - 81.8562270460849) / 94.76833208608777 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateConstrainedBillboardBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 202.7525104409778 < 235.96440433989633.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 114.57941032526261 (T) = (0 -199.7187479617307) / Math.Sqrt((2.9904430791430885 / (299)) + (1.5020207357493685 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.19438769264400918 = (247.90925627390806 - 199.7187479617307) / 247.90925627390806 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarXYZWithCenterBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 79.85112660182524 < 88.19835732148587.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 43.38008559866888 (T) = (0 -80.28540552152863) / Math.Sqrt((1.9564652894298507 / (299)) + (0.7657229018207616 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.14056319604729292 = (93.41629908363403 - 80.28540552152863) / 93.41629908363403 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateTranslationFromVectorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 70.70448680936826 < 71.09480339057194.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 9.542921831392142 (T) = (0 -71.29774469984936) / Math.Sqrt((1.9396900967347437 / (299)) + (1.459653232761002 / (7))) is greater than 1.9677981407170237 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (7) - 2, .975) and 0.0584364317902482 = (75.72270965773644 - 71.29774469984936) / 75.72270965773644 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByMatrixBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 475.29439262434295 < 530.7316763309741.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 119.16719870125235 (T) = (0 -474.92567419345414) / Math.Sqrt((25.561880025217853 / (299)) + (3.7149761229505978 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.15046725270554728 = (559.0433985105018 - 474.92567419345414) / 559.0433985105018 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByMatrixOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 474.83837832598147 < 533.0937660820395.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 63.88675925042646 (T) = (0 -473.481359723878) / Math.Sqrt((24.718231936372174 / (299)) + (12.13525051628531 / (7))) is greater than 1.9677981407170237 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (7) - 2, .975) and 0.1538646460758728 = (559.5811090128908 - 473.481359723878) / 559.5811090128908 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateLookAtBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 153.09264826889583 < 174.93428182457862.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 155.33999750092735 (T) = (0 -153.29659553171626) / Math.Sqrt((2.572483112579258 / (299)) + (0.2889436394397068 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.16974749540571688 = (184.6385222368311 - 153.29659553171626) / 184.6385222368311 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Matrix4x4.CreateBillboardBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 158.86845064864644 < 180.7412595682558.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 77.90093627468863 (T) = (0 -159.9119453023233) / Math.Sqrt((2.457492079287013 / (299)) + (1.3899187289491928 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.16420757206936348 = (191.3297368561403 - 159.9119453023233) / 191.3297368561403 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in SIMD.RayTracerBench

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Bench - Duration of single invocation	854.90 ms	720.81 ms	0.84	0.00	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'SIMD.RayTracerBench*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter SIMD.RayTracerBench* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'SIMD.RayTracerBench*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter SIMD.RayTracerBench* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

SIMD.RayTracerBench.Bench

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 720.8106152857142 < 812.0533657898352.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 174.1059938612519 (T) = (0 -721351364.0518517) / Math.Sqrt((12740357930276.13 / (299)) + (5022856891456.707 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.1575876719302272 = (856292506.6691401 - 721351364.0518517) / 856292506.6691401 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in PerfLabTests.CastingPerf2.CastingPerf

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IFooFooIsIFoo - Duration of single invocation	174.02 μs	152.93 μs	0.88	0.02	False
FooObjIsNull - Duration of single invocation	173.99 μs	152.97 μs	0.88	0.02	False
IFooObjIsIFoo - Duration of single invocation	195.86 μs	174.07 μs	0.89	0.04	False
ObjFooIsObj - Duration of single invocation	173.98 μs	153.00 μs	0.88	0.04	False
FooObjIsFoo - Duration of single invocation	304.38 μs	280.91 μs	0.92	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf2.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf2.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 152.92732920792082 < 165.35612521340812.
IsChangePoint: Marked as a change because one of 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 17.567312388711382 (T) = (0 -153034.09346740105) / Math.Sqrt((171992672.29980397 / (299)) + (10822.185560458132 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.08017717665854414 = (166373.4466943009 - 153034.09346740105) / 166373.4466943009 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.CastingPerf2.CastingPerf.FooObjIsNull

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 152.97240691747572 < 165.32540809001068.
IsChangePoint: Marked as a change because one of 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 19.333885840478995 (T) = (0 -152991.84814381506) / Math.Sqrt((133744837.90364215 / (299)) + (555.0308661511096 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.07793772268613172 = (165923.55192049238 - 152991.84814381506) / 165923.55192049238 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.CastingPerf2.CastingPerf.IFooObjIsIFoo

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 174.07013872863246 < 186.07836530368587.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 20.219682331615225 (T) = (0 -174082.93282850453) / Math.Sqrt((125651689.59579478 / (299)) + (17139.68971467009 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.07018848537470153 = (187223.89440257428 - 174082.93282850453) / 187223.89440257428 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.CastingPerf2.CastingPerf.ObjFooIsObj

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 153.00263890574433 < 165.39607462068196.
IsChangePoint: Marked as a change because one of 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 15.099739894458672 (T) = (0 -152980.08287029708) / Math.Sqrt((292148715.49894917 / (299)) + (992.668386539176 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.08889866298594541 = (167906.77025198704 - 152980.08287029708) / 167906.77025198704 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.CastingPerf2.CastingPerf.FooObjIsFoo

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 280.910106993007 < 289.3427727564103.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 15.167868097018923 (T) = (0 -280890.73320618906) / Math.Sqrt((329483913.09234595 / (299)) + (7959.89784982625 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.05366460564028138 = (296819.43091248005 - 280890.73320618906) / 296819.43091248005 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Benchstone.BenchI.IniArray

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Test - Duration of single invocation	313.11 ms	165.40 ms	0.53	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Benchstone.BenchI.IniArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Benchstone.BenchI.IniArray* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Benchstone.BenchI.IniArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Benchstone.BenchI.IniArray* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.IniArray.Test

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 165.39589873333333 < 297.4705464473901.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/8/2023 9:36:36 PM, 3/13/2023 12:04:12 PM, 3/29/2023 11:36:35 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 13.730012662731784 (T) = (0 -164614336.50740743) / Math.Sqrt((4002020083233731 / (299)) + (275044530100.90826 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.23400638363390902 = (214903013.5373 - 164614336.50740743) / 214903013.5373 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Quaternion

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
InverseBenchmark - Duration of single invocation	41.73 ns	35.56 ns	0.85	0.02	False
ConjugateBenchmark - Duration of single invocation	28.74 ns	23.96 ns	0.83	0.02	False
NegationOperatorBenchmark - Duration of single invocation	20.33 ns	19.15 ns	0.94	0.02	False
MultiplyByScalarBenchmark - Duration of single invocation	20.15 ns	19.08 ns	0.95	0.01	False
NegateBenchmark - Duration of single invocation	20.27 ns	19.16 ns	0.95	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Quaternion*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Quaternion* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Quaternion*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Quaternion* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Quaternion.InverseBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 35.56194399862205 < 39.53980469121658.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/8/2023 7:09:09 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 403.14178188640557 (T) = (0 -35.53601872745185) / Math.Sqrt((0.008011486866041459 / (299)) + (0.0018093428302944764 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.14620158824639562 = (41.62108788005933 - 35.53601872745185) / 41.62108788005933 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Quaternion.ConjugateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 23.961316306314554 < 27.32011950776359.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 73.79785972051612 (T) = (0 -24.077645548683574) / Math.Sqrt((0.004805569456147259 / (299)) + (0.03194213672186846 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.1625220998771773 = (28.75018617822918 - 24.077645548683574) / 28.75018617822918 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Quaternion.NegationOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.14960585718434 < 19.272273576858883.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/8/2023 7:09:09 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 65.5244695715164 (T) = (0 -19.18566573414095) / Math.Sqrt((0.06421800725541736 / (299)) + (0.000687038987725512 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.05506310542095876 = (20.30364762367327 - 19.18566573414095) / 20.30364762367327 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Quaternion.MultiplyByScalarBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.082061691107555 < 19.148419231732227.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 55.81365819145865 (T) = (0 -19.101470760368674) / Math.Sqrt((0.0027626604064674687 / (299)) + (0.0029450871233011053 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.05371347147775244 = (20.185715620614577 - 19.101470760368674) / 20.185715620614577 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Quaternion.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.162595946825196 < 19.2769405419492.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/8/2023 7:09:09 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 206.7568387187587 (T) = (0 -19.160181711592873) / Math.Sqrt((0.0022054425427247893 / (299)) + (0.00017610494792688294 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.05526666250103379 = (20.281047520050958 - 19.160181711592873) / 20.281047520050958 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in PerfLabTests.LowLevelPerf

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SealedClassInterfaceMethod - Duration of single invocation	217.49 μs	152.39 μs	0.70	0.01	False
GenericClassGenericStaticMethod - Duration of single invocation	217.56 μs	174.33 μs	0.80	0.32	False
GenericClassWithSTringGenericInstanceMethod - Duration of single invocation	87.07 μs	76.14 μs	0.87	0.04	False
ClassVirtualMethod - Duration of single invocation	217.80 μs	152.28 μs	0.70	0.03	False
InterfaceInterfaceMethodSwitchCallType - Duration of single invocation	587.03 μs	500.38 μs	0.85	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.LowLevelPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.LowLevelPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.LowLevelPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.LowLevelPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.SealedClassInterfaceMethod

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 152.39223708391123 < 206.69649190872065.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 18.603203853229175 (T) = (0 -152536.62621272684) / Math.Sqrt((644012982.1714052 / (299)) + (8664.961263922378 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.15184413937242025 = (179845.04180618376 - 152536.62621272684) / 179845.04180618376 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 174.3286115046296 < 206.72564672743053.
IsChangePoint: Marked as a change because one of 2/15/2023 8:48:29 AM, 3/9/2023 8:37:21 PM, 3/20/2023 4:51:16 PM, 3/29/2023 11:36:35 PM, 4/3/2023 9:14:01 PM, 4/11/2023 10:44:16 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 6.457093930321099 (T) = (0 -180107.1805225764) / Math.Sqrt((211963365.0773877 / (299)) + (101795543.82381716 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.10165848984856425 = (200488.54303995735 - 180107.1805225764) / 200488.54303995735 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 76.14389658460472 < 82.69208503473676.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 16.639010291267894 (T) = (0 -76176.09725451119) / Math.Sqrt((52154096.179103285 / (299)) + (6675.71643220673 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.08376201736147437 = (83140.07790328006 - 76176.09725451119) / 83140.07790328006 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.LowLevelPerf.ClassVirtualMethod

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 152.2821261435773 < 206.7126711183449.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 15.343280116239283 (T) = (0 -153518.00788587562) / Math.Sqrt((645198944.3250545 / (299)) + (5415469.670850963 / (7))) is greater than 1.9677981407170237 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (7) - 2, .975) and 0.14611730215561541 = (179788.17028782732 - 153518.00788587562) / 179788.17028782732 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.LowLevelPerf.InterfaceInterfaceMethodSwitchCallType

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 500.3756052707373 < 558.0158427904541.
IsChangePoint: Marked as a change because one of 1/30/2023 9:42:46 PM, 2/2/2023 5:46:53 AM, 2/8/2023 7:09:09 PM, 3/13/2023 12:04:12 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 4.983484780309939 (T) = (0 -530235.0993920828) / Math.Sqrt((128496496.1818978 / (299)) + (809899652.9582807 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.0820385553524593 = (577622.4072195877 - 530235.0993920828) / 577622.4072195877 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Vector3

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ReflectBenchmark - Duration of single invocation	54.01 ns	39.22 ns	0.73	0.03	False
LerpBenchmark - Duration of single invocation	49.93 ns	35.53 ns	0.71	0.02	True
SubtractOperatorBenchmark - Duration of single invocation	15.08 ns	10.53 ns	0.70	0.01	True
ClampBenchmark - Duration of single invocation	33.62 ns	25.39 ns	0.76	0.01	True
DivideByVector3Benchmark - Duration of single invocation	17.45 ns	10.94 ns	0.63	0.02	True
MaxBenchmark - Duration of single invocation	19.66 ns	13.24 ns	0.67	0.00	True
MultiplyFunctionBenchmark - Duration of single invocation	15.11 ns	10.43 ns	0.69	0.04	True
NegateBenchmark - Duration of single invocation	15.10 ns	9.84 ns	0.65	0.00	True
DistanceBenchmark - Duration of single invocation	23.17 ns	13.66 ns	0.59	0.01	True
DivideByScalarBenchmark - Duration of single invocation	25.49 ns	16.45 ns	0.65	0.00	True
DivideByScalarOperatorBenchmark - Duration of single invocation	25.44 ns	16.37 ns	0.64	0.01	True
MultiplyOperatorBenchmark - Duration of single invocation	15.08 ns	10.41 ns	0.69	0.01	True
DivideByVector3OperatorBenchmark - Duration of single invocation	17.47 ns	10.89 ns	0.62	0.02	True
NegateOperatorBenchmark - Duration of single invocation	15.05 ns	9.81 ns	0.65	0.02	True
AddFunctionBenchmark - Duration of single invocation	15.14 ns	10.33 ns	0.68	0.01	True
NormalizeBenchmark - Duration of single invocation	36.62 ns	28.19 ns	0.77	0.09	False
AbsBenchmark - Duration of single invocation	26.74 ns	19.11 ns	0.71	0.01	False
CrossBenchmark - Duration of single invocation	16.93 ns	11.91 ns	0.70	0.01	True
MinBenchmark - Duration of single invocation	19.55 ns	13.20 ns	0.68	0.00	True
MultiplyByScalarOperatorBenchmark - Duration of single invocation	23.25 ns	16.22 ns	0.70	0.00	True
MultiplyByScalarBenchmark - Duration of single invocation	23.34 ns	16.35 ns	0.70	0.01	True
AddOperatorBenchmark - Duration of single invocation	15.11 ns	10.45 ns	0.69	0.02	True
DistanceSquaredBenchmark - Duration of single invocation	16.91 ns	9.42 ns	0.56	0.01	True
SubtractFunctionBenchmark - Duration of single invocation	15.14 ns	10.43 ns	0.69	0.01	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector3* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector3* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector3.ReflectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 39.21882697003968 < 51.44763699782673.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 92.84203514858363 (T) = (0 -39.40621715561248) / Math.Sqrt((0.3657484065889466 / (299)) + (0.22044188547589938 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.2742172889741165 = (54.29478624520052 - 39.40621715561248) / 54.29478624520052 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.LerpBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 35.52792023849395 < 47.122623229391486.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 156.3108981918168 (T) = (0 -35.90718638037336) / Math.Sqrt((0.43026381121132384 / (299)) + (0.05944979195871289 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.280805168800266 = (49.92692497591276 - 35.90718638037336) / 49.92692497591276 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.SubtractOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.530746181657577 < 14.336159827709528.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 268.42070580772935 (T) = (0 -10.44834670313702) / Math.Sqrt((0.026508188286969277 / (299)) + (0.0017309973472036527 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.3097181969323829 = (15.136349613599105 - 10.44834670313702) / 15.136349613599105 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.ClampBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 25.39015716287521 < 31.928396047499735.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 167.7535479007131 (T) = (0 -25.050611894535646) / Math.Sqrt((0.02654348745640917 / (299)) + (0.022509835719674492 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.254173348415538 = (33.5877135006066 - 25.050611894535646) / 33.5877135006066 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.DivideByVector3Benchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.944624339305914 < 17.013462780201053.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 40.04223449100527 (T) = (0 -11.015892642829732) / Math.Sqrt((0.04136120722829104 / (299)) + (0.2373044184283699 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.3717758369311167 = (17.53497125773856 - 11.015892642829732) / 17.53497125773856 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.MaxBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 13.240022240368967 < 18.673051273122486.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 105.4238566022558 (T) = (0 -13.367760347526149) / Math.Sqrt((0.007860472274625885 / (299)) + (0.03235093890828094 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.3218284133747983 = (19.71147215714004 - 13.367760347526149) / 19.71147215714004 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.MultiplyFunctionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.428387044411027 < 14.334175698385112.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 203.66854124782174 (T) = (0 -10.475575649351946) / Math.Sqrt((0.04569846113487079 / (299)) + (0.003338474132901906 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.30793893810831297 = (15.136779434921387 - 10.475575649351946) / 15.136779434921387 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.836632725348602 < 14.328340314831227.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 132.71170463648653 (T) = (0 -9.83526934259936) / Math.Sqrt((0.005487068417669515 / (299)) + (0.012448416228447143 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.348705288980716 = (15.101104271531772 - 9.83526934259936) / 15.101104271531772 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.DistanceBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 13.659923605062097 < 21.96465061283717.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 99.21666233886205 (T) = (0 -13.94586737769699) / Math.Sqrt((0.012916928367992726 / (299)) + (0.059976552169036086 / (7))) is greater than 1.9677981407170237 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (7) - 2, .975) and 0.3976614984092758 = (23.152873908719354 - 13.94586737769699) / 23.152873908719354 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.DivideByScalarBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.45082247361065 < 24.592832763934684.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 432.0479598852343 (T) = (0 -16.456920134801727) / Math.Sqrt((0.02551330427429061 / (299)) + (0.0032271351907132545 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.3561380698828696 = (25.55970366474053 - 16.456920134801727) / 25.55970366474053 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.DivideByScalarOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.36735984638799 < 24.222735502229007.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 162.12390245736253 (T) = (0 -16.128391696370876) / Math.Sqrt((0.05059912698851513 / (299)) + (0.0289935269438559 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.3692175020956628 = (25.56886367322269 - 16.128391696370876) / 25.56886367322269 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.414917147613634 < 14.338336421004945.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 317.2759918611052 (T) = (0 -10.41456752496723) / Math.Sqrt((0.03013335194541409 / (299)) + (0.0010898578106433294 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.31214075263142804 = (15.140550286717085 - 10.41456752496723) / 15.140550286717085 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.DivideByVector3OperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.887577502562753 < 16.597420622544902.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 70.47318114759524 (T) = (0 -10.982416086074938) / Math.Sqrt((0.07770876126418508 / (299)) + (0.07531430977205075 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.3734543913446803 = (17.52851817068065 - 10.982416086074938) / 17.52851817068065 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.NegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.806731145013005 < 14.328703612279874.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 295.26815596071214 (T) = (0 -9.81439786031449) / Math.Sqrt((0.016682506212547504 / (299)) + (0.0021240577688908914 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.3503430233050381 = (15.107046044889492 - 9.81439786031449) / 15.107046044889492 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.AddFunctionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.331367519521988 < 14.328095641416908.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 63.84292645653406 (T) = (0 -10.40185689737753) / Math.Sqrt((0.009474933192590016 / (299)) + (0.04892910982355844 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.31217805882721117 = (15.122891950264876 - 10.40185689737753) / 15.122891950264876 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.NormalizeBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 28.19223582660893 < 33.12801005243554.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 90.73042332920937 (T) = (0 -28.349510550948416) / Math.Sqrt((0.6131154448483531 / (299)) + (0.03213426099844834 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.19351387117597188 = (35.15188859142072 - 28.349510550948416) / 35.15188859142072 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.AbsBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.105068913347075 < 25.483750748702327.
IsChangePoint: Marked as a change because one of 3/8/2023 9:36:36 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 61.07653902904293 (T) = (0 -19.06917748612798) / Math.Sqrt((2.969753548421966 / (299)) + (0.008791096836762446 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.2517198807143757 = (25.484009256230323 - 19.06917748612798) / 25.484009256230323 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.CrossBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 11.908516893781133 < 16.041508496029152.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 91.79233210606886 (T) = (0 -11.931801292387897) / Math.Sqrt((0.04391587812087744 / (299)) + (0.025398077123919456 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.29536606229930434 = (16.93333325857506 - 11.931801292387897) / 16.93333325857506 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.MinBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 13.204391273421193 < 18.547510125248387.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 159.54958037953156 (T) = (0 -13.307589815988058) / Math.Sqrt((0.012867458695692804 / (299)) + (0.011421972302538148 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.3149753372449546 = (19.426438987564822 - 13.307589815988058) / 19.426438987564822 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.MultiplyByScalarOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.221761314674275 < 22.109515762527234.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 93.64178755760317 (T) = (0 -16.3793575779041) / Math.Sqrt((0.00797678962620926 / (299)) + (0.044195003994537845 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.2987140667205195 = (23.356175848712628 - 16.3793575779041) / 23.356175848712628 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.MultiplyByScalarBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.351548215615903 < 22.202970971181784.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 194.64808959849708 (T) = (0 -16.435061476078882) / Math.Sqrt((0.01564657990092092 / (299)) + (0.011072709690697236 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.297831670788364 = (23.406156034595654 - 16.435061476078882) / 23.406156034595654 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.453847275334368 < 14.524367185673517.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 70.59974593949731 (T) = (0 -10.498045702096782) / Math.Sqrt((0.0703248301296923 / (299)) + (0.03743160675290339 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.30834035245593305 = (15.178051429446636 - 10.498045702096782) / 15.178051429446636 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.DistanceSquaredBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.415940346899534 < 16.096624634521593.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 68.68329029431372 (T) = (0 -9.35477919098716) / Math.Sqrt((0.006334451485072639 / (299)) + (0.10930681887300686 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.44746448567216524 = (16.930638752456204 - 9.35477919098716) / 16.930638752456204 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_Vector3.SubtractFunctionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.433226528577016 < 14.352291992301502.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 36.964056250043406 (T) = (0 -10.547260731815486) / Math.Sqrt((0.029246039040621247 / (299)) + (0.13983281399963873 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.3046930592956261 = (15.1692153700216 - 10.547260731815486) / 15.1692153700216 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_Plane

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TransformByQuaternionBenchmark - Duration of single invocation	36.79 ns	34.33 ns	0.93	0.01	False
CreateFromVerticesBenchmark - Duration of single invocation	93.83 ns	74.37 ns	0.79	0.01	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Plane*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Plane* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Plane*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Plane* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Plane.TransformByQuaternionBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 34.33042609817571 < 34.96132472479008.
IsChangePoint: Marked as a change because one of 2/8/2023 7:09:09 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 71.01767938761587 (T) = (0 -34.44657659023065) / Math.Sqrt((0.10280552769607068 / (299)) + (0.007194529500010484 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.06776524463169423 = (36.95053889792121 - 34.44657659023065) / 36.95053889792121 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_Plane.CreateFromVerticesBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 74.37407022652758 < 89.66266810812249.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 298.87588395439326 (T) = (0 -74.23757742416436) / Math.Sqrt((0.28645307138605747 / (299)) + (0.03154690725418972 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.2119549960575256 = (94.2047434509001 - 74.23757742416436) / 94.2047434509001 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ForEach_Array - Duration of single invocation	29.02 ns	26.36 ns	0.91	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 26.364152074538513 < 27.62793880350558.
IsChangePoint: Marked as a change because one of 2/15/2023 8:48:29 AM, 3/13/2023 12:04:12 PM, 3/29/2023 11:36:35 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 29.09251788962236 (T) = (0 -26.481173872281676) / Math.Sqrt((0.6272443065535924 / (299)) + (0.04789292832442291 / (7))) is greater than 1.9677981407170237 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (7) - 2, .975) and 0.09409903179212083 = (29.231863969268858 - 26.481173872281676) / 29.231863969268858 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Reflection.Invoke

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Property_Get_class - Duration of single invocation	13.59 ns	12.51 ns	0.92	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Reflection.Invoke*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Reflection.Invoke* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Reflection.Invoke*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Reflection.Invoke* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Reflection.Invoke.Property_Get_class

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 12.510935171862513 < 13.313739332656437.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/29/2023 11:36:35 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 6.424508922124631 (T) = (0 -12.622532667539753) / Math.Sqrt((0.25645754946935473 / (299)) + (0.1918954724014828 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.07045911530278036 = (13.579319506372551 - 12.622532667539753) / 13.579319506372551 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in PerfLabTests.CastingPerf

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ObjFooIsObj2 - Duration of single invocation	87.02 μs	76.17 μs	0.88	0.13	False
CheckIsInstAnyIsInterfaceNo - Duration of single invocation	152.33 μs	117.37 μs	0.77	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'PerfLabTests.CastingPerf*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter PerfLabTests.CastingPerf* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.ObjFooIsObj2

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 76.17222483963245 < 82.70429844384347.
IsChangePoint: Marked as a change because one of 2/20/2023 10:31:48 AM, 3/6/2023 8:17:12 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 33.66459011307414 (T) = (0 -76475.3180912282) / Math.Sqrt((25224315.9642291 / (299)) + (608847.0624279506 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.14648755767198504 = (89600.70679536482 - 76475.3180912282) / 89600.70679536482 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

PerfLabTests.CastingPerf.CheckIsInstAnyIsInterfaceNo

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 117.37058599624058 < 144.65952081498497.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/13/2023 12:04:12 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 19.473713686094197 (T) = (0 -117521.55097083352) / Math.Sqrt((150509550.80081406 / (299)) + (72599.43753775266 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.10604130011339123 = (131461.94671603973 - 117521.55097083352) / 131461.94671603973 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Benchstone.BenchF.InvMt

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Test - Duration of single invocation	6.05 ms	5.08 ms	0.84	0.00	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Benchstone.BenchF.InvMt*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Benchstone.BenchF.InvMt* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Benchstone.BenchF.InvMt*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Benchstone.BenchF.InvMt* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

Benchstone.BenchF.InvMt.Test

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 5.084046974759615 < 5.747992615235805.
IsChangePoint: Marked as a change because one of 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 265.3120076907737 (T) = (0 -5088186.59667627) / Math.Sqrt((2512616557.4454155 / (299)) + (33755404.9076006 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.15382103365548672 = (6013132.91756376 - 5088186.59667627) / 6013132.91756376 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.MathBenchmarks.Single

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Ceiling - Duration of single invocation	32.72 μs	30.54 μs	0.93	0.22	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.MathBenchmarks.Single*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.MathBenchmarks.Single* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.MathBenchmarks.Single*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.MathBenchmarks.Single* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Single.Ceiling

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 30.53803505789302 < 31.022337842564806.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/22/2023 7:51:01 PM, 4/3/2023 12:37:39 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 7.861033501747513 (T) = (0 -28724.377978958175) / Math.Sqrt((2118539.8268812844 / (299)) + (1824381.9122236113 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.11138796594530719 = (32324.993223297075 - 28724.377978958175) / 32324.993223297075 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b26dae224e06bbce1dc4713f9c2f475f91dd9d3d
Compare	e7ed417c2bf4a24a0694a387f33ba843a2318fdc
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in Benchstone.BenchI.CSieve

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Test - Duration of single invocation	8.44 ms	6.67 ms	0.79	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Benchstone.BenchI.CSieve*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Benchstone.BenchI.CSieve* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Benchstone.BenchI.CSieve*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter Benchstone.BenchI.CSieve* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.CSieve.Test

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.668559816506411 < 8.015720069080528.
IsChangePoint: Marked as a change because one of 2/15/2023 12:21:17 PM, 3/8/2023 9:36:36 PM, 3/13/2023 12:04:12 PM, 3/29/2023 11:36:35 PM, 4/11/2023 10:53:42 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 15.541777127623478 (T) = (0 -6675140.169622824) / Math.Sqrt((495550877949.9001 / (299)) + (28704584.953558195 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.08666587081799862 = (7308541.262551089 - 6675140.169622824) / 7308541.262551089 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[kotlarmilos]

Could be dotnet/runtime#84289.