Skip to content

[Perf] Linux/arm64: 131 Improvements on 4/12/2023 2:59:13 PM #16580

@performanceautofiler

Description

@performanceautofiler

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7a57f6dd800e77508a4ba405e623059935c82f43
Compare 887c043eb94be364188e2b23a87efa214ea57f1e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Float

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EqualsStaticBenchmark - Duration of single invocation 32.55 ns 1.03 ns 0.03 0.07 False
ConditionalSelectBenchmark - Duration of single invocation 15.46 ns 1.57 ns 0.10 0.08 False
GreaterThanBenchmark - Duration of single invocation 33.75 ns 1.03 ns 0.03 0.05 False
DotBenchmark - Duration of single invocation 66.27 ns 0.20 ns 0.00 0.06 False
GreaterThanOrEqualAnyBenchmark - Duration of single invocation 17.64 ns 0.02 ns 0.00 0.06 True
AndNotBenchmark - Duration of single invocation 13.52 ns 1.13 ns 0.08 0.10 False
EqualsAnyBenchmark - Duration of single invocation 17.05 ns 0.01 ns 0.00 0.09 True

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Float* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Float* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.EqualsStaticBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0270593503047862 < 32.128148146500514.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 134.13511909897468 (T) = (0 -1.1129266448739992) / Math.Sqrt((21.87177611818811 / (299)) + (0.0021429163217725963 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9703060011183878 = (37.47985070354307 - 1.1129266448739992) / 37.47985070354307 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.ConditionalSelectBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.573285622443706 < 14.7003108836523.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 130.10007951853783 (T) = (0 -1.251146094564185) / Math.Sqrt((0.4163178643412197 / (299)) + (0.04682975608600526 / (4))) is greater than 1.9678765312856872 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (4) - 2, .975) and 0.9224896784502512 = (16.14167080652809 - 1.251146094564185) / 16.14167080652809 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.GreaterThanBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.029227763386266 < 32.025086275978914.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 134.98229198053733 (T) = (0 -1.1455294158601286) / Math.Sqrt((21.133947561567314 / (299)) + (0.010969337751584903 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9694470175721855 = (37.49321096775417 - 1.1455294158601286) / 37.49321096775417 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.DotBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.195524138156402 < 63.358839027896096.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/17/2023 5:51:51 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 518.3392843296874 (T) = (0 -0.16921182231001283) / Math.Sqrt((2.5253784708334206 / (299)) + (0.023834842107550165 / (3))) is greater than 1.9679030112610798 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (3) - 2, .975) and 0.9974566420171842 = (66.53087117633495 - 0.16921182231001283) / 66.53087117633495 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.GreaterThanOrEqualAnyBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.01842510142153889 < 16.68780206471084.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 269.2958510206996 (T) = (0 -0.04804386793891949) / Math.Sqrt((1.0908362529282378 / (299)) + (0.00813349738924498 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9974842841514355 = (19.097493847062974 - 0.04804386793891949) / 19.097493847062974 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AndNotBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1322844179415192 < 12.800946063383801.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 260.5925427745718 (T) = (0 -1.0937381296032478) / Math.Sqrt((0.39604284862210853 / (299)) + (0.004541984909976479 / (4))) is greater than 1.9678765312856872 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (4) - 2, .975) and 0.9219808975323849 = (14.018850448289204 - 1.0937381296032478) / 14.018850448289204 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.EqualsAnyBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.012085050809949812 < 16.113008196055574.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 272.01424823543283 (T) = (0 -0.05725467574118801) / Math.Sqrt((1.140713175148538 / (299)) + (0.00516558552419191 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9969313585717198 = (18.657988259409084 - 0.05725467574118801) / 18.657988259409084 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 7a57f6dd800e77508a4ba405e623059935c82f43
Compare 887c043eb94be364188e2b23a87efa214ea57f1e
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LessThanOrEqualBenchmark - Duration of single invocation 13.12 ns 1.19 ns 0.09 0.08 True
GreaterThanOrEqualAnyBenchmark - Duration of single invocation 3.42 ns 0.00 ns 0.00 0.24 False
GreaterThanOrEqualBenchmark - Duration of single invocation 13.72 ns 1.22 ns 0.09 0.07 True
LessThanAllBenchmark - Duration of single invocation 3.03 ns 0.00 ns 0.00 0.41 False
MaxBenchmark - Duration of single invocation 14.21 ns 1.12 ns 0.08 0.05 True
LessThanOrEqualAnyBenchmark - Duration of single invocation 4.95 ns 0.58 ns 0.12 0.29 False
LessThanOrEqualAllBenchmark - Duration of single invocation 3.28 ns 0.00 ns 0.00 0.19 False
EqualsBenchmark - Duration of single invocation 5.86 ns 0.82 ns 0.14 0.17 False
LessThanAnyBenchmark - Duration of single invocation 4.29 ns 0.08 ns 0.02 0.32 False
GreaterThanOrEqualAllBenchmark - Duration of single invocation 4.89 ns 0.15 ns 0.03 0.28 False
LessThanBenchmark - Duration of single invocation 12.92 ns 1.34 ns 0.10 0.07 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.LessThanOrEqualBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1921295231129616 < 12.579529756137198.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 199.5727597700924 (T) = (0 -1.0853466122521669) / Math.Sqrt((1.122996683000519 / (299)) + (0.004994300033061019 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.925679997781348 = (14.603694562051267 - 1.0853466122521669) / 14.603694562051267 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.GreaterThanOrEqualAnyBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 3.310756328052121.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 75.63258042293425 (T) = (0 -0.011218355116822485) / Math.Sqrt((1.6504707923268571 / (299)) + (0.0002692290785766892 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.9980171858490557 = (5.657794560060046 - 0.011218355116822485) / 5.657794560060046 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.GreaterThanOrEqualBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.224844762929218 < 13.142185903945364.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 187.86536353909662 (T) = (0 -1.214408109309215) / Math.Sqrt((1.202713086114938 / (299)) + (0.009271019680668163 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9202739565141932 = (15.23226358931771 - 1.214408109309215) / 15.23226358931771 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.LessThanAllBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 2.8719232021751373.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 69.10114428203778 (T) = (0 -0) / Math.Sqrt((1.656382492020901 / (299)) + (0 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 1 = (5.143159118210431 - 0) / 5.143159118210431 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.MaxBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.11664531121939 < 13.552196702568395.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 279.4007917480251 (T) = (0 -1.109228356994248) / Math.Sqrt((0.4967317716922627 / (299)) + (0.005203517557524402 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9268263377638659 = (15.15884709193216 - 1.109228356994248) / 15.15884709193216 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.LessThanOrEqualAnyBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.5797852117099194 < 4.66949913870711.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 57.03017464295864 (T) = (0 -0.1061880543070784) / Math.Sqrt((2.062042019675009 / (299)) + (0.053944462118759785 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.985442774236246 = (7.294525483796252 - 0.1061880543070784) / 7.294525483796252 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.LessThanOrEqualAllBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 3.2421105156754852.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 76.30117321356546 (T) = (0 -0.004689616826882656) / Math.Sqrt((1.6574256383454449 / (299)) + (6.960045471266301E-05 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9991760273860689 = (5.691471715921829 - 0.004689616826882656) / 5.691471715921829 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.EqualsBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.8224492670161839 < 5.629990974533576.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 57.844905567635536 (T) = (0 -0.8541599936885236) / Math.Sqrt((4.796556706481057 / (299)) + (0.028774489233355717 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9071992580252787 = (9.204236685103165 - 0.8541599936885236) / 9.204236685103165 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.LessThanAnyBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.07742400896756549 < 4.151242950033223.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/22/2023 9:02:35 AM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 80.43457815607918 (T) = (0 -0.012904001494594248) / Math.Sqrt((2.0046933409980645 / (299)) + (0.0009990795274349433 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9980683588258673 = (6.680330522768292 - 0.012904001494594248) / 6.680330522768292 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.GreaterThanOrEqualAllBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.15043864256336736 < 4.699600066627464.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 85.2884584272848 (T) = (0 -0.02919505692288053) / Math.Sqrt((2.034866072616266 / (299)) + (0.0036258668036127178 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9960391373609127 = (7.370883462297354 - 0.02919505692288053) / 7.370883462297354 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.LessThanBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3391025014232116 < 12.086815698722802.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/12/2023 6:21:10 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 9:22:58 PM and 4/18/2023 4:36:42 PM.
IsImprovementStdDev: Marked as improvement because 142.59473351739845 (T) = (0 -1.1279969689786298) / Math.Sqrt((1.3113365448090784 / (299)) + (0.02375026117590926 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.9203024769663438 = (14.153475867779207 - 1.1279969689786298) / 14.153475867779207 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions