Benchmark results analysis for .NET Core 2.1 vs 3.0

[adamsitnik]

I took all of our benchmarks and run them for:

1. .NET Core 2.1 using native CpuMathNative which uses SSE
2. .NET Core 3.0 using new managed Hardware Intrinsics API which uses AVX with Tiered compilation enabled
3. .NET Core 3.0 using new managed Hardware Intrinsics API which uses AVX with Tiered compilation disabled

I will add a separate comment for every benchmark in this issue with some analysis.

Environment info (updated):

BenchmarkDotNet=v0.11.1.786-nightly, OS=Windows 10.0.17134.285 (1803/April2018Update/Redstone4)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
Frequency=3507500 Hz, Resolution=285.1033 ns, Timer=TSC
.NET Core SDK=2.2.100-preview2-009404
  [Host]             : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
  Core 2.1           : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
  Core 3.0 NonTiered : .NET Core 3.0.0-preview1-27004-04 (CoreCLR 4.6.27003.04, CoreFX 4.6.27003.02), 64bit RyuJIT

[adamsitnik]

KMeansAndLogisticRegressionBench.TrainKMeansAndLR

Job	Toolchain	Mean	StdDev	Scaled
Core 2.1	netcoreapp2.1	496.564 ms	70.6073 ms	1.00
Core 3.0 NonTiered	netcoreapp3.0	595.176 ms	112.4640 ms	1.22
Core 3.0 Tiered	netcoreapp3.0	597.032 ms	106.1578 ms	1.22

The profile shows that AddScaleSU which should be faster with AVX is actually slower and takes much more time. This can be confirmed running our CpuMathBenchmarks:

..\..\Tools\dotnetcli\dotnet.exe run -c Release-Intrinsics -f netcoreapp3.0 -- -F *.AddScaleSU --join -d

Type	Method	Mean
AvxPerformanceTests	AddScaleSU	4.012 ms
NativePerformanceTests	AddScaleSU	2.966 ms
SsePerformanceTests	AddScaleSU	2.916 ms

This issue has been spotted by @eerhardt in August #691 (comment)

I reported issues #1195 and #1196 which got improved in #1200 but still exist.

Edit: This benchmark is multimodal, I believe that it's very dependent on the alignment.

Summary: 20% slower due to invalid AddScaleSU AVX implementation.

[adamsitnik]

MultiClassClassificationTest.Test_Multiclass_WikiDetox_BigramsAndTrichar_OVAAveragedPerceptron

Job	Mean	Error	StdDev	Median	Scaled
Core 2.1	4,889.920 ms	73.2056 ms	68.4765 ms	4,882.153 ms	1.00
Core 3.0 NonTiered	5,258.516 ms	99.5758 ms	93.1432 ms	5,253.034 ms	1.08
Core 3.0 Tiered	5,284.164 ms	73.6200 ms	68.8642 ms	5,273.463 ms	1.08

The PerfView diff for time to execute the benchmark once (the positive values mean more time spent in given method):

Name	Exc %	Exc	Inc %	Inc
module microsoft.ml.data <<microsoft.ml.data!SequencePool.GetCore>>	28,1	995	43,9	1 555
module coreclr <<coreclr!Thread::intermediateThreadProc>>	-22,8	-809	98,5	3 489
LIB <<system.threading.tasks.parallel.il!Parallel.For>>	13,4	475	53,4	1 889
OS <<kernelbase!SleepEx>>	13,2	466	13,2	466
module ? <>	12,8	452,319	12,7	451,319
LIB <<system.threading.tasks.parallel!Parallel.For>>	-11,5	-406	-46,3	-1 640
LIB <<system.private.corelib!System.Collections.Generic.Dictionary`2[System.Char,System.Char].FindEntry(!0)>>	10,8	382	10,9	386
module microsoft.ml.cpumath <<microsoft.ml.cpumath!CpuMathUtils.DotProductSparse>>	10,8	382	11,0	388
module microsoft.ml.core <<microsoft.ml.core!ReadOnlyMemoryUtils.FindInPool>>	9,0	319	11,5	408
LIB <<system.private.corelib!Array.Sort>>	8,7	308	8,1	288
module microsoft.ml.core <<microsoft.ml.core!Hashing.MurmurHash>>	8,4	296	15,1	533
module cpumathnative <<cpumathnative!DotSU>>	-8,3	-294	-8,4	-296
module microsoft.ml.transforms <<microsoft.ml.transforms!Microsoft.ML.Runtime.Data.LpNormNormalizerTransform+<>c__DisplayClass30_0.b__5(value class Microsoft.ML.Runtime.Data.VBuffer`1&)>>	8,2	292	91,9	3 253
module coreclr <<coreclr!ThreadNative::SpinWait>>	-4,6	-163	-4,6	-163
module clrcompression <<clrcompression!inflate>>	-4,5	-160	-4,5	-160
module clrcompression <<clrcompression!CompressionNative_Inflate>>	4,4	156	4,4	156
module coreclr <<coreclr!JIT_GetSharedNonGCStaticBase_SingleAppDomain>>	-4,0	-141	-4,0	-142
OS <<kernelbase!GetQueuedCompletionStatus>>	4,0	141	4,0	141
OS <<kernelbase!PostQueuedCompletionStatus>>	2,6	93	2,6	93
module microsoft.ml.core <<microsoft.ml.core!Microsoft.ML.Runtime.Internal.Utilities.FixedSizeQueue`1[System.UInt32].AddLast(!0)>>	2,3	82	2,4	84
module coreclr <<coreclr!JIT_NewArr1>>	2,1	76	2,3	81
OS <<kernelbase!SwitchToThread>>	2,0	72	2,0	72
module microsoft.ml.core <<microsoft.ml.core!ReadOnlyMemoryUtils.SplitOne>>	2,0	70	3,1	108

Summary: 8% slower. Top method from the diff - SequencePool.GetCore was using same IL, but different runtimes.

TODO: investigate if SequencePool.GetCore produces different asm for 3.0. The method is really hot on the profiles so any perf improvement will be great

[adamsitnik]

PredictionEngineBench.MakeIrisPredictions

Job	Mean	StdDev	Scaled
Core 2.1	2.838 ms	0.0019 ms	1.00
Core 3.0 NonTiered	3.433 ms	0.0026 ms	1.21
Core 3.0 Tiered	3.452 ms	0.0017 ms	1.22

Summary: 21% slower. PredictCore calling DotProductDense can't be inlined. The DotU is faster for AVX but we need to make sure we don't pay any extra cost for the call.

The issue allready exists #833

[helloguo]

I'm curious if you saw the run to run variance when running TrainKMeansAndLR benchmark? I run it on my machine 10 times (.net core 3 and Tiered JIT enabled) and the perf number ranges from 466 ms to 533 ms.

[adamsitnik]

@helloguo the TrainKMeansAndLR is multimodal, I believe that it's very dependent on the alignment (/cc @tannergooding )

[tannergooding]

Yes, the current implementation does not do anything to deal with misaligned inputs and will incur a penalty anytime a read crosses a cache-line or page boundary. @Anipik is working on improving some of these algorithms using the template I provided (which will incur at most 2 unaligned reads for the input data).

[adamsitnik]

Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_FastTreeRanking

Job	Mean	StdDev	Scaled
Core 2.1	30.48 s	0.4394 s	1.00
Core 3.0 NonTiered	30.36 s	0.6783 s	1.00

no significant difference

[adamsitnik]

Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_LightGBMRanking

Job	Mean	StdDev	Scaled
Core 2.1	28.06 s	0.2593 s	1.00
Core 3.0 NonTiered	28.54 s	0.7388 s	1.02

no significant difference (the 2% here is noise)

[Anipik]

Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_LightGBMRanking
Microsoft.ML.Benchmarks.RankingTrain.TrainTest_Ranking_MSLRWeb10K_RawNumericFeatures_FastTreeRanking

These results are expected for these tests because they don't use anything from CpuMath library

[adamsitnik]

These results are expected for these tests because they don't use anything from CpuMath library

good to know (I am still getting familiar with ML.NET), I also wanted to check if ML.NET can get affected by tiered jitting

[danmoseley]

It seems next step here is to re-measure after @Anipik is done.

[danmoseley]

Thank you for doing this verification @adamsitnik