Investigate regressions of binary-trees and fannkuch on benchmarksgame site

[danmoseley]

Benchmarks game site has a comparison up of 3.1 vs . 5.0 preview 7 : https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/csharpcore-csharppreview.html

There are 4 regressions, 2 may be noise level. The other two are binary-trees (7%) and fannkuch-redux (4%)

I'll check those two using the source of the latest submissions.

[danmoseley]

Here's the results i get on Ubuntu using the latest sources for these two games (BinaryTrees_6 and FannkuchRedux_9 from dotnet/performance#1453 but with the official input parameters)

I included the various other flavors we already have, which are older.

There is no evidence of a regression here for the "official" sources, so it's unclear why they're seeing one unless its from slightly different hardware or builds. I guess we'll see whether it goes away in final official results.
cc @AndyAyersMS

BenchmarkDotNet=v0.12.1.1405-nightly, OS=ubuntu 18.04
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-preview.7.20366.6
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-HFLFME : .NET Core 2.1.21 (CoreCLR 4.6.29130.01, CoreFX 4.6.29130.02), X64 RyuJIT
  Job-XLBCVM : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-HEKPDG : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT
  Job-YYWTGM : .NET Core 2.1.21 (CoreCLR 4.6.29130.01, CoreFX 4.6.29130.02), X64 RyuJIT
  Job-DISMCE : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-HDJRLZ : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  IterationTime=250.0000 ms  
MinIterationCount=15  WarmupCount=1  

Type	Method	Job	Runtime	Toolchain	MaxIterationCount	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
BinaryTrees_2	BinaryTrees_2	Job-HFLFME	.NET Core 2.1	netcoreapp2.1	20	123.4 ms	1.74 ms	1.71 ms	122.9 ms	120.9 ms	127.8 ms	1.00	0.00	53000.0000	1500.0000	500.0000	232789.44 KB
BinaryTrees_2	BinaryTrees_2	Job-XLBCVM	.NET Core 3.1	netcoreapp3.1	20	150.3 ms	2.72 ms	2.54 ms	150.6 ms	145.7 ms	154.1 ms	1.22	0.03	55000.0000	1000.0000	-	232789.44 KB
BinaryTrees_2	BinaryTrees_2	Job-HEKPDG	.NET Core 5.0	netcoreapp5.0	20	177.2 ms	6.09 ms	7.01 ms	177.7 ms	162.5 ms	189.3 ms	1.43	0.06	50000.0000	1000.0000	-	232789.72 KB
BinaryTrees_5	BinaryTrees_5	Job-YYWTGM	.NET Core 2.1	netcoreapp2.1	40	140.0 ms	2.77 ms	3.88 ms	140.2 ms	130.7 ms	145.6 ms	1.00	0.00	39000.0000	12500.0000	2500.0000	1.7 KB
BinaryTrees_5	BinaryTrees_5	Job-DISMCE	.NET Core 3.1	netcoreapp3.1	40	145.3 ms	2.89 ms	4.41 ms	145.7 ms	133.5 ms	152.2 ms	1.04	0.04	38000.0000	11000.0000	4000.0000	232799.7 KB
BinaryTrees_5	BinaryTrees_5	Job-HDJRLZ	.NET Core 5.0	netcoreapp5.0	40	141.7 ms	2.82 ms	5.01 ms	141.6 ms	132.0 ms	154.0 ms	1.01	0.04	38000.0000	9000.0000	3000.0000	232793.72 KB
BinaryTrees_6	BinaryTrees_6	Job-YYWTGM	.NET Core 2.1	netcoreapp2.1	40	8,559.7 ms	164.30 ms	153.69 ms	8,558.8 ms	8,295.0 ms	8,832.5 ms	1.00	0.00	1942000.0000	359000.0000	9000.0000	65551.55 KB
BinaryTrees_6	BinaryTrees_6	Job-DISMCE	.NET Core 3.1	netcoreapp3.1	40	8,494.8 ms	154.52 ms	158.68 ms	8,498.8 ms	8,143.3 ms	8,747.1 ms	0.99	0.02	1940000.0000	370000.0000	9000.0000	9546425.66 KB
BinaryTrees_6	BinaryTrees_6	Job-HDJRLZ	.NET Core 5.0	netcoreapp5.0	40	7,929.7 ms	158.14 ms	147.92 ms	7,900.6 ms	7,603.8 ms	8,176.0 ms	0.93	0.03	1947000.0000	367000.0000	9000.0000	9546422.11 KB

BenchmarkDotNet=v0.12.1.1405-nightly, OS=ubuntu 18.04
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-preview.7.20366.6
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-YVYDTA : .NET Core 2.1.21 (CoreCLR 4.6.29130.01, CoreFX 4.6.29130.02), X64 RyuJIT
  Job-DOYOOY : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-ZKDSPK : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  IterationTime=250.0000 ms  
MaxIterationCount=20  MinIterationCount=15  WarmupCount=1  

Type	Method	Job	Runtime	Toolchain	n	expectedSum	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
FannkuchRedux_5	FannkuchRedux_5	Job-YVYDTA	.NET Core 2.1	netcoreapp2.1	10	38	37.02 ms	1.385 ms	1.595 ms	36.31 ms	35.61 ms	40.08 ms	1.00	0.00	-	-	-	3120 B
FannkuchRedux_5	FannkuchRedux_5	Job-DOYOOY	.NET Core 3.1	netcoreapp3.1	10	38	37.87 ms	0.632 ms	0.592 ms	37.70 ms	36.97 ms	39.07 ms	1.03	0.04	-	-	-	4632 B
FannkuchRedux_5	FannkuchRedux_5	Job-ZKDSPK	.NET Core 5.0	netcoreapp5.0	10	38	36.98 ms	0.698 ms	0.619 ms	36.92 ms	36.30 ms	38.26 ms	1.01	0.04	-	-	-	4673 B
FannkuchRedux_2	FannkuchRedux_2	Job-YVYDTA	.NET Core 2.1	netcoreapp2.1	10	73196	150.94 ms	1.181 ms	0.986 ms	150.94 ms	149.68 ms	152.25 ms	1.00	0.00	-	-	-	224 B
FannkuchRedux_2	FannkuchRedux_2	Job-DOYOOY	.NET Core 3.1	netcoreapp3.1	10	73196	143.72 ms	1.420 ms	1.108 ms	143.52 ms	142.36 ms	146.02 ms	0.95	0.01	-	-	-	224 B
FannkuchRedux_2	FannkuchRedux_2	Job-ZKDSPK	.NET Core 5.0	netcoreapp5.0	10	73196	153.09 ms	1.336 ms	1.184 ms	152.98 ms	151.65 ms	155.17 ms	1.01	0.01	-	-	-	1168 B
FannkuchRedux_9	FannkuchRedux_5	Job-YVYDTA	.NET Core 2.1	netcoreapp2.1	12	3968050	4,430.65 ms	7.464 ms	6.233 ms	4,431.80 ms	4,421.10 ms	4,439.71 ms	1.00	0.00	-	-	-	2368 B
FannkuchRedux_9	FannkuchRedux_5	Job-DOYOOY	.NET Core 3.1	netcoreapp3.1	12	3968050	4,702.96 ms	62.185 ms	55.126 ms	4,679.59 ms	4,655.59 ms	4,827.31 ms	1.06	0.01	-	-	-	2536 B
FannkuchRedux_9	FannkuchRedux_5	Job-ZKDSPK	.NET Core 5.0	netcoreapp5.0	12	3968050	4,411.65 ms	48.697 ms	45.551 ms	4,389.69 ms	4,373.25 ms	4,502.68 ms	1.00	0.01	-	-	-	2824 B

[danmoseley]

cc @benaadams

[billwert]

If this is to track a potential regression in the product should we move it to dotnet/runtime @danmosemsft?

[adamsitnik]

FWIW the regression is Windows-specific

BenchmarksGame.BinaryTrees_2.RunBench

Conclusion	Base	Diff	Base/Diff	Modality	Operating System	Bit	Processor Name	Base Runtime	Diff Runtime
Slower	104517050.00	115141650.00	0.91		Windows 10.0.18363.959	X64	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 3.1.6	.NET Core 5.0.0
Same	121320538.00	126695979.00	0.96		ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 3.1.6	.NET Core 5.0.0
Same	189125801.00	185862914.00	1.02		macOS Mojave 10.14.5	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	.NET Core 3.1.6	.NET Core 5.0.0

BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)

Conclusion	Base	Diff	Base/Diff	Modality	Operating System	Bit	Processor Name	Base Runtime	Diff Runtime
Slower	24379150.00	39701257.14	0.61	bimodal	Windows 10.0.18363.959	X64	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 3.1.6	.NET Core 5.0.0
Same	27439371.19	27543953.25	1.00		ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 3.1.6	.NET Core 5.0.0
Same	82078762.25	87040901.88	0.94		macOS Mojave 10.14.5	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	.NET Core 3.1.6	.NET Core 5.0.0

[danmoseley]

They measure on Ubuntu so if there's a windows regression that's coincidental 🙂

[AndyAyersMS]

Benchmarks games now uses a slightly more up to date Ivy Bridge cpu.

Measured on a quad-core 3.0GHz Intel® i5-3330® with 15.8 GiB of RAM and 2TB SATA disk drive; using Ubuntu™ 20.04 x86_64 GNU/Linux 5.4.0-40-generic.

[AndyAyersMS]

I'll investigate but may not get around to it until next week sometime. So if anyone wants to drill in the meantime, please go ahead.

cc @dotnet/jit-contrib