[Perf - 10-20x regression] System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse in ja

[DrewScoggins]

Run Information

Architecture	x64
OS	Windows 10.0.18362
Changes	diff

Regressions in System.Globalization.Tests.Perf_DateTimeCultureInfo

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
Parse	518.38 ns	5.52 μs	10.65		False

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Globalization.Tests.Perf_DateTimeCultureInfo*';

Histogram

System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: ja)

[-265.889 ; 1287.875) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[1287.875 ; 2817.846) | 
[2817.846 ; 4347.817) | 
[4347.817 ; 6269.985) | @@@@@@@@@@@@@@@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Tagging subscribers to this area: @tarekgh, @safern, @krwq
Notify danmosemsft if you want to be subscribed.

[DrewScoggins]

Also likely ICU related @adamsitnik

[safern]

Yeah I think this is definitely related to the ICU change on Windows.

Might be interesting to run the benchmarks with environment variable: DOTNET_SYSTEM_GLOBALIZATION_USENLS=1 -- to see if that also affects the performance.

[danmoseley]

Without env var, the huge regression for ja is clear - it's 1887% (20x) regressed:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.421 (2004/?/20H1)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-rc.1.20401.3
[Host] : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT
Job-FVRXJJ : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000 Runtime=.NET Core 3.1 Arguments=/p:DebugType=portable
Toolchain=netcoreapp3.1 IterationTime=250.0000 ms MaxIterationCount=20
MinIterationCount=15 WarmupCount=1

Method	culturestring	Mean	Error	StdDev	Median	Min	Max	Gen 0	Gen 1	Gen 2	Allocated
Parse		440.9 ns	8.69 ns	8.93 ns	442.1 ns	426.0 ns	455.3 ns	-	-	-	-
Parse	da	529.6 ns	10.75 ns	12.38 ns	529.8 ns	507.5 ns	551.5 ns	-	-	-	-
Parse	fr	442.0 ns	6.01 ns	5.33 ns	441.9 ns	434.5 ns	451.2 ns	-	-	-	-
Parse	ja	455.6 ns	8.24 ns	7.31 ns	458.6 ns	443.9 ns	465.7 ns	-	-	-	-

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.421 (2004/?/20H1)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-rc.1.20401.3
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.40101, CoreFX 5.0.20.40101), X64 RyuJIT
Job-ZUZTSW : .NET Core 5.0.0 (CoreCLR 5.0.20.40101, CoreFX 5.0.20.40101), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000 Runtime=.NET Core 5.0 Arguments=/p:DebugType=portable
Toolchain=netcoreapp5.0 IterationTime=250.0000 ms MaxIterationCount=20
MinIterationCount=15 WarmupCount=1

Method	culturestring	Mean	Error	StdDev	Median	Min	Max	Gen 0	Gen 1	Gen 2	Allocated
Parse		399.1 ns	7.67 ns	6.41 ns	399.3 ns	390.7 ns	414.3 ns	-	-	-	-
Parse	da	425.6 ns	8.17 ns	7.64 ns	424.3 ns	415.2 ns	436.6 ns	-	-	-	-
Parse	fr	403.4 ns	10.60 ns	12.20 ns	402.6 ns	386.0 ns	433.3 ns	-	-	-	-
Parse	ja	8,588.9 ns	3,503.94 ns	3,749.18 ns	6,850.0 ns	4,300.0 ns	16,000.0 ns	-	-	-	-

But with DOTNET_SYSTEM_GLOBALIZATION_USENLS=1 the regression vanishes.
C:\git>py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter "System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse"

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.421 (2004/?/20H1)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-rc.1.20401.3
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.40101, CoreFX 5.0.20.40101), X64 RyuJIT
Job-GXOISF : .NET Core 5.0.0 (CoreCLR 5.0.20.40101, CoreFX 5.0.20.40101), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000 Runtime=.NET Core 5.0 Arguments=/p:DebugType=portable
Toolchain=netcoreapp5.0 IterationTime=250.0000 ms MaxIterationCount=20
MinIterationCount=15 WarmupCount=1

Method	culturestring	Mean	Error	StdDev	Median	Min	Max	Gen 0	Gen 1	Gen 2	Allocated
Parse		402.2 ns	3.94 ns	3.69 ns	400.9 ns	397.2 ns	409.2 ns	-	-	-	-
Parse	da	481.0 ns	2.70 ns	2.53 ns	480.8 ns	477.6 ns	485.7 ns	-	-	-	-
Parse	fr	406.7 ns	3.93 ns	3.49 ns	406.4 ns	400.8 ns	412.5 ns	-	-	-	-
Parse	ja	412.6 ns	4.55 ns	4.03 ns	413.1 ns	403.4 ns	419.3 ns	-	-	-	-

This regression would be painful to ship with.