Unexpectedly high latency of string utility functions on Linux (about 3 times higher than the same function on windows)

[dmeytin]

Issue Title

Unexpected latency of string utility functions on Linux

General

I have used https://github.com/dotnet/BenchmarkDotNet to test String.IndexOf on Windows and on Linux and the results below reveal almost 3 times latency difference for the function invocation on the random string of sizes 200 ... 1000, small constant pattern of the size 4 and OrdinalIgnoreCase culture

Latency of 1K executions are as following:
Linux: Avg : 3.151195819999976ms; Mean: 3.0258999999999796ms; Percentile_95: 3.8338250000000005ms
Windows: Avg : 1.136806029999999ms; Mean: 1.0643499999999961ms; Percentile_95: 1.5009250000000038ms

Here is the benchmark code:

public class StringUtils
    {
        string randomString;
        string toSearch = " -p ";
        char[] pattern;

        [Params(200, 300, 400, 500, 600, 700, 800, 900, 1000)]
        public int N;

        public StringUtils()
        {
            pattern = toSearch.ToLower().ToCharArray();
        }

        [IterationSetup]
        public void BeforeEachStep()
        {
            randomString = GenerateRandomString(N);
        }
        
        [Benchmark]
        public void IndexOf()
        {
            randomString.IndexOf(toSearch, StringComparison.OrdinalIgnoreCase);
        } 

        private string GenerateRandomString(int length)
        {
            Random random = new Random();
            const string chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 -abcdefghijklmnopqrstuvwxyz!#@";
            return new string(Enumerable.Repeat(chars, length)
              .Select(s => s[random.Next(s.Length)]).ToArray());
        }
    }

Although both VMs are created with the same Azure SKU (Standard_D2s_v3), the actual hardware is a bit different, but not significantly.

BenchmarkDotNet=v0.12.1, OS=ubuntu 18.04
Intel Xeon CPU E5-2673 v4 2.30GHz, 1 CPU, 2 logical cores and 1 physical core
.NET Core SDK=3.1.301
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
Job-YVSZZG : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.14393.3750 (1607/AnniversaryUpdate/Redstone1)
Intel Xeon Platinum 8171M CPU 2.60GHz, 1 CPU, 2 logical cores and 1 physical core
.NET Core SDK=3.1.301
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT
Job-BSDYRT : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64 RyuJIT

InvocationCount=1 UnrollFactor=1

And detailed results are as following:

OS	Method	N	Mean	Error	StdDev	Median
Linux	IndexOf	200	5.329 us	0.1135 us	0.3145 us	5.300 us
Windows	IndexOf	200	3.159 us	0.1491 us	0.4303 us	3.200 us
Linux	IndexOf	300	7.700 us	0.1570 us	0.1927 us	7.700 us
Windows	IndexOf	300	4.096 us	0.1676 us	0.4915 us	4.100 us
Linux	IndexOf	400	8.800 us	0.1714 us	0.1519 us	8.800 us
Windows	IndexOf	400	4.886 us	0.2148 us	0.6334 us	5.050 us
Linux	IndexOf	500	11.604 us	0.3069 us	0.8192 us	11.500 us
Windows	IndexOf	500	5.679 us	0.2132 us	0.6220 us	5.650 us
Linux	IndexOf	600	12.933 us	0.2662 us	0.7107 us	12.601 us
Windows	IndexOf	600	6.604 us	0.2407 us	0.7058 us	6.700 us
Linux	IndexOf	700	14.443 us	0.2824 us	0.2504 us	14.450 us
Windows	IndexOf	700	7.204 us	0.1426 us	0.2045 us	7.100 us
Linux	IndexOf	800	17.629 us	0.3564 us	0.9575 us	17.500 us
Windows	IndexOf	800	8.074 us	0.3290 us	0.9385 us	8.200 us
Linux	IndexOf	900	19.018 us	0.5516 us	1.5194 us	18.700 us
Windows	IndexOf	900	8.852 us	0.3884 us	1.1328 us	9.050 us
Linux	IndexOf	1000	20.609 us	0.4151 us	0.8194 us	20.350 us
Windows	IndexOf	1000	9.607 us	0.3468 us	1.0060 us	9.700 us

#Expected results
Linux latency should be at least as good as Windows

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

Tagging subscribers to this area: @tarekgh, @safern, @krwq
Notify danmosemsft if you want to be subscribed.

[GrabYourPitchforks]

This is almost certainly an ICU / NLS behavior. ICU has a different philosophy for IndexOf than does NLS. For ICU, the typical calling pattern is that you create a pattern object which represents the target substring you want to locate within the search space, and you invoke the pattern matcher over and over again to keep finding instances of the target substring. ICU's tradeoff is that the initial setup is a little expensive, but subsequent checks are fast assuming you use the same pattern object. On the managed side, since each call to IndexOf necessarily creates a new pattern matcher, we're paying for this overhead on every method invocation.

Related: #37815, which proposes that we special-case IndexOf(..., OrdinalIgnoreCase) for ASCII inputs to avoid calling into ICU / NLS at all.

[dmeytin]

@GrabYourPitchforks, thanks for your response.
Do we have any possible workaround to improve the performance?

[tarekgh]

This is properly duplicate of #26054 and #37819

[tarekgh]

Do we have any possible workaround to improve the performance?

The best would be to avoid using ignore casing for now.

[weltkante]

The best would be to avoid using ignore casing for now.

So you propose workarounds like comparing ToUpper/ToLower normalized strings?