[Perf -35%] System.Memory.Span<Char>.IndexOfValue

[DrewScoggins]

Run Information

Architecture	x64
OS	ubuntu 18.04
Changes	diff

Regressions in System.Memory.Span

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
IndexOfValue	11.56 ns	15.57 ns	1.35	Bimodal	False

Related Issue on x64 Windows

[Perf -14%] System.Memory.Span.IndexOfValue

Related Issue on x86 Windows

[Perf -11%] System.Memory.Span (2)

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Memory.Span<Char>*';

Histogram

System.Memory.Span.IndexOfValue(Size: 512)

[10.151 ; 10.937) | @@@@@
[10.937 ; 11.666) | @@@@@@@@@@@@
[11.666 ; 12.565) | @@@@@@@@@@@@@@@@@@@@@@@@@@@
[12.565 ; 13.174) | @@@
[13.174 ; 14.111) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[14.111 ; 14.945) | @@@@@@@@@
[14.945 ; 15.674) | @@@@@@@@@@@@@@@@@@@@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[DrewScoggins]

This one is a bit noisy, but @AndyAyersMS and @kunalspathak wanted to take a look at it.

[kunalspathak]

I compared JIT output of baseline vs. test and didn't find anything noticeably different:

• I saw some diffs in Chronometer:.cctor(). This probably calls StopWatch.
• LocalAppContextSwitches:GetSwitchDefaultValue: Introduce opt-in BinaryFormatter killbit #38963
• get_ProcessorArchitecture - Changes in don't always aquite lock when calling RuntimeInformation.OSArchitecture and OSArchitecture.ProcessArchitecture #39196 and add OSPlatform.macOS, switch to OrdinalIgnoreCase for OSPlatform comparisons #39209. These look safe and I don't think it would even impact the benchmark code.
• OSPlatform:.cctor() - Changes in add OSPlatform.macOS, switch to OrdinalIgnoreCase for OSPlatform comparisons #39209 and Add OSPlatform entries for iOS/tvOS/watchOS/Android #36704 but again, don't think it would impact benchmark.

I also saw that we are eliminating redundant test instruction done in #38586.

Below is the baseline and test output I produced by locally building clrjit.dll for baseline and test and running benchmarks with below environment variables:

COMPlus_JitDiffableDasm=1
COMPlus_JitDisasm=*
COMPlus_TieredCompilation=0

indexofvalue.test.txt

indexofvalue.baseline.txt

Update:

• There are also changes around Guid:TryParseHex() made in Add methods to convert between hexadecimal strings and bytes #37546, but I don't see how it will be related to the benchmark.

[danmoseley]

@DrewScoggins thanks for making sure the diff link is against the runtime repo. However this still produces a unicorn as it covers July 14-July 21. From eyeballing the graph, shouldn't the SHA's be about July 14-July 15?

(I realize that viewing even the tightly bracketed diff in Github is often not going to be feasible, but having well bracketed SHA's to examine offline is certainly important.)

[DrewScoggins]

Yes, getting a tight bracket is very important, and it is something that we are thinking about how to solve in a consistent, accurate, and automated fashion. In the meantime, we believed that having a link that can at least bracket to the time of the report, in this case a week, was still useful as a starting point. For what it is worth, when I clicked the link I was able to get the comparison and didn't see an error, although I have had that happen plenty of times.

For now what we will do is when a bug is copied over we will use the tools in the generated report to try and tighten the window as best we can for the diff link. It might not be perfect, but hopefully will be a start.

[danmoseley]

@DrewScoggins makes sense. Sounds good. I hope it didn't appear as a criticism. I see this as an iterative process to make these as useful as possible.

[adamsitnik]

I was able to reproduce the regression. Link to asm diff https://www.diffchecker.com/daxuXSFS full details below:

BenchmarkDotNet=v0.12.1.1405-nightly, OS=Windows 10.0.18363.959 (1909/November2019Update/19H2)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-rc.1.20413.9
  [Host]     : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT
  Job-RHSWLL : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Runtime=.NET Core 3.1  Arguments=/p:DebugType=portable  
Toolchain=netcoreapp3.1  IterationTime=250.0000 ms  MaxIterationCount=20  
MinIterationCount=15  WarmupCount=1  

Method	Size	Mean	Error	StdDev	Median	Min	Max	Gen 0	Gen 1	Gen 2	Allocated	Code Size
IndexOfValue	512	13.70 ns	0.044 ns	0.039 ns	13.68 ns	13.65 ns	13.77 ns	-	-	-	-	518 B

.NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT

; System.Memory.Span`1[[System.Char, System.Private.CoreLib]].IndexOfValue()
       sub       rsp,28
       mov       rdx,[rcx+18]
       test      rdx,rdx
       jne       short M00_L00
       xor       r8d,r8d
       xor       eax,eax
       jmp       short M00_L01
M00_L00:
       lea       r8,[rdx+10]
       mov       eax,[rdx+8]
M00_L01:
       movzx     ecx,word ptr [rcx+2C]
       mov       [rsp+24],ecx
       mov       edx,[rsp+24]
       mov       rcx,r8
       mov       r8d,eax
       call      System.SpanHelpers.IndexOf(Char ByRef, Char, Int32)
       nop
       add       rsp,28
       ret
; Total bytes of code 56

; System.SpanHelpers.IndexOf(Char ByRef, Char, Int32)
       push      rsi
       vzeroupper
       xor       r9d,r9d
       movsxd    r10,r8d
       mov       rax,r10
       mov       r11,rcx
       test      r11b,1
       jne       short M01_L01
       cmp       r8d,10
       jl        short M01_L01
       mov       rax,rcx
       neg       eax
       mov       r8d,eax
       shr       r8d,1F
       add       eax,r8d
       sar       eax,1
       mov       eax,eax
       and       rax,7
       cmp       rax,4
       jl        short M01_L02
M01_L00:
       lea       r8,[rcx+r9*2]
       movzx     r11d,dx
       movzx     esi,word ptr [r8]
       cmp       r11d,esi
       je        near ptr M01_L16
       movzx     esi,word ptr [r8+2]
       cmp       r11d,esi
       je        near ptr M01_L15
       movzx     esi,word ptr [r8+4]
       cmp       r11d,esi
       je        near ptr M01_L14
       movzx     r8d,word ptr [r8+6]
       cmp       r11d,r8d
       je        near ptr M01_L13
       add       r9,4
       add       rax,0FFFFFFFFFFFFFFFC
M01_L01:
       cmp       rax,4
       jge       short M01_L00
M01_L02:
       test      rax,rax
       jle       short M01_L04
       movzx     r11d,dx
M01_L03:
       movzx     r8d,word ptr [rcx+r9*2]
       cmp       r8d,r11d
       je        near ptr M01_L16
       inc       r9
       dec       rax
       test      rax,rax
       jg        short M01_L03
M01_L04:
       cmp       r9,r10
       jge       near ptr M01_L12
       lea       rax,[rcx+r9*2]
       mov       r11,rax
       test      r11b,1F
       je        short M01_L06
       movzx     r11d,dx
       vmovd     xmm0,r11d
       vpbroadcastw xmm0,xmm0
       vmovupd   xmm1,[rax]
       vpcmpeqw  xmm0,xmm0,xmm1
       vpmovmskb r11d,xmm0
       test      r11d,r11d
       jne       short M01_L05
       add       r9,8
       jmp       short M01_L06
M01_L05:
       xor       eax,eax
       tzcnt     eax,r11d
       mov       r10d,eax
       shr       r10d,1F
       add       eax,r10d
       sar       eax,1
       add       eax,r9d
       jmp       near ptr M01_L17
M01_L06:
       mov       rax,r10
       sub       rax,r9
       and       rax,0FFFFFFFFFFFFFFF0
       test      rax,rax
       jle       short M01_L08
       movzx     r11d,dx
       vmovd     xmm0,r11d
       vpbroadcastw ymm0,xmm0
M01_L07:
       vmovupd   ymm1,[rcx+r9*2]
       vpcmpeqw  ymm1,ymm0,ymm1
       vpmovmskb r8d,ymm1
       test      r8d,r8d
       jne       short M01_L09
       add       r9,10
       add       rax,0FFFFFFFFFFFFFFF0
       test      rax,rax
       jg        short M01_L07
M01_L08:
       mov       rax,r10
       sub       rax,r9
       and       rax,0FFFFFFFFFFFFFFF8
       test      rax,rax
       jle       short M01_L11
       movzx     r11d,dx
       vmovd     xmm0,r11d
       vpbroadcastw xmm0,xmm0
       vmovupd   xmm1,[rcx+r9*2]
       vpcmpeqw  xmm0,xmm0,xmm1
       vpmovmskb eax,xmm0
       test      eax,eax
       jne       short M01_L10
       add       r9,8
       jmp       short M01_L11
M01_L09:
       xor       eax,eax
       tzcnt     eax,r8d
       mov       r10d,eax
       shr       r10d,1F
       add       eax,r10d
       sar       eax,1
       add       eax,r9d
       jmp       short M01_L17
M01_L10:
       tzcnt     eax,eax
       mov       edx,eax
       shr       edx,1F
       add       eax,edx
       sar       eax,1
       add       eax,r9d
       jmp       short M01_L17
M01_L11:
       cmp       r9,r10
       jge       short M01_L12
       mov       rax,r10
       sub       rax,r9
       jmp       near ptr M01_L01
M01_L12:
       mov       eax,0FFFFFFFF
       vzeroupper
       pop       rsi
       ret
M01_L13:
       lea       eax,[r9+3]
       jmp       short M01_L17
M01_L14:
       lea       eax,[r9+2]
       jmp       short M01_L17
M01_L15:
       lea       eax,[r9+1]
       jmp       short M01_L17
M01_L16:
       mov       eax,r9d
M01_L17:
       vzeroupper
       pop       rsi
       ret
; Total bytes of code 462

BenchmarkDotNet=v0.12.1.1405-nightly, OS=Windows 10.0.18363.959 (1909/November2019Update/19H2)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-rc.1.20413.9
  [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.20.40416, CoreFX 5.0.20.40416), X64 RyuJIT
  Job-GTRGSX : .NET Core 5.0.0 (CoreCLR 5.0.20.40416, CoreFX 5.0.20.40416), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Runtime=.NET Core 5.0  Arguments=/p:DebugType=portable  
Toolchain=netcoreapp5.0  IterationTime=250.0000 ms  MaxIterationCount=20  
MinIterationCount=15  WarmupCount=1  

Method	Size	Mean	Error	StdDev	Median	Min	Max	Gen 0	Gen 1	Gen 2	Allocated	Code Size
IndexOfValue	512	16.95 ns	0.143 ns	0.127 ns	16.90 ns	16.82 ns	17.25 ns	-	-	-	-	499 B

.NET Core 5.0.0 (CoreCLR 5.0.20.40416, CoreFX 5.0.20.40416), X64 RyuJIT

; System.Memory.Span`1[[System.Char, System.Private.CoreLib]].IndexOfValue()
       sub       rsp,28
       mov       rdx,[rcx+18]
       test      rdx,rdx
       jne       short M00_L00
       xor       r8d,r8d
       xor       eax,eax
       jmp       short M00_L01
M00_L00:
       lea       r8,[rdx+10]
       mov       eax,[rdx+8]
M00_L01:
       movzx     edx,word ptr [rcx+2C]
       mov       rcx,r8
       mov       r8d,eax
       call      System.SpanHelpers.IndexOf(Char ByRef, Char, Int32)
       nop
       add       rsp,28
       ret
; Total bytes of code 48

; System.SpanHelpers.IndexOf(Char ByRef, Char, Int32)
       push      rsi
       vzeroupper
       xor       r9d,r9d
       movsxd    r10,r8d
       mov       rax,r10
       mov       r11,rcx
       test      r11b,1
       jne       short M01_L01
       cmp       r8d,10
       jl        short M01_L01
       mov       rax,rcx
       neg       eax
       mov       r8d,eax
       shr       r8d,1F
       add       eax,r8d
       sar       eax,1
       mov       eax,eax
       and       rax,7
       cmp       rax,4
       jl        short M01_L02
M01_L00:
       lea       r8,[rcx+r9*2]
       movzx     r11d,dx
       movzx     esi,word ptr [r8]
       cmp       r11d,esi
       je        near ptr M01_L16
       movzx     esi,word ptr [r8+2]
       cmp       r11d,esi
       je        near ptr M01_L15
       movzx     esi,word ptr [r8+4]
       cmp       r11d,esi
       je        near ptr M01_L14
       movzx     r8d,word ptr [r8+6]
       cmp       r11d,r8d
       je        near ptr M01_L13
       add       r9,4
       add       rax,0FFFFFFFFFFFFFFFC
M01_L01:
       cmp       rax,4
       jge       short M01_L00
M01_L02:
       test      rax,rax
       jle       short M01_L04
       movzx     r11d,dx
M01_L03:
       movzx     r8d,word ptr [rcx+r9*2]
       cmp       r8d,r11d
       je        near ptr M01_L16
       inc       r9
       dec       rax
       test      rax,rax
       jg        short M01_L03
M01_L04:
       cmp       r9,r10
       jge       near ptr M01_L12
       lea       rax,[rcx+r9*2]
       mov       r11,rax
       mov       r8d,r11d
       test      r8b,1F
       je        short M01_L06
       movzx     r11d,dx
       vmovd     xmm0,r11d
       vpbroadcastw xmm0,xmm0
       vmovupd   xmm1,[rax]
       vpcmpeqw  xmm0,xmm0,xmm1
       vpmovmskb r11d,xmm0
       test      r11d,r11d
       jne       short M01_L05
       add       r9,8
       jmp       short M01_L06
M01_L05:
       xor       r10d,r10d
       tzcnt     r10d,r11d
       shr       r10d,1
       mov       edx,r10d
       mov       eax,edx
       add       eax,r9d
       jmp       near ptr M01_L17
M01_L06:
       mov       rax,r10
       sub       rax,r9
       and       rax,0FFFFFFFFFFFFFFF0
       jle       short M01_L08
       movzx     r11d,dx
       vmovd     xmm0,r11d
       vpbroadcastw ymm0,xmm0
M01_L07:
       vmovupd   ymm1,[rcx+r9*2]
       vpcmpeqw  ymm1,ymm0,ymm1
       vpmovmskb r8d,ymm1
       test      r8d,r8d
       jne       short M01_L09
       add       r9,10
       add       rax,0FFFFFFFFFFFFFFF0
       test      rax,rax
       jg        short M01_L07
M01_L08:
       mov       rax,r10
       sub       rax,r9
       and       rax,0FFFFFFFFFFFFFFF8
       jle       short M01_L11
       movzx     r11d,dx
       vmovd     xmm0,r11d
       vpbroadcastw xmm0,xmm0
       vmovupd   xmm1,[rcx+r9*2]
       vpcmpeqw  xmm0,xmm0,xmm1
       vpmovmskb eax,xmm0
       test      eax,eax
       jne       short M01_L10
       add       r9,8
       jmp       short M01_L11
M01_L09:
       xor       eax,eax
       tzcnt     eax,r8d
       shr       eax,1
       mov       r10d,eax
       mov       eax,r10d
       add       eax,r9d
       jmp       short M01_L17
M01_L10:
       xor       edx,edx
       tzcnt     edx,eax
       shr       edx,1
       mov       ecx,edx
       mov       eax,ecx
       add       eax,r9d
       jmp       short M01_L17
M01_L11:
       cmp       r9,r10
       jge       short M01_L12
       mov       rax,r10
       sub       rax,r9
       jmp       near ptr M01_L01
M01_L12:
       mov       eax,0FFFFFFFF
       vzeroupper
       pop       rsi
       ret
M01_L13:
       lea       eax,[r9+3]
       jmp       short M01_L17
M01_L14:
       lea       eax,[r9+2]
       jmp       short M01_L17
M01_L15:
       lea       eax,[r9+1]
       jmp       short M01_L17
M01_L16:
       mov       eax,r9d
M01_L17:
       vzeroupper
       pop       rsi
       ret
; Total bytes of code 451

[kunalspathak]

Thanks @adamsitnik for collecting the numbers. Few things:

1. The original issue was regression between July 15 and July 21 but it looks like this is a regresison between .NET Core 3.1 and .NET 5. Which version of .NET 5 did you check? Was it preview8 or rc1? Can you share the commit hash, if possible?
2. Also, the original issue was Ubuntu only (or that is what was reported), but looks like it also regress on Windows.
3. I wanted to see the latest numbers for this benchmark to see if this has bimodel trend, but I am not sure how to get that data.

Update: Looking at the jitdiff, I don't see anything suspicious. The code size reduced from 462 -> 451 bytes.