JIT_NewS_MP_FastPortable is slow for Orchard CMS benchmark on Linux-arm64

[EgorBo]

Orchard CMS is one of the most complicated benchmark in our test suite (way more complicated than TE benchmarks) so, presumably, it's closer to real-world. Currently, ARM64 is twice slower than x64 (comparable HW) on it while for TE benchmark the same ARM64 hardware is typically 1.2-1.5x faster than that x64 AMD machine.
The perf trace is pointing to JIT_NewS_MP_FastPortable:

(sorted by 'self' aks exclusive time).

The annoted asm for it:

Or (output of bjdump -d):

000000000033fbf0 <_Z24JIT_NewS_MP_FastPortableP21CORINFO_CLASS_STRUCT_>:
  33fbf0:	a9bf7bfd 	stp	x29, x30, [sp, #-16]!
  33fbf4:	910003fd 	mov	x29, sp
  33fbf8:	aa0003e8 	mov	x8, x0
  33fbfc:	90001920 	adrp	x0, 663000 <_ZTV12EEJitManager+0x50>
  33fc00:	f947dc01 	ldr	x1, [x0, #4024]
  33fc04:	913ee000 	add	x0, x0, #0xfb8
  33fc08:	d63f0020 	blr	x1
  33fc0c:	d53bd049 	mrs	x9, tpidr_el0
  33fc10:	b940050a 	ldr	w10, [x8, #4]
  33fc14:	f8606929 	ldr	x9, [x9, x0]
  33fc18:	a945ad20 	ldp	x0, x11, [x9, #88]
  33fc1c:	cb00016b 	sub	x11, x11, x0
  33fc20:	eb0a017f 	cmp	x11, x10
  33fc24:	54000082 	b.cs	33fc34 <_Z24JIT_NewS_MP_FastPortableP21CORINFO_CLASS_STRUCT_+0x44>  // b.hs, b.nlast
  33fc28:	aa0803e0 	mov	x0, x8
  33fc2c:	a8c17bfd 	ldp	x29, x30, [sp], #16
  33fc30:	14000006 	b	33fc48 <_Z7JIT_NewP21CORINFO_CLASS_STRUCT_>
  33fc34:	8b0a000a 	add	x10, x0, x10
  33fc38:	f9002d2a 	str	x10, [x9, #88]
  33fc3c:	f9000008 	str	x8, [x0]
  33fc40:	a8c17bfd 	ldp	x29, x30, [sp], #16
  33fc44:	d65f03c0 	ret

If I read this correctly, we spend most of them time reading data from TLS (to get current allocator ?) - I'd expect to see time spent inside the fallback call to gc, but it's not (if the trace is accurate). Another hint from the general trace is that we don't really spend a lot of time in GC itself.

PS: what is blr x1 call ? I presume it should be part of the GetThread()

cc @mangod9 @kunalspathak

[mangod9]

would make sense to compare with a similar gen0size setting and collect hardware perf counters in case that shows any memory access issues.

[EgorBo]

And just for the reference, a hand-written asm for NEWSFAST helper on x64:

runtime/src/coreclr/vm/amd64/JitHelpers_InlineGetThread.asm

Lines 160 to 211 in 6aca07a

	LEAF_ENTRY JIT_NewArr1VC_MP_InlineGetThread, _TEXT
	; We were passed a (shared) method table in RCX, which contains the element type.
	; The element count is in RDX
	; NOTE: if this code is ported for CORINFO_HELP_NEWSFAST_ALIGN8, it will need
	; to emulate the double-specific behavior of JIT_TrialAlloc::GenAllocArray.
	; Do a conservative check here. This is to avoid overflow while doing the calculations. We don't
	; have to worry about "large" objects, since the allocation quantum is never big enough for
	; LARGE_OBJECT_SIZE.
	; For Value Classes, this needs to be 2^16 - slack (2^32 / max component size),
	; The slack includes the size for the array header and round-up ; for alignment. Use 256 for the
	; slack value out of laziness.
	; In both cases we do a final overflow check after adding to the alloc_ptr.
	cmp rdx, (65535 - 256)
	jae OversizedArray
	movzx r8d, word ptr [rcx + OFFSETOF__MethodTable__m_dwFlags] ; component size is low 16 bits
	imul r8d, edx
	add r8d, dword ptr [rcx + OFFSET__MethodTable__m_BaseSize]
	; round the size to a multiple of 8
	add r8d, 7
	and r8d, -8
	INLINE_GETTHREAD r11
	mov r10, [r11 + OFFSET__Thread__m_alloc_context__alloc_limit]
	mov rax, [r11 + OFFSET__Thread__m_alloc_context__alloc_ptr]
	add r8, rax
	jc AllocFailed
	cmp r8, r10
	ja AllocFailed
	mov [r11 + OFFSET__Thread__m_alloc_context__alloc_ptr], r8
	mov [rax], rcx
	mov dword ptr [rax + OFFSETOF__ArrayBase__m_NumComponents], edx
	ret
	OversizedArray:
	AllocFailed:
	jmp JIT_NewArr1
	LEAF_END JIT_NewArr1VC_MP_InlineGetThread, _TEXT

[kunalspathak]

And just for the reference, a hand-written asm for NEWSFAST helper on x64:

runtime/src/coreclr/vm/amd64/JitHelpers_InlineGetThread.asm

Lines 160 to 211 in 6aca07a

	LEAF_ENTRY JIT_NewArr1VC_MP_InlineGetThread, _TEXT
	; We were passed a (shared) method table in RCX, which contains the element type.
	; The element count is in RDX
	; NOTE: if this code is ported for CORINFO_HELP_NEWSFAST_ALIGN8, it will need
	; to emulate the double-specific behavior of JIT_TrialAlloc::GenAllocArray.
	; Do a conservative check here. This is to avoid overflow while doing the calculations. We don't
	; have to worry about "large" objects, since the allocation quantum is never big enough for
	; LARGE_OBJECT_SIZE.
	; For Value Classes, this needs to be 2^16 - slack (2^32 / max component size),
	; The slack includes the size for the array header and round-up ; for alignment. Use 256 for the
	; slack value out of laziness.
	; In both cases we do a final overflow check after adding to the alloc_ptr.
	cmp rdx, (65535 - 256)
	jae OversizedArray
	movzx r8d, word ptr [rcx + OFFSETOF__MethodTable__m_dwFlags] ; component size is low 16 bits
	imul r8d, edx
	add r8d, dword ptr [rcx + OFFSET__MethodTable__m_BaseSize]
	; round the size to a multiple of 8
	add r8d, 7
	and r8d, -8
	INLINE_GETTHREAD r11
	mov r10, [r11 + OFFSET__Thread__m_alloc_context__alloc_limit]
	mov rax, [r11 + OFFSET__Thread__m_alloc_context__alloc_ptr]
	add r8, rax
	jc AllocFailed
	cmp r8, r10
	ja AllocFailed
	mov [r11 + OFFSET__Thread__m_alloc_context__alloc_ptr], r8
	mov [rax], rcx
	mov dword ptr [rax + OFFSETOF__ArrayBase__m_NumComponents], edx
	ret
	OversizedArray:
	AllocFailed:
	jmp JIT_NewArr1
	LEAF_END JIT_NewArr1VC_MP_InlineGetThread, _TEXT

This is the current disassembly of the C++ helper on x64:

[kunalspathak]

PS: what is blr x1 call ? I presume it should be part of the GetThread()

Yes it is call to get the offset of the TLS variable

[mangod9]

so is the call inlined for arm64 but not x64 per the disasm above?

[kunalspathak]

so is the call inlined for arm64 but not x64 per the disasm above?

Seems for unix, we should be just using the same C++ code:

runtime/src/coreclr/vm/jitinterfacegen.cpp

Lines 75 to 79 in e9e33e1

	#ifdef TARGET_UNIX
	SetJitHelperFunction(CORINFO_HELP_NEWSFAST, JIT_NewS_MP_FastPortable);
	SetJitHelperFunction(CORINFO_HELP_NEWSFAST_ALIGN8, JIT_NewS_MP_FastPortable);
	SetJitHelperFunction(CORINFO_HELP_NEWARR_1_VC, JIT_NewArr1VC_MP_FastPortable);
	SetJitHelperFunction(CORINFO_HELP_NEWARR_1_OBJ, JIT_NewArr1OBJ_MP_FastPortable);

and as seen below when I attached debugger (left= x64, right= arm64)

@EgorBo - do you have similar assembly level instrumentation on which portion of the helper is hot? The instruction ldr x0, [x9, x0] is basically accessing the field m_pThread of the gCurrentThreadInfo and if we see lot of cache misses around here (yet to know), one theory could be that newer or lot of threads accessing the helper and are touching the memory that is not in cache?

runtime/src/coreclr/vm/threads.inl

Lines 29 to 45 in e9e33e1

	#ifdef _MSC_VER
	__declspec(selectany) __declspec(thread) ThreadLocalInfo gCurrentThreadInfo;
	#else
	EXTERN_C __thread ThreadLocalInfo gCurrentThreadInfo;
	#endif
	inline Thread* GetThreadNULLOk()
	{
	return gCurrentThreadInfo.m_pThread;
	}
	inline Thread* GetThread()
	{
	Thread* pThread = gCurrentThreadInfo.m_pThread;
	_ASSERTE(pThread);
	return pThread;
	}

[kunalspathak]

Just to verify the performance of TLS access, arm64 is 4X better on this c++ benchmark that basically access a field of a struct that is a threadLocal.

arm64:

real    0m1.514s
user    0m1.488s
sys     0m0.006s

and is very simple:

x64:

real    0m4.062s
user    0m4.061s
sys     0m0.001s

x64 is more involved where it makes the syscall into __tls_get_addr

---

To summarize, I do not think TLS access here is slowing arm64 down.

[mangod9]

yeah thats what it felt like looking at the disasm. There is still something going on here since its showing up in multiple benchmarks -- doesnt feel like its a profiling issue though?

[EgorBo]

would make sense to compare with a similar gen0size setting and collect hardware perf counters in case that shows any memory access issues.

Just made both machines to use the same gen0size - it didn't change the results, arm64 is still 2x slower

[mangod9]

would make sense to compare with a similar gen0size setting and collect hardware perf counters in case that shows any memory access issues.

Just made both machines to use the same gen0size - it didn't change the results, arm64 is still 2x slower

but was the memory utilization comparable?

[EgorBo]

would make sense to compare with a similar gen0size setting and collect hardware perf counters in case that shows any memory access issues.

Just made both machines to use the same gen0size - it didn't change the results, arm64 is still 2x slower

but was the memory utilization comparable?

the working set (mainly, GC heap) was the same - 20GB for both. I'll post a detailed crank comparison between them later

[EgorBo]

cache-misses for arm64