JIT: Generating rep movsb for cpblk with non-constant arguments

[jamesqo]

Separated out from the discussion in dotnet/coreclr#6638

Currently the JIT makes a helper call to JIT_MemCpy for non-constant sizes. If it could generate rep movsb inline instead for amd64/x86, that would be much more efficient. Doing this would allow for a better implementation of Buffer.MemoryCopy, which right now has xmm as the best thing available.

category:cq
theme:block-opts
skill-level:intermediate
cost:medium

[benaadams]

Is it this that needs to change?
https://github.com/dotnet/coreclr/blob/master/src/vm/amd64/CrtHelpers.asm

Older CPUs seem to work better with rep movsb 16 byte aligned; > 128 byte copy it improves over vectors (setup costs)

[benaadams]

CPUID EAX=7, EBX=0 => Bit 9 Enhanced REP MOVSB/STOSB i.e. whether MOVSB is fast
(Preferably cached as CPUID isn't fast?)

[redknightlois]

For the fastest memcopy working on almost every any architecture, you should have to check for the cpuid and relink using an stub on load. Then use the fastest method and instruction set for the executing architecture. That is probably something a JIT helper could do.

[benaadams]

For the fastest memcopy working on almost every any architecture, you should have to check for the cpuid and relink using an stub on load. Then use the fastest method and instruction set for the executing architecture.

Lots of variant code bases, might only want to target a few common scenarios. At least it seems to be standardising.

[jamesqo]

@benaadams

Is it this that needs to change?

It's this part in the code generator and here in the lowerer; currently, it looks as if for AMD64 it's generating a couple of xmm writes for constants divisible by 16 under 64, and otherwise generating a call.

In fact, right now I don't the JIT ever generates a rep movsb for x64, even for constant values; this assert will clearly always fail, since if you do a GitHub search CPBLK_UNROLL_LIMIT is 64 and CPBLK_MOVS_LIMIT is 16. Perhaps there was a typo in the latter's definition?

cc @BruceForstall

[redknightlois]

@benaadams not really. Even though there are a few variants, even the standard prefetch/non-temporal version should be unbeatable against any potential memcopy written in managed code at any size.

[RussKeldorph]

/Cc @dotnet/jit-contrib @lt72

[tannergooding]

@redknightlois, that may be true when looking at a purely unsafe FCALL to memcpy. However, when you have to do a QCALL and do argument validation and exception handling, the managed implementation beats out the unmanaged implementation for small buffers (QCALLs are expensive).

Additionally, looking at the memcpy implementation on Windows, the prefetch/non-temporal version is only used when the number of 128-byte blocks to copy is greater than 8239 (1,054,592 bytes). Feel free to look at the disassembly yourself to confirm 😄

The Intel Optimization Manual (http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html) has some great documentation on the exact benefits and drawbacks of various implementations and even provides heuristics for how to implement memcpy using xmm and when to use prefetching and non-temporal stores.

[redknightlois]

@tannergooding I know, we have our own memory copy implementation since 1.5 years ago (that's why I a following this thread closely). Any clever improvement in memcpy (even if not "officially aproved" for inclusion in CoreCLR) would have a real payoff for us.

And as you mention that is the point, we are still thinking in terms of QCALL instead of implementing it at the JIT/CoreCLR level as a FCALL. The potential improvement of doing that straight as a code-generation solution is too big to pass. Also handcoding your memcpy implementation to be "faster" based on what the JIT can recognize and optimize today is a very brittle solution for such a core function. We benchmark our memcpy algorithm every 6 months on multiple hardware and also JIT versions to be sure that we are not regressing when optimizations and/or changes to the code generation happen. We pay the cost of finding the 'sweet spot' as we cannot control the environment our code will run (something at jitting time we can know), but we would certainly would like to avoid it altogether.

EDIT: Probably this comment is more relevant in the context of the parent issue. dotnet/coreclr#6638

[benaadams]

We benchmark our memcpy algorithm every 6 months on multiple hardware

If you'd be willing to contribute it; then changing the asm here https://github.com/dotnet/coreclr/blob/master/src/vm/amd64/CrtHelpers.asm would avoid a QCALL or an FCALL I believe?

As its a STDCALL https://github.com/dotnet/coreclr/blob/master/src/vm/jitinterface.h#L406

[benaadams]

Although would the linux version be a better approach? As that just calls the platform version https://github.com/dotnet/coreclr/blob/master/src/vm/amd64/crthelpers.S#L16-L38

[redknightlois]

@benaadams it is a managed version. So yes, I believe doing all this stuff there with proper CPU dispatching should be the way to go. I have a few bigger fishes to fry on our side regarding IO, but if the time comes and we need to make memcpy faster there would probably be time to invest in some assembler work. Question that has been bugging me for a long time already is: when that one is selected when jitting?

[benaadams]

Question that has been bugging me for a long time already is: when that one is selected when jitting?

Is cpblk so copying structs as arguments and for assignment I think? or Unsafe.CopyBlock

[Anderman]

movsb has some overhead see SO.

I created a test with rep movsb. movsb was called by pinvoke. The movsb test is corrected by PInvoke overhead. Arraycopy also use movsb for size>128. I used vector copy

[tannergooding]

Possibly: https://github.com/dotnet/designs/issues/13