[RyuJIT] Improve VZEROUPPER insertion

[fiigii]

Currently, RyuJIT inserts VZEROUPPER instruction very conservatively https://github.com/dotnet/coreclr/issues/7761#issuecomment-272287724, which inserts VZEROUPPER at function prologue and epilogue when the method uses 128-bit or 256-bit AVX instructions. Recently, we come across certain scenarios having CQ problems from VZEROUPPER, e.g., https://github.com/dotnet/coreclr/issues/20820, https://github.com/dotnet/coreclr/issues/21055#issuecomment-439465776, etc.

The better insertion strategy that is adopted most of the native compilers is:

1. Adding VZEROUPPER instruction after 256-bit AVX instructions are executed (only epilogue, exceptions are if any arguments supplied or return values are in YMM/ZMM registers with __vectorcall that we may support in the future).
2. Adding VZEROUPPER before any function call that might execute legacy SSE code (P/Invoke, calling VM, and AOTed code).

category:cq
theme:vector-codegen
skill-level:intermediate
cost:medium
impact:small

[fiigii]

@CarolEidt @jkotas @AndyAyersMS @tannergooding

[CarolEidt]

To clarify, this would mean that in the cases where we are currently inserting a vzeroupper in both prolog and epilog, we would change it to epilog only, and then before certain function calls. The problem with the latter is that I'm not sure we have reliable information about whether code is precompiled (we do know about the P/Invoke case). As we continue down the tiered jitting path, we could have cases where the called method changes from AOT to JIT'd.

Also, as a shorter-term adjustment, would it make sense to stop emitting the vzeroupper in the prolog?

[fiigii]

To clarify, this would mean that in the cases where we are currently inserting a vzeroupper in both prolog and epilog, we would change it to epilog only

For methods that use 256-bit AVX instructions, yes. For methods that use 128-bit AVX instructions only, we do not need any vzeroupper neither in prolog nor epilog.

The problem with the latter is that I'm not sure we have reliable information about whether code is precompiled (we do know about the P/Invoke case).

Agree. Actually, RyuJIT does not generate vzeroupper for call-sites to precompiled code, and it just handles P/Invoke call-sites now.

Also, as a shorter-term adjustment, would it make sense to stop emitting the vzeroupper in the prolog?

Yes, I think so.

[fiigii]

As we continue down the tiered jitting path

So, in the future, all the precompiled code will be JITed at Tier1, right?

[fiigii]

Also, as a shorter-term adjustment, would it make sense to stop emitting the vzeroupper in the prolog?

Yes, I think so.

To clarify, once we can correctly handle all AVX->SSE call-sites (i.e., inserting VZEROUPPER before any function call that might execute legacy SSE code), methods that use 256-bit AVX instructions would not need VZEROUPPER in prolog. Because pure SSE code is in "clean upper state", and "clean upper state"-> AVX code does not have any penalty in user code. SSE code could become "dirty upper state" iff 256-bit code calls it without VZEROUPPER.

[CarolEidt]

@fiigii - can you explain the context in which we would want to add a VZEROUPPER in the prolog currently?

[fiigii]

I do not know the exact reason that RyuJIT inserts VZEROUPPER in the prolog currently, that code was added before I joined CoreCLR 😄 . But I guess it is because we are not able to insert VZEROUPPER before calling to SSE code (i.e., JIT code -> crossge code) sometimes, which makes 256-bit AVX code not sure its caller is "clean upper state" or not.

Assume there are three methods, A and B have 256-bit AVX code and compiled by JIT, C has FP code and compiled by AOT. A calls to C and C calls to B.

The correct way should be: A -> VZEROUPPER(before call-site) -> C(clean upper state) -> B -> VZEROUPPER(B's epilogue) -> C(clean upper state) -> A.

But if we cannot insert the first VZEROUPPER, it would become: A -> C(dirty upper state, penalty) -> (penalty) B -> VZEROUPPER(B's epilogue) -> C(clean upper state) -> A.

Then inserting VZEROUPPER in B's prologue could make it better: A -> C(dirty upper state, penalty) -> VZEROUPPER(B's prologue)-> B -> VZEROUPPER(B's epilogue) -> C(clean upper state) -> A.

[CarolEidt]

@fiigii - thanks. What I don't understand is this bit: C(dirty upper state, penalty) -> (penalty) B

Is it the case that the AVX code gets a penalty when transitioning from SSE with dirty upper state? I guess that makes sense, but it wasn't clear to me previously (or it was, but I have since forgotten it!)

[fiigii]

Is it the case that the AVX code gets a penalty when transitioning from SSE with dirty upper state?

Yes, SSE -> AVX transition penalty only happens when the SSE code is in dirty upper state, which dirty upper is from previous AVX->SSE transition. And AVX->SSE transition penalty always happens if we do not insert VZEROUPPER before 256-bit-AVX -> SSE.

[CarolEidt]

Great - thanks!

[tannergooding]

It's probably handy to leave these diagrams here, which are from the architecture optimization manual:

Skylake and newer micro-architectures:

Older micro-architectures:

[tannergooding]

@fiigii and I had a brief discussion on this here: dotnet/coreclr#20788 (comment)

[CarolEidt]

Now if I can just remember to come back here when I have these questions ...

[fiigii]

and I had a brief discussion on this here: dotnet/coreclr#20788 (comment)

In that discussion, I worried about 128-bit AVX <-> SSE penalty (Preserved Non-INIT Upper state in the older micro-architecture diagram). But I consulted our native compiler experts, they said that state should be correctly handled by OS code, so we do not need any VZEROUPPER in 128-bit only AVX code.

[AndyAyersMS]

@CarolEidt perhaps you can look at setting an appropriate milestone for this?