Arm64: Add SVE/SVE2 support in .NET 9

[kunalspathak]

Until Armv8, NEON architecture enabled users to write vectorized code using SIMD instructions. The vector length for NEON instructions remains fixed at 128 bits. To take into account High Performance Computing applications, newer versions of Arm like v9 has developed Scalable Vector Extension (SVE). SVE is a new programming model that contains SIMD instructions operating on flexible vector length, ranging from 128 bits to 2048 bits. SVE2 extends SVE to enable domains like Computer vision, multimedia, etc. More details about SVE can be found on Arm's website. In .NET 9, we want to start the foundational work in .NET libraries and RyuJIT to add support for SVE2 feature.

• Design: The instructions present in SVE2 needs to be exposed through .NET libraries to our customers. However, we want to make sure that the customers don't have to rewrite their code to consume SVE2 functionality and the experience of using SVE2 features should be seamless. Currently, .NET has Vector<T> that represents flexible vector length depending on the underlying hardware and the idea is to expose SVE2 "flexible vector length" functionality using Vector<T>. We need to think about the common instructions and validate if they can be represented by APIs that just takes Vector<T> as parameter. Additionally, SVE2 introduces "predicate register" to mark vector lanes active/inactive. We do not want to expose this concept as well to our consumer through .NET libraries because of reason mentioned above. Hence, for every API, need to come up with a pseudo code of how the API should be implemented internally in JIT such that the "predicate register" concept is created and consumed inside JIT implicitly. There is a good discussion that happened in the past about the API proposal in [API Proposal]: Example usages of a VectorSVE API #88140.
• Implement APIs in System.Runtime.Intrinsic.Arm.SVE2: Once design is finalized, need to add all the APIs in a new SVE2 class under System.Runtime.Intrinsic.Arm. They need to be plumbed through the JIT (by transforming tree nodes to represent "predicate register" concept, if needed). all the way to generating code. Arm64: Implement SVE APIs #99957
• If possible, automatically generate the test template for various APIs
• Update Antigen to pick up the new SVE APIs
• SVE hardware availability in CI
• Backend support: Regardless of API design, we need to add instruction encoding of all the SVE2 instructions that we are planning to support. Here is a rough list of things that needs to happen to add the new instructions. Covered in Arm64: Implement SVE encodings #94549.
  • Add new entries in hwinstrinsiclistarm64.h for AdvSimd.SVE2 APIs
  • Depending on the API call the right emitIns_*() code
  • emitIns_*() methods add the new instructions support
  • emitfmtsarm64.h - needs to add new instruction formatting Arm64: SVE/SVE2 encodings #94285
  • instrsarm64.h - needs to add new instruction to instruction formatting mapping Arm64: SVE/SVE2 encodings #94285
  • Add the encoding for new instructions in emitOutputInstr()
  • Add new Zx registers and predicate registers LSRA: Add support to track more than 64 registers #99658
  • Add JitStressRegs mode to always allocate high Z/P register unless the candidate specifically says to use low registers.
  • Make sure TP regression is minimum
  • Testing if the encoding matches the display ins using windbg/msvc/cordistools
  • Add entries for the new instructions in genArm64EmitterUnitTests()
  • Fix the formatEncode* data
  • Refactor and consolidate the common code/asserts, convert some methods lookup like insSveIsLslN / insSveGetLslOrModN into table drive loop up.
  • Possibly frame layout code update such that the SVE_simd and SVE_mask are towards the end of the frame in order for offset in stack to take into consider the variable VL.
• Automation for encoding: If we see above list, it is very time consuming to add each instruction's encoding in the code base. There are 800+ instructions and considering 30 mins for each instruction, it will take 400+ human hours to add and validate the encoding. Hence, there is an experiment going on to generate the encoding data and the C++ code around it automatically. The understanding is that the C++ code generated will not be accurate and manual inspection will still be required.
  Edit: A working prototype of the tool that generates 2 C++ files for encoding is in Arm64: SVE/SVE2 encodings #94285.
  • Produce a json/xml file that contains all the encoding data of all the instructions. A good resource from which this can be extracted is here.
  • Recreate instrsarm64.h that contains the existing as well as the new formats along with encoding binary representation and hexadecimal representation.
  • If there are new instruction formats, add their entries in emitfmtsarm64.h file.
  • Based on the mmmm, dddd, etc. in the binary representation of encoding, have to tool write a logic to produce the instruction bytes. In other words, this will generate code that can be pasted in emitOutputInstr() function.
  • Depending on the number of encodings in each group of instructions, they can be tied to appropriate INST* like INST9 or INST8, etc. and regenerated in sorted order.
    Note that if we need to regenerate existing files like instrsarm64.h and emitfmtsarm64.h , existing instruction's encoding also needs to be generated by the tool.
• MOVPRFX instruction support: GCStress, sanity check. Also, when emitting every single instruction, need to make sure that if previous instruction was movprfx, we verify it follows all the rules with regards to destination registers, size, etc. Arm64/Sve: Validation for movprfx in emitter #105514
• Before consuming the new APIs in libraries, make sure that Mono has support for it or else SVE.IsSupported should return false for it.
• On Windows, make sure that new SVE registers as well as predicate registers are available during debugging as well as Vector<T> shows the right number of elements.
• ABI related changes for SVE/predicate registers in VM
• Support breakpoint patching for SVE instructions
  • ICorDebug needs to be updated to know which SVE instructions have relative read/write/jumps - walker.cpp is specifically the place that needs to be updated
  • Similar work has been done for AVX512 - Support breakpoints on AVX-512 instructions #89705

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Overview

As we did in the past for .NET 5, .NET 7 and .NET 8, we would like to continue improving Arm64 in .NET 9 as well. Here are the top-level themes we plan to address. Some of the issues are from the past releases that we did not get time to work upon. while others are about adding instructions of newer arm versions or exposing the Arm functionality to the .NET API level.

SVE2 support

Until Armv8, NEON architecture enabled users to write vectorized code using SIMD instructions. The vector length for NEON instructions remains fixed at 128 bits. To take into account High Performance Computing applications, newer versions of Arm like v9 has developed Scalable Vector Extension (SVE). SVE is a new programming model that contains SIMD instructions operating on flexible vector length, ranging from 128 bits to 2048 bits. SVE2 extends SVE to enable domains like Computer vision, multimedia, etc. More details about SVE can be found on Arm's website. In .NET 9, we want to start the foundational work in .NET libraries and RyuJIT to add support for SVE2 feature.

• Design: The instructions present in SVE2 needs to be exposed through .NET libraries to our customers. However, we want to make sure that the customers don't have to rewrite their code to consume SVE2 functionality and the experience of using SVE2 features should be seamless. Currently, .NET has Vector<T> that represents flexible vector length depending on the underlying hardware and the idea is to expose SVE2 "flexible vector length" functionality using Vector<T>. We need to think about the common instructions and validate if they can be represented by APIs that just takes Vector<T> as parameter. Additionally, SVE2 introduces "predicate register" to mark vector lanes active/inactive. We do not want to expose this concept as well to our consumer through .NET libraries because of reason mentioned above. Hence, for every API, need to come up with a pseudo code of how the API should be implemented internally in JIT such that the "predicate register" concept is created and consumed inside JIT implicitly. There is a good discussion that happened in the past about the API proposal in [API Proposal]: Example usages of a VectorSVE API #88140.
• Implement APIs in System.Runtime.Intrinsic.Arm.SVE2: Once design is finalized, need to add all the APIs in a new SVE2 class under System.Runtime.Intrinsic.Arm. They need to be plumbed through the JIT (by transforming tree nodes to represent "predicate register" concept, if needed). all the way to generating code.
• Backend support: Regardless of API design, we need to add instruction encoding of all the SVE2 instructions that we are planning to support. Here is a rough list of things that needs to happen to add the new instructions:
  • Add new entries in hwinstrinsiclistarm64.h for AdvSimd.SVE2 APIs
  • Depending on the API call the right emitIns_*() code
  • emitIns_*() methods add the new instructions support
  • emitfmtsarm64.h - needs to add new instruction formatting
  • instrsarm64.h - needs to add new instruction to instruction formatting mapping
  • Add the encoding for new instructions in emitOutputInstr()
  • Add new Zx registers and predicate registers
  • Make sure TP regression is minimum
  • Testing if the encoding matches the display ins using windbg/msvc/cordistools
  • Add entries for the new instructions in genArm64EmitterUnitTests()
  • Fix the formatEncode* data
• Automation for encoding: If we see above list, it is very time consuming to add each instruction's encoding in the code base. There are 800+ instructions and considering 30 mins for each instruction, it will take 400+ human hours to add and validate the encoding. Hence, there is an experiment going on to generate the encoding data and the C++ code around it automatically. The understanding is that the C++ code generated will not be accurate and manual inspection will still be required.
  • Produce a json/xml file that contains all the encoding data of all the instructions. A good resource from which this can be extracted is here.
  • Recreate instrsarm64.h that contains the existing as well as the new formats along with encoding binary representation and hexadecimal representation.
  • If there are new instruction formats, add their entries in emitfmtsarm64.h file.
  • Based on the mmmm, dddd, etc. in the binary representation of encoding, have to tool write a logic to produce the instruction bytes. In other words, this will generate code that can be pasted in emitOutputInstr() function.
  • Depending on the number of encodings in each group of instructions, they can be tied to appropriate INST* like INST9 or INST8, etc. and regenerated in sorted order.
    Note that if we need to regenerate existing files like instrsarm64.h and emitfmtsarm64.h , existing instruction's encoding also needs to be generated by the tool.

New instructions

• [API Proposal]: Arm64 [Load/Store]Vector64 and [Load/Store]Vector128 for 2,3 and 4 variants #84510
• Explore new instructions added in Armv8.3 ~ Armv9 and see if we can use them in JIT
• Start using Post increment addressing mode in JIT wherever applicable.

Performance improvements

• Review the multi-op instruction usage for Arm64 #68028
• Consider using SIMD registers for "hot" local variables instead of placing them on stack when out of free GP registers #10444
• Arm64: Use VectorTableLookup/VectorTableLookupExtension APIs in libraries #84328
• Consume LoadVector/StoreVector in .NET libraries

Stretch goals

• Consume SVE2 APIs in .NET libraries
• Investigate instability in TE for Arm64 #77916
• Experiment how much TP impact we will see if enabled pointer authentication for coreclr. Depending on that, decide if it should be enabled for JIT.

Author:	kunalspathak
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	-

[kunalspathak]

cc: @a74nh @TamarChristinaArm @SwapnilGaikwad

[jkotas]

Currently, .NET has Vector that represents flexible vector length depending on the underlying hardware and the idea is to expose SVE2 "flexible vector length" functionality using Vector.

We should think about how this is going to work with the new SVE streaming mode. Do we expect to support the SVE streaming mode in .NET eventually? If yes, how it is going to affect the design?

[tannergooding]

Do we expect to support the SVE streaming mode in .NET eventually

I'd presume it would be desirable to support. Giving users the power of the per hardware lightup is a significant advantage to many backing frameworks that power .NET applications.

If yes, how it is going to affect the design?

The streaming instructions are more abstract and I don't think the design for C++ has been finalized yet either. I at least don't see any of the new instructions under https://developer.arm.com/architectures/instruction-sets/intrinsics

I expect they will be a completely separate consideration for how they're supported as compared to the more standard SIMD processing instructions and we'll need to cross that bridge when we come to it.

This is particularly given how it allows the effective SVE length to be changed and requires explicit start/stop instructions. And so I imagine the runtime itself will require some complex changes to work with the concept and ensure it doesn't impact the ABI, field usage, inlining boundaries, etc.

[jkotas]

The streaming instructions are more abstract and I don't think the design for C++ has been finalized yet either. I at least don't see any of the new instructions under

My understanding that the streaming mode uses the same instructions as regular SVE mode (subset of them), except that the instructions operate on larger vector sizes. If we were to use the current Vector<T> in both streaming and non-streaming mode, the size of Vector<T> would need to change between these two modes somehow. I do not see how we would pull this off.

This observation made me think that reusing Vector<T> for SVE registers may be a poor choice. Would it make more sense to have a new special type that can be only use on stack for locals and arguments and that does not have a constant size at runtime?

[tannergooding]

Would it make more sense to have a new special type that can be only use on stack for locals and arguments and that does not have a constant size at runtime?

That is effectively how C describes its SVE support. They are similar to "incomplete" types in many aspects, but with a few less restrictions. They can't be used with atomics, arrays, sizeof, pointer arithmetic, fields, captures, etc. They can still be used with ref T, T*, templates, parameters/return types, etc.

At least for non-streaming, a lot of that is only restrictive because C/C++ has existing requirements for things like sizeof(T) to be constant and is namely AOT. There's not actually anything "preventing" most of it from working, it would just require the relevant cnt* instruction to be executed and included as part of the size computation dynamically (and can be constant in a JIT environment).

If we did define a new type, I think we'd functionally be defining a ref struct ScalableVector + writing an analyzer to help enforce it doesn't get used for some other edge cases. I don't think that a platform specific vector type would warrant a proper language feature or language level enforcement for C#/F#/etc.

My understanding that the streaming mode uses the same instructions as regular SVE mode (subset of them)

This is my understanding as well.

For reference, the Procedure Call Standard` is here: https://github.com/ARM-software/abi-aa/releases/download/2023Q1/aapcs64.pdf

While the Vector Function ABI is here: https://github.com/ARM-software/abi-aa/releases/download/2023Q1/vfabia64.pdf

Non-streaming SVE has clear and concrete benefits and can easily integrate with our existing types and concepts. It doesn't really require any significant changes to make light up (no more than AVX-512 did, anyways). The biggest open question for it is how exactly to support it for AOT. But we also have a couple solutions to support both the VLA (vector length agnostic) and VLS (vector length specific) concepts. We can also section that work if required and only support VLS for AOT in the first iteration (much as is already done for Vector<T> today). -- Any work to improve Vector<T> for AOT or R2R scenarios then benefits all platforms and can likely be shared.

Streaming SVE still has some concrete benefits, however, it works significantly differently from anything we've exposed so far. There are simply too many open ended questions around it, how the user will start/stop the functionality, how it will impact the managed/native ABI boundary, inlining, callee/caller save state, etc. Trying to solve all of these would likely push non-streaming SVE support out of .NET 9 and potentially put us in a place where non-streaming SVE is functionally worse off from a general usability perspective. So I think it's ultimately better to, for now, treat it like it doesn't exist and say its UB if a user enables it (much as we define for other similar user togglable CPU state such as: floating-point exceptions, changing the default rounding mode, flushing denormals, enabling strict alignment validation, etc).

Then, when we're ready to take a deeper look at the feature, we can determine whether using Vector<T> + an analyzer is feasible for it or whether we truly need a new type. We can also, at the same time, answer the other questions about how it works at the ABI level for managed code, including GC pause points. -- I imagine, based on the general blog posts and other areas around the topic, we likely don't want to look at it until we have the ability to generate multiple versions of a function (one streaming aware, one streaming unaware), to efficiently handle the various dynamic callee/caller save state support around the ZA storage area, to do relevant state tracking for streaming enablement, etc

[jkotas]

Non-streaming SVE has clear and concrete benefits and can easily integrate with our existing types and concepts. It doesn't really require any significant changes to make light up (no more than AVX-512 did, anyways). The biggest open question for it is how exactly to support it for AOT.

Good point about AOT support for non-streaming SVE. How are we going to do that with Vector<T>? I do not see any good options.

[tannergooding]

How are we going to do that with Vector? I do not see any good options.

It ultimately depends on the code the user writes and how much we want to invest.

For recommended coding patterns, where the user treats Vector<T> and Vector64/128/256/512<T> like a ref struct, then it works much as it does in C/C++ and we can trivially generate VLA (vector length agnostic) code. The ABI already exists to support passing any number of Vector<T> arguments, declaring Vector<T> locals, and other scenarios; so we just need to support it as it is supported in native.

If the user deviates from this and starts using Vector<T> as the field of a struct, declares arrays of it, or other scenarios that depend on the size, then it gets a little more involved. However, we can still generate VLA code by making some computations dynamic. For example, sizeof(Vector<int>) functionally becomes cntd * 4. Since the packing of Vector<T> is well-defined, regardless of size, we can then use this to make offset calculations or other sizeof calculations work just as well and also for cheap.

We also have the option of generation VLS (vector length specific) code. This functionally means we can say a given method only support Vector<T> if it is 128-bits. This is basically how Vector<T> works on x64 when you say it should be 32-bytes, after all. If we ever get to the point of being able to generate multiple versions of a function, we could support generating both 128-bit and 256-bit versions in the future, or however we need based on the sizes we or the user wants to support. -- In the worst case, this simply fails to launch the app if SVE support exists and the size is larger than appropriate, but we also have the option of explicitly using predication if the actual size is larger. In practice, most SVE hardware today remains 128-bit and only a couple exceptions are larger (a 512-bit super computer and a 256-bit implementation for Graviton3). We can likewise guard paths based on some if (Sve.IsSupported and sizeof(Vector<T>) == ...) check if that's desirable. It really just comes down to how much effort we want to spend supporting the usage of Vector<T> in already non-standard scenarios (noting that support can be incremental over time as well).