AVX-512 support in System.Runtime.Intrinsics.X86

[twest820]

I presume supporting AVX-512 intrinsics is in plan somewhere, but couldn't find an existing issue tracking their addition. There seem to be two parts to this.

1. Support for EVEX encoding and use of zmm registers. I'm not entirely clear on compiler versus jit distinctions but perhaps this would allow jit to update existing 128 and 256 bit wide code using the Sse*, Avx*, or other System.Runtime.Intrinsics.X86 classes to EVEX.
2. Addition of Avx512 classes with the new instructions at 128, 256, and 512 bit widths.

There is some interface complexity with the (as of this writing) 17 AVX-512 subsets since Knights Landing/Mill, Skylake, Cannon Lake, Cascade Lake, Cooper Lake, and Ice/Tiger Lake all support different variations. To me, it seems most natural to deprioritize support for the Knights (they're no longer in production, so presumably nearly all code targeting them has already been written) and implement something in the direction of

class Avx512FCD : Avx2 // minimum common set across all Intel CPUs with AVX-512
class Avx512VLDQBW : Avx512FCD // common set for enabled Skylake μarch cores and Sunny Cove

plus non-inheriting classes for BITALG, IMFA52, VBMI, VBMI2, VNNI, BF16, and VP2INTERSECT (the remaining four subsets—4FMAPS, 4NNIW, ER, and PF—are specific to Knights). This is similar to the existing model for Bmi1, Bmi2, and Lzcnt and aligns to current hardware in a way which composes with existing inheritance and IsSupported properties. It also helps with incremental roll out.

Finding naming for code readability that's still clear as to which instructions are available where seems somewhat tricky. Personally, I'd be content with idioms like

using Avx512 = System.Runtime.Intrinsics.X86.Avx512VLDQBW; // loose terminology

but hopefully others will have better ideas.

Tagging subscribers to this area: @tannergooding
Notify danmosemsft if you want to be subscribed.

[Symbai]

#8264 & #31420 but looks like a tracking issue is still missing.

[tannergooding]

There isn't an explicit tracking issue right now.

AVX-512 represents a significant investment as it nearly triples the current surface area (from ~1500 APIs to ~4500 APIs). It additionally adds a new encoding, additional registers that would require support (this is extending to 512 bits, supporting 16 more registers, and adding 8 mask registers), a new SIMD type (TYP_SIMD64 and Vector512<T>), and more. While this support could be added piece by piece, I'm not sure if this meets the bar for trying to drive through API review any time soon (.NET 5) and so I won't have time to create the relevant API proposals, etc. I do imagine that will change as the hardware support starts becoming more prevalent and the scenarios it can be used and will be beneficial increases.

If someone does want to create a rough proposal of what AVX-512F would look like (since that is the base for the rest of the AVX-512 support), then I'd be happy to provide feedback and continue the discussion until it does bubble up.

CC. @CarolEidt, @echesakovMSFT, @BruceForstall as they may have additional or different thoughts/opinions

[twest820]

Totally agree. Visual C++'s main AVX-512 roll out seems to have spanned the entire Visual Studio 2017 lifecycle and is still receiving attention in recent VS 2019 updates. It seems to me an initial question here could be what an AVX-512 roadmap might look like across multiple .NET annual releases. In the meantime, there is the workaround of calling intrinsics from C++, C++ from C++/CLI, and C++/CLI from C#. But I wouldn't have opened this issue if that layering was a great developer experience compared to intrinsics from C#. :-)

+3000 APIs is maybe ultimately on the low side. My current scrape of the Intel Intrinsics Guide lists 4255 AVX-512 intrinsics and 540 instructions. Only 380 of the intrinsics are not in the F+CD+VL+DQ+BW group supported from Skylake-SP and X and Ice Lake supports 4124 of the 4255 (give or take errors in the Guide I haven't caught or just on my part). Depending how exactly AVX-512F is defined I count it as totaling either 1435 or 2654 intrinsics. So it might make more sense to try to start with the initial 1500 intrinsics prioritized for Visual C++ 2017. Or even some subset thereof. I don't have that list, though.

Within this context, @tannergooding, if you can give me some more definition of what you're looking for in an AVX-512F sketch I can probably put something together.

I touched on this in #226, but the ability to jit existing 128 and 256 bit System.Runtime.Intrinsics.X86 APIs to EVEX for access to zmm registers 16-31 would be a valuable minimum increment even if not headlined by the addition of an Avx512 class. Definitely for most of the kernels in the various numerical codes I've written and perhaps also for the CLR's internal use of SIMD. (I can suggest some other pragmatically minded clickstops if there's interest.)

[tannergooding]

At the most basic level, there would need to be a Vector512<T> type that mirrors the Vector64/128/256 types and a new Avx512F class to contain the methods.

The methods proposed would likely have signatures like the following at a minimum (essentially mirroring SSE/AVX, but extending to V512):

/// <summary>
/// __m512d _mm512_add_pd (__m512d a, __m512d b);
///   VADDPD zmm, zmm, zmm/m512
/// </summary>
public static Vector512<double> Add(Vector512<double> left, Vector512<double> right)

On top of that minimum, there would need to be a proposal for a new x86 specific Mask8 register and overloads using the mask would need to be provided in order to fully support the EVEX encoding:

/// <summary>
/// __m512d _mm512_mask_add_pd (__m512d s, __mmask8 k, __m512d a, __m512d b);
///   VADDPD zmm, zmm, zmm/m512
/// </summary>
public static Vector512<double> Add(Vector512<double> value, Mask8 mask, Vector512<double> left, Vector512<double> right); // This overload merges values not written to by the mask

/// <summary>
/// __m512d _mm512_maskz_add_pd (__mmask8 k, __m512d a, __m512d b);
///   VADDPD zmm, zmm, zmm/m512
/// </summary>
public static Vector512<double> Add(Mask8 mask, Vector512<double> left, Vector512<double> right); // This overload zeros values not written to by the mask

EVEX additionally has support for broadcast versions which take right as a T* and broadcast the value to all elements of the V512, but I'm not sure those are explicitly needed and warrant further discussion. I imagine the JIT could recognize a Vector128.Create(value) call and optimize it to generate the ideal code (noting C++ does similar).

EVEX additionally has support for rounding versions which take an immediate that specifies the rounding behavior done for the given operation. This would likewise require some additional thought and consideration.

Then there are 128-bit and 256-bit versions for most of these, but they fall under AVX512VL which would require its own thought into how to expose. My initial thought is that we would likely try to follow the normal hierarchy, but where something inherits from multiple classes we would need additional consideration in how they get exposed. This would require at least a breakdown of what ISAs exist and what their dependencies are.

[john-h-k]

EVEX additionally has support for rounding versions which take an immediate that specifies the rounding behavior done for the given operation. This would likewise require some additional thought and consideration.

Does this mean rounding immediates on operations like add/sub rather than explicit rounding instructions?

[tannergooding]

No, the rounding instructions convert floats to integrals, while the rounding mode impacts the returned result for x + y (for example).

IEEE 754 floating-point arithmetic is performed taking the inputs as given, computing the "infinitely precise result" and then rounding to the nearest representable result.
When the "infinitely precise result" is equally close to two representable values, you need a tie breaker to determine which to choose.
The default tie breaker is "ToEven", but you can (not .NET, but in other languages or in hardware) set the rounding mode to do something like AwayFromZero, ToZero, ToPositiveInfinity, or ToNegativeInfinity instead.
EVEX supports doing this on a per operation basis without having to use explicit instructions to modify and restore the floating-point control state

[john-h-k]

Ah, brilliant, I think that is what I meant but I didn't word it great 😄

[scalablecory]

I would appreciate the "compare into mask" instructions in AVX-512BW to speed up parsing and IndexOf.

[saucecontrol]

there would need to be a proposal for a new x86 specific Mask8 register and overloads using the mask would need to be provided in order to fully support the EVEX encoding:

For the new mask and maskz instruction variants, couldn't the JIT recognize VectorNNN<T>.Zero in the value arg and use the zero source encoding? That would mean only doubling the API surface area instead of tripling it 😄

[tannergooding]

Yes, there are likely some tricks we can do to help limit the number of exposed APIs and/or the number of APIs we need to review.

[twest820]

This would require at least a breakdown of what ISAs exist and what their dependencies are.

Dependencies exist only on F and VL and seem unlikely to be concerns on Intel hardware. Probably not AMD either if they implement AVX-512. It seems github doesn't support tables in comments so I made a small repo with details.

At the most basic level, there would need to be a Vector512 type that mirrors the Vector64/128/256 types and a new Avx512F class to contain the methods.

Actually, if I had to pick just one width for initial EVEX and new intrinsic support it'd be 128.

there would need to be a proposal for a new x86 specific Mask8

Also 16, 32, and 64 bit masks. And rounding, comparison, minmax, and mantissa norm and sign enums. The BF16 subset planned for Tiger Lake would require #936 but that's unimportant at this point. I'll see about getting something sketched, hopefully in the next week or so.

[tannergooding]

Dependencies exist only on F and VL and seem unlikely to be concerns on Intel hardware

It is a bit more in depth than this...

ANDPD for example depends on AVX512DQ for the 512-bit variant. The 128 and 256-bit variant depend on both AVX512DQ and AVX512VL.
Since this has two dependencies, it can't be modeled using the existing inheritance hierarchy.

Now, given how VL works, it might be feasible to expose it as the following:

public abstract class AVX512F : ??
{
    public abstract class VL
    {
    }
}

public abstract class AVX512DQ : AVX512F
{
    public abstract class VL : AVX512F.VL
    {
    }
}

This would key off the existing model we have used for 64-bit extensions (e.g. Sse41.X64) and still maintains the rule that AVX512F.VL.IsSupported means AVX512F.IsSupported, etc.
There are also other considerations that need to be taken into account, such as what AVX512F depends on (iirc, it is more than just AVX2 and also includes FMA, which needs to be appropriately exposed).

Actually, if I had to pick just one width for initial EVEX and new intrinsic support it'd be 128.

I think this is a non-starter. The 128-bit EVEX support is not baseline, it (and the 256-bit support) is part of the AVX512VL extension and so the AVX512F class would need to be vetted first.