Question: How do HWY_NATIVE_* macros behave under dynamic dispatch

[pauxav06]

Hello,

I am trying to port code that uses VNNI / Neon Dotprod, and I am having troubles detecting support for these operations with dynamic dispatch.

I modified skeleton.cc / skeleton.h by adding the following lines to CodepathDemo:

#ifdef HWY_NATIVE_U8_I8_SUMOFMULQUADACCUMULATE
  const char* gather2 = "Has VNNI";
#else
  const char* gather2 = "No VNNI";
#endif

#ifdef HWY_NATIVE_U8_I8_SATWIDENMULPAIRWISEADD
  const char* gather3 = "Has Fallback";
#else
  const char* gather3 = "No Fallback";
#endif

printf("Target %15s: %15s %15s %15s %d\n",
       hwy::TargetName(HWY_TARGET),
       gather, gather2, gather3,
       HWY_TARGET <= HWY_AVX3_DL);

Then I replaced dynamic dispatch with:

#define VISITOR(TARGET, NAMESPACE) NAMESPACE::CodepathDemo();
HWY_VISIT_TARGETS(VISITOR)

(I also tried a SetSupportedTargetsForTest + HWY_DYNAMIC_DISPATCH loop for the same exact results)

The programm was compiled on with gcc, with O3 and no march flags enabed .

The output shows unexpected toggling of the macros:

Target            AVX2:       Has int64        Has VNNI    Has Fallback 0
Target            AVX3:       Has int64         No VNNI     No Fallback 0
Target         AVX3_DL:       Has int64        Has VNNI    Has Fallback 1
Target        AVX3_SPR:       Has int64        Has VNNI    Has Fallback 1
Target       AVX3_ZEN4:       Has int64         No VNNI     No Fallback 1
Target            SSE2:       Has int64         No VNNI     No Fallback 0
Target            SSE2:       Has int64         No VNNI     No Fallback 0
Target            SSE4:       Has int64         No VNNI     No Fallback 0
Target           SSSE3:       Has int64        Has VNNI    Has Fallback 0

It seems that the HWY_NATIVE_* macros are toggled on/off at each iteration of the dynamic dispatch codegen process:

1. SSE2 -> OFF
2. SSSE3 -> ON
3. SSE4 -> OFF
4. AVX2 -> ON
5. AVX3 -> OFF
6. AVX3_DL -> ON
7. AVX3_ZEN4 -> OFF
8. AVX3_SPR -> ON

However, testing other macros like HWY_NATIVE_FMA which are defined in set_macros-inl.h works perfectly!

Questions:

• Is this meant to work and did i mess something up, or are most HWY_NATIVE_* macros intended only for static dispatch?
• If I want to detect VNNI / Dotprod support reliably in a dynamically dispatched function, should I :
• • Use something like HWY_TARGET <= HWY_AVX3_DL?
• • Build static targets separately, aggregate via CMake, and probe dispatch function once to get the correct function table?
• • Should i not be trying to manually route kernels and instead let auto-tune decide for its-self what to do. For instance generate 3/4 kernels per target, even the ones using non native operations that will be very slow, then probe the target function pointers at startup and tune to find the most efficient one.
• • Am i way off topic and this is not the way to go at all?

Thank you in advance!

[johnplatts]

The SumOfMulQuadAccumulate and SatWidenMulPairwiseAccumulate ops were added as there are some non-x86 Google Highway targets with neural network SIMD instructions such as PPC8/PPC9/PPC10, NEON_BF16, and SVE/SVE2 in addition to AVX3_DL/AVX3_SPR/AVX10_2 that support the AVX512 VNNI instructions.

Here are the AVX512 VNNI intrinsics that have Google Highway equivalents (supported on all Google Highway targets other than HWY_SCALAR):

• _mm_dpbusd_epi32(sum, a, b) - equivalent to Google Highway SumOfMulQuadAccumulate(di32, a, b, sum)
• _mm_dpwssds_epi32(sum, a, b) - equivalent to Google Highway SatWidenMulPairwiseAccumulate(di32, a, b, sum)
• _mm_dpwssd_epi32(sum, a, b) is equivalent to Google Highway Add(sum, WidenMulPairwiseAdd(di32, a, b))

Clang 10 and later also already optimize Add(sum, WidenMulPairwiseAdd(di32, a, b)) down to a single vpdpwssd instruction on the AVX3_DL/AVX3_SPR/AVX10_2 targets when optimizations are enabled according to a snippet at https://godbolt.org/z/TcarbdWEd.

There are some targets where ReorderWidenMulAccumulate(di32, a, b, sum0, sum1) is implemented using Add(sum, WidenMulPairwiseAdd(di32, a, b, sum0)) or equivalents (such as AVX3_DL _mm_dpwssd_epi32(sum, a, b) or PPC vec_msum(a, b, sum))

[jan-wassenberg]

+1 to John's comments on how to generate the VNNI instructions. On the question of feature detection:

It seems that the HWY_NATIVE_* macros are toggled on/off at each iteration of the dynamic dispatch codegen process:

Yes, that is correct. Anything which is checked in the following way is for internal Highway use only:

#if (defined(HWY_NATIVE_U8_I8_SATWIDENMULPAIRWISEADD) == defined(HWY_TARGET_TOGGLE))

Unfortunately our naming convention does not distinguish between that and HWY_NATIVE_FMA, which is set by set_macros-inl.h explicitly for users to consult, and is also documented in quick_reference.

It is tempting to make decisions based on feature detection. We do that with HWY_NATIVE_DOT_BF16. Unfortunately, for int8 that is made much harder by the various signed/unsigned combinations. It's not clear whether (partial) emulation is faster than alternatives. Hence I'd vote for your auto-tuning idea to find that out at runtime, hwy/auto_tune.h can help with that. We use auto-tuning to good effect in gemma.cpp.