Feature/prefetch2

[maddyscientist]

This work is latest towards optimizing QUDA for Blackwell:

• Adds supports for "spatial prefetching", where we over fetch data to L2 when issuing a global load. Exposed as an optional template parameter to vector_load. At present, not deployed anywhere.
• Add support for prefetching instructions, in the form of both per-thread prefetching (which works on all CUDA architectures), and TMA-based prefetching, which is Hopper+ only. Prefetching type is set using QUDA_DSLASH_PREFETCH CMake parameter, with 0=per-thread, 1=TMA bulk, and 2=TMA descriptor
• Add an experimental L1 prefetch (using LDGSTS). Disabled, but left for future experiments.
• Add single-threaded execution region helper function target::is_thread_zero() which should be used for TMA issuance.
• Optionally store the backward shifted gauge field. This simplifies all dslash indexing, as all spatial indices thus correspond to "this" site. Enabled with QUDA_DSLASH_DOUBLE_STORE=ON which is required for TMA-based prefetching (for alignment reasons).* Prefetching is exposed for both ColorSpinorFields and GaugeFields, though only latter actually used at present.
• Added prefetching support to both Wilson and Staggered dslash kernels, parameterized using QUDA_DSLASH_PREFETCH_DISTANCE_WILSON and QUDA_DSLASH_PREFETCH_DISTANCE_STAGGERED CMake parameters.
• Optimization of the neighbor indexing for the dslash kernels. This reduced integer instruction overheads.
• Reduction in pointer arithmetic overhead (use more 32-bit integer operations where possible). Added three operand and four operand variants of vector_load and vector_store to this end (respectively).
• Optimize FFMA2 issuance to reduce total number of floating point instructions on Blackwell
• Optimization of short <-> float conversation to reduce instruction overheads
• Optimization of staggered packing kernels (replace division by int with division by fast_intdiv)
• Extends some OpenMP parallelization on the host that was missing.

The end result of this work is that both Staggered and Wilson dslash kernels can saturate over 90% memory bandwidth for most variants. Outstanding are half precision variants using reconstruction, that are still lagging. These will be the focus of a subsequent PR.

[maddyscientist]

Fix this with a specific tag