AOS/SOA for input particle 4-momenta (and random numbers)

[roiser]

Done, including AOSOA (AV)

eemumu_AV/master

Steered by typedefs. We need to go to the end with this idea.

Should go back and time AOS… actually AOSOA5.0E8, AOS4.8E8

[valassi]

Actually I have timed this yesterday, and AOSOA does seem to pay off, even if not by much.
See https://docs.google.com/document/d/1g2xwJ2FsSlxHvSUdPZjCyFW7zhsblMQ4g8UHlrkWyVw/edit#

Description	CUDA tput
BASELINE	5.00E8/s
SOA	4.65E8/s speed -7%
AOS	4.85E8/s speed -3%

I would say that we should keep some AOSOA-like structure as baseline

To do in any case

• understand better what causes the performance improvement (memory access via coalescing? SIMD vectorized computations??)
• massive clean up of the code, should use the same array dimensions everywehere: goal should be AOSOA[ngpublocks][nparticles][np4=4(E,px,py,pz)][ngputhreadsinblock]... also the random number arrays should use ngputhreads as last dimnesions (now it is hardoded as 32, the number if threads in a warp)

[valassi]

Some food for thought from Vincenzo, found by chance from google "godbolt cuda soa"
https://indico.cern.ch/event/851670/contributions/3585184/

[valassi]

And another interesting talk at CERN from the same google for godbolt
https://indico.cern.ch/event/932905/contributions/3920347/

[valassi]

I would like to do some changes to the AOSOA, but first I's like to understand a bit better if/why the layour makes a difference. I can think of two things, memory coalescing, and instruction vectorization. So I am doing some research to understand which metrics are relevant, eg in the profiler.

About memory coalescing, google "nvidia nsight coalesce" brought me here: https://developer.nvidia.com/blog/using-nsight-compute-to-inspect-your-kernels. This is avery interesting article pointing to two metrics, and showing how to focus on them in the tools. I added them here 0429183

I have then profiled the BASELINE ASA against AOS and SOA. At face value I confiorm that SOA is similar, just ~2% slower than ASA, while AOS is quite a bit slower, around 7-10%. The profiles are very interesting.

First, looking at the metrics in the blog above, indeed the number of requests for memory in sgnakin is the same in ASA and AOS, but the number of sectors (transactions) is a factor 4 higher with AOS than with ASA. This is a clear indication that the memory (the allmomenta memory) is not coalesced in AOS and is better in ASA. The distinction between ASA and SOA is much more subtle and unclear, the numbers of requests and sectors is lower in SOA than ASA, with a ratiob etween them that remains comparable. Note also that the ratio is around 6, while the optimale should be around 4?

This is ASA

This is SOA

This is AOS

Second, SOA has a higher number of registers that ASA (182 against 152) and this may be the reason for a penalty elsehwere, eg in the number of active warps. Maybe this explains the slightly lower throughput of SOA, maybe not. TThis is SOA compared to ASA

[valassi]

I have made many studies related to this and I will dump a few results an ddecisions here.

First, I cleanly spearated in the code the ASA structure for random (based on neppR) and for momenta (based on neppM). In issue #23 I studied the impact of doing a compile time vs a runtime choice for ASA parameters. The difference is small but visible, at the levl of 5% throughput for neppM, and visble in other profiler parameters.

[valassi]

Next, I studied whether AOS is any different from ASA with neppM=1. They are actually almost identical. There are differences below 1% in some metrics and I am not even sure why.

This is an overview

This is the warp state

This is the instruction mix

The #registers are essentially the same.

Conclusion: I will drop AOS from the code to make it simpler. This can always be recovered by setting neppM=1 in the ASA option (and indeed it is interesting for some studies, see later).

[valassi]

Then, I compare AOSOA (in my default with neppM=32) to SOA (which is essentially AOSOA with a much larger neppM=16384*32). The latter is worse in all relevant metrics an ddoes give a lower throughput. I am not sure why, but the issues seem to come from a much larger number of registers (operations not vectorized??). Anyway, even from first principles this is not really a sound choice.

Overview

Memory

Compute workload

Scheduler and stalls

Insruction mix

Conclusion: I will finally drop also AOS. I will then concentrate only on AOSOA. This will allow a much cleaner code. The relevant parameter in any case is neppM (smaller as 1 means AOS, larger as ndim means SOA) and various options can still be studied.

[valassi]

Coming now to the optimal nepp size in AOSOA. So far I am using a default of 32, which I included as this is the number of threads in a warp. I am instead getting the idea that 4 is a better size for our doubles (8 bytes), because the cache lines are 32 bytes and fit 4 doubles. (Probably 8 would be better for floats).

To start with, at face value an ASA32 vs ASA04 (where npeeR=neppM=32 or 4). The throughput is marginally better. Especially, the FP64 seems a bit better utilized, and in the end that's the only thing we do: compute a lot of FP64, so we should do that as fast as possible.

Note that the memory difference comes from a large increase in data pipe LSU wavefront, but I am not sure that's any good in itself. I would say 4 is better than 32 because it takes overall a bit less time, and FP64 is better (both only at the level of 1%).

I also compared ASA04 to ASA08, but the latter is not better, and maybe slightly worse, for doubles.

Instead, ASA01, ie AOS, is very interesting. This is clearly worse than ASA32 (or ASA04). The big difference here is that the number of requests is the same, but the number of sectors is a factor 4 higher: this is an indication of non-coalesced memory access.

For the record I also tried ASA02, but clearly this needs twice as many transactions as ASA04.

Note that indeed for FLOAT the minimum top get fully coalesced access is ASA08 (8 floats of 4 bytes are a cache line of 32 bytes). Using ASA04 there results in twice the number of transactions as one could use.

Conclusion: in other words, we need at least ASA04 (double) or ASA08 (float) to have coalesced memory access. We do not seem to get anything better by using higher nepp. I will set the new defaults.