btl/vader performance regression (that can be observed on fat nodes)

[ggouaillardet]

Following the discussion on the ML related to a performance regression between Open MPI 3.1.1 and 4.1.0,
here are my findings:

The performance regression can be observed on fat nodes with the fftw builtin benchmark and the following command line

mpirun --map-by core --rank-by core --bind-to core --mca pml ob1 --mca btl vader,self  ./mpi-bench -opatient -r1000 -s icf1000000

git bisect pointed me to 7c8a1fb437

I ran this on a quad socket intel node with 18 cores per socket.
At 112 cores, the degradation is over 15%

If I read the commit message between the lines, the workaround that is needed for gcc < 6 was applied on all x86_64 based on the assumption it does not introduce any performance penalty.
The data above show otherwise, so we might want to improve this, for example by only applying the workaround for gcc < 6, or by not applying it for gcc >= 6 (depending on how we want to treat non GNU compilers)

@hjelmn could you please comment on this issue?

[BiplabRaut]

Although originally I reported the performance regression between Open MPI 3.1.1 and 4.1.0 when using GCC compiler, now I have found that AMD's AOCC (based on llvm/clang version 12) compiler also has this regression.
So, I am just wondering if there is more than just this commit 7c8a1fb that is causing this regression.

On a separate topic, it is also found by me that openMPI v4.1.0 is not getting compiled with llvm/clang - I checked with the compiler version 12. There are errors related to AVX512 ISA support. Does openMPI4.1.0 by default try to compile AVX512 without checking compiler support flags?
Older version openMPI4.0.5 could be compiled with llvm/clang.

[ggouaillardet]

the workaround for gcc<6 bug is applied on x86_64 regardless the compiler that is used, so I am not surprised there is a similar regression with AOCC.

several changes were made to op/avx (that can use AVX512 instructions), so I strongly encourage you to try the latest 4.1.1rc1 and see if it helps.
If not, please open a new issue with the required information.

[hjelmn]

clang also exposes gcc built-ins. So, this is expected. Someone should just change the problematic code to match the x86_64 atomics (just use asm memory clobber) when on that architecture and PR it to v4.0.x and v4.1.x. I can review but I don't have the time right now to test the fix as my primary Open MPI development machine is not running x86.