Performance regression: 2.0.x branch degradation over 1.10 for MTLs

[matcabral]

This is a place holder to fix the performance regressions seen on 2.0.x branch with regards to 1.10 that is impacting MTLs (tested with OFI and PSM2). The degradation is mostly impacting latency in small messages sizes, with some impact in bw.

Building with:

./configure CFLAGS=-O3 --prefix=<install path> --with-libfabric=no
--with-psm2=/usr --disable-oshmem --with-devel-headers --disable-debug
--disable-mem-profile --disable-mem-debug

The below tests assume same system setup, only changing OMPI 1.10 for 2.0.x

Two ranks on different nodes running osu_latency over PSM2.
1 -10%
2 -10%
4 -12%
8 -12%
16 -10%
32 -11%
64 -8%
128 -10%
256 -12%
512 -10%
1024 -12%
2048 -5%
4096 -18%
8192 0%
16384 -6%
32768 -4%
65536 -5%
131072 -3%
262144 -2%
524288 8%
1048576 0%
2097152 0%
4194304 18%

Two ranks on same node running osu_latency over PSM2.
1 -19%
2 -16%
4 -16%
8 -16%
16 -19%
32 -4%
64 -6%
128 -6%
256 -4%
512 -6%
1024 -14%
2048 -31%
4096 -1%
8192 -8%
16384 -24%
32768 -18%
65536 -12%
131072 -6%
262144 -8%
524288 -5%

Two ranks on different nodes running osu_bw over PSM2.
1 -7%
2 -9%
4 -7%
8 -7%
16 -5%
32 -4%
64 -6%
128 -2%
256 -5%
512 -7%
1024 -7%
2048 -5%
4096 -2%
8192 -2%
16384 -1%
32768 2%
65536 0%
131072 1%
262144 0%
524288 0%
1048576 0%
2097152 0%
4194304 0%

[matcabral]

Some of the tests done:

• Latency comparison against 2.x looks similar impact as 2.0.x, same with master.

• BW comparison against 2.x looks similar impact as 2.0.x. However, master does not show such a negative impact.

• Confirmed no impact on BTLS (openib, vader runs better)

• Confirmed opal/timer: add code to check if rtdtsc is core invariant #2611 does not solve the problem.

• Branches and branch misses: osu_latency 2 ranks. (compare 1.10 against 2.0.x)
  Running with PSM2 over the wire, 2.0.x shows less number of branches but slightly more % of branch misses (0.09% to 0.13%)
  However, running PSM2 over its shm, numbers are the same. (with a more significant degradation than over the wire)
  Testing Vader btl, I see 2.0.x has fewer branches (25% less), more misses (0.39% to 0.57%), but better performance (ranging from 10% to 20%).

• Cache misses and minor faults: osu_latency 2 ranks in the same node (compare 1.10 against 2.0.x)
  PSM2 MTL (over psm internal shm): 2.0.x shows worse numbers: minor faults > 20%, cache misses > 30%.
  VADER BTL: 2.0.x shows significantly worse numbers minor faults > 29%, cache misses > 70%, but better benchmark performance.
  Therefore, Vader BTL still runs better regardless of the higher number of misses and faults.

[ggouaillardet]

@matcabral i noted you configure with
--with-psm2=/usr
we recommend against that (e.g. no /usr nor /usr/local)
what if you configure with
--with-psm2
if this does not work, then this is a bug and i will try to fix it

[jsquyres]

@hjelmn @bosilca Thoughts?

[bosilca]

I would assume this is dependent on #2448

[jsquyres]

@rhc54 @matcabral (cross reference) #2448 has been merged; it would be good to see how it affected the performance of the PSM / OFI MTLs.

[hppritcha]

I'm not seeing any noticeable difference in performance with the commits from #2448 applied to 2.0.x when using PSM2.

[bosilca]

This is extremely puzzling as this commit removes a global lock from the critical path for all multithreaded builds.

[matcabral]

(Catching up after vacations)
@ggouaillardet --with-psm2 works fine.
@bosilca as @hppritcha mentions, I also tested applying #2448 on 2.0.x makes not change on the latency degradation. Just to be on the same page, I'm not building multithreaded.