mpicc / mpirun / etc. carry unnecessary external dependencies

[bwbarrett]

Currently, Open MPI executable (like mpicc/mpirun/etc.) inherit dependencies on BTL communication libraries, like ugni. This leads to problems on some large scale systems, where compute not libraries are not in the default library search paths on the head nodes (I assume they must be available, otherwise linking of applications wouldn't work). For example:

mpicc --version
mpicc: error while loading shared libraries:
libugni.so.0: cannot open shared object file: No such file or directory

This has been a problem with Open MPI since the BTLs moved to OPAL, but is considerably more noticeable with the change to avoid building DSOs by default. #8800 proposed a fix by making components with external dependencies build as DSOs by default, but this defeats the entire reason we build without DSOs by default. Launch scalability with the old behavior was terrible because of the mass DSO loading at launch. The systems likely to run into the library dependency problem are the very ones that need the change in default behavior, and are likely to have many components with external dependencies.

The right solution is probably to move the BTLs back into the OMPI layer, but I assume @bosilca will object to that plan. A second plan, and likely the one we will have to implement, is to split OPAL into two libraries. The first is just the base portability code (with minimal MCA inclusion) that is safe to use on the front-end and the second is the full opal with communication libraries. We already have a bit of this split, in that we have two different initialization routines (opal_init() and opal_init_util()). We just don't expose that split through libraries, leading to linking problems.

[bosilca]

The BTLs are an easy target for discussion because they are obviously not necessary when not running an MPI job, but the issue remains true for many other shared libraries (hwloc, cuda).

Let me add a 3rd possible approach, implement the dependencies in OPAL and do lazy loading of all frameworks.

[bwbarrett]

The very reason that we made this change was to avoid dynamic loading to avoid killing large system filesystems. The very problem is lazy loading.

[bosilca]

Then I'm not sure I understand what the problem is. I would understand if everything is compiled statically, but it does not seem so, or the example above would not try to load libugni.so. Thus, as long as we depend on any way on a shared library there will be loading storm.

What I am proposing is a little like RTLD_LAZY, but at the level of the OPAL components. Thus, the BTL framework will not be initialized until some other component request it, which will result in less necessary dependencies for our wrappers. We can combine this with the allowing some components to declare themselves as DSO, and we will get the best of both world, no impact on mpicc and little impact on an MPI application.

Or maybe I'm missing some critical point in this entire discussion ?

[bwbarrett]

are you suggesting that we add all the infrastructure to dynamically load libugni? That would be a hell of an ask. Just look at how big of a pain it is to deal with CUDA. To do that with all dependencies seems insane.

In general, I think the right answer is to use the dso infrastructure we already have. However, a small set of components (the BTLs in OMPI, one or two components in PRRTE) cause problems because the large systems we're trying to make scale don't include some system libraries on the interactive nodes. I think that's a bit silly, but whatever. In the case of the utilities and OPAL, keep in mind that we already split the initialization code so that we don't initialize the BTLs for mpicc and friends. Solving the library inclusion problem isn't a huge reach from there.

[bosilca]

What I was proposing is to load OPAL frameworks only when needed. But after playing around a little bit, it does not seem like a satisfactory approach.

Thus, after going back and forth with @jsquyres, we might have found al alternative approach, using the libtool convenience libraries to create a libopen-tiny.la that would contain the minimum OPA support, aka. exactly what opal_init_util needs. More to come.

[rhc54]

Not sure I get how that solves anything. The 'mpirun' issue is a side one - what we are trying to do is eliminate backend loading of DSOs. Period. It doesn't scale. Brian is only noting that the problem also caused symptoms in mpicc, so the solution should address that too. Solving that problem by a method that results in DSOs being loaded again on the backend just takes us backward.

Just trying to avoid people spending time on the wrong problem.

[jsquyres]

@bosilca @bwbarrett and I are talking on the phone about this tomorrow -- we'll get it resolved.

[rhc54]

FWIW: I've started looking into changing the way we handle this in PMIx/PRRTE. Instead of linking against these dependencies at build, I'm building all components and letting the main library absorb them (which is now our default). I then use dlopen on the backend to see if the required lib is present - if not, disqualify the component. If so, then use dlsym to map the few APIs I actually need.

Eliminates this problem and helps the distros so they don't have to load a bunch of stuff just to build us.

🤷‍♂️ What you folks do in OMPI is up to you.