cuda: libmpi.so.0 links libcuda.so which causes link issues when cuda is not installed

[AboorvaDevarajan]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

ompi - master

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git clone

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

$ git submodule status
 e46276629146e13c2f08ee9f71ba8133b4a0a7d7 3rd-party/openpmix (v1.1.3-2960-ge462766)
 b6ddc6e36dc88d1599f130886c915909327b0b8d 3rd-party/prrte (dev-31155-gb6ddc6e)

---

Details of the problem

Build

./autogen.pl
./configure --prefix=`pwd`/ompi-install --disable-nvml --disable-gl --disable-man-pages --enable-mca-no-build=btl-uct --enable-mpi1-compatibility  --with-cuda=/usr/local/cuda 
make -j40 install

$ ldd `pwd`/ompi-install/libmpi.so.0 | grep cuda
        libcuda.so.1 => /lib64/libcuda.so.1 (0x00007f904f39b000)

Looks like libmpi.so is linking libcuda.so this causes linker issues when the same ompi build is used in a node without cuda support.

This change in behavior / dependency is recently introduced here:
#8740

https://github.com/open-mpi/ompi/blob/master/3rd-party/romio341/mpl/configure.ac#L990

[bwbarrett]

@ggouaillardet it looks like ROMIO has a different CUDA detection behavior than Open MPI. We'll need to account for this in our wrapper to ROMIO's configure script.

[gpaulsen]

@ggouaillardet do you have time to look at this?

[gpaulsen]

Thanks @janjust

[janjust]

@ggouaillardet Can you offer some input on this issue - seems to be this is a ROMIO cuda detection logic.

[ggouaillardet]

My apologies for the late reply, I failed to notice these notifications

Indeed, ROMIO (MPL to be more specific) detects CUDA and add it to its dependencies.

Did we change the default to --disable-dlopen in master and v5.0.x?
If not this is a separate issue.
Anyway, that means that libcuda.so ultimately get added to the list of libmpi.so dependencies.

A simple fix is to force MPL not to detect libcuda.so with the patch below

diff --git a/ompi/mca/io/romio341/configure.m4 b/ompi/mca/io/romio341/configure.m4
index 553649e426..e015728bab 100644
--- a/ompi/mca/io/romio341/configure.m4
+++ b/ompi/mca/io/romio341/configure.m4
@@ -79,10 +79,10 @@ AC_DEFUN([MCA_ompi_io_romio341_CONFIG],[
                         AS_IF([test ! -z $target], [io_romio341_flags="$io_romio341_flags --target=$target"])])
                    AS_IF([test "$enable_grequest_extensions" = "yes"],
                          [io_romio341_flags="$io_romio341_flags --enable-grequest-extensions"])
-                   io_romio341_flags_define="$io_romio341_flags FROM_OMPI=yes CC='$CC' CFLAGS='$CFLAGS -D__EXTENSIONS__' CPPFLAGS='$CPPFLAGS' FFLAGS='$FFLAGS' LDFLAGS='$LDFLAGS' --$io_romio341_shared-shared --$io_romio341_static-static $io_romio341_flags $io_romio341_prefix_arg --disable-aio --disable-weak-symbols --enable-strict --disable-f77 --disable-f90"
+                   io_romio341_flags_define="$io_romio341_flags FROM_OMPI=yes CC='$CC' CFLAGS='$CFLAGS -D__EXTENSIONS__' CPPFLAGS='$CPPFLAGS' FFLAGS='$FFLAGS' LDFLAGS='$LDFLAGS' --$io_romio341_shared-shared --$io_romio341_static-static $io_romio341_flags $io_romio341_prefix_arg --disable-aio --disable-weak-symbols --enable-strict --disable-f77 --disable-f90 ac_cv_lib_cuda_cuMemGetAddressRange=no"
                    AC_DEFINE_UNQUOTED([MCA_io_romio341_COMPLETE_CONFIGURE_FLAGS], ["$io_romio341_flags_define"], [Complete set of command line arguments given to ROMIOs configure script])
 
-                   io_romio341_flags="$io_romio341_flags FROM_OMPI=yes CC="'"'"$CC"'"'" CFLAGS="'"'"$CFLAGS -D__EXTENSIONS__"'"'" CPPFLAGS="'"'"$CPPFLAGS"'"'" FFLAGS="'"'"$FFLAGS"'"'" LDFLAGS="'"'"$LDFLAGS"'"'" --$io_romio341_shared-shared --$io_romio341_static-static $io_romio341_flags $io_romio341_prefix_arg --disable-aio --disable-weak-symbols --enable-strict --disable-f77 --disable-f90"
+                   io_romio341_flags="$io_romio341_flags FROM_OMPI=yes CC="'"'"$CC"'"'" CFLAGS="'"'"$CFLAGS -D__EXTENSIONS__"'"'" CPPFLAGS="'"'"$CPPFLAGS"'"'" FFLAGS="'"'"$FFLAGS"'"'" LDFLAGS="'"'"$LDFLAGS"'"'" --$io_romio341_shared-shared --$io_romio341_static-static $io_romio341_flags $io_romio341_prefix_arg --disable-aio --disable-weak-symbols --enable-strict --disable-f77 --disable-f90 ac_cv_lib_cuda_cuMemGetAddressRange=no"
 
                    opal_show_subtitle "Configuring ROMIO distribution"
                    OPAL_CONFIG_SUBDIR([3rd-party/romio341],

A cleaner fix would require to patch the (third party) ROMIO.
So if this is good enough, I will issue a PR and then backport it.

[janjust]

@gpaulsen @awlauria What do you guys think? I think it's a good fix for now, and we can revisit with an open issue down the road?

[jsquyres]

What does ROMIO use CUDA for -- will not using ROMIO's CUDA functionality be a problem? (I realize that OMPIO is the default, so this may not be an important issue, but it's still worth asking the question)

I'm not too wild about using an $ac_cv_* variable -- those are undocumented. At a minimum, we should probably add comments that we're specifically forcing ROMIO to not use CUDA by an undocumented mechanism.

[edgargabriel]

I don't know the precise answer for ROMIO, but in OMPIO we have the ability to read/write an unmanaged cuda buffer using MPI_File_read/write and friends, which requires the ability to recognize a device buffer. I would suspect its the same for ROMIO