[feature request] packaging for multiple Fortran compilers

[jeffhammond]

Summary

Please support the ability to build Open-MPI Fortran bindings for multiple Fortran compilers in the same build. This will dramatically improve user experience.

I recognize this is not a small thing to implement, but at least two vendors do this already, and since some vendors use Open-MPI as the basis for their MPI product, it might be the sort of thing that makes sense for the project to support.

Alternative / Workaround

The current solution for multiple Fortran compilers is to build the MPI library once for every toolchain, e.g. one has installation directories open-mpi-intel15, open-mpi-intel16, open-mpi-gcc5, open-mpi-gcc6, etc. This naturally leads to mpi.mod and mpi_f08.mod for every Fortran compiler.

While simple, the current solution is a waste of sysadmin time, processor cycles, filesystem metadata, etc. given that one could instead build Open-MPI once with the preferred C compiler once and then support O(n) Fortran compiler wrappers and modules.

Future Workaround

In theory, there could be an implementation-agnostic implementation of the MPI Fortran 2008 bindings that could be compiled independent of the MPI C library, but such a project does not yet exist and in any case, very few users have adopted use mpi_f08.

Background

The Fortran bindings are largely if not completely independent of the internals of MPI, which are compiled with a C compiler, and C allows for a standard ABI. On the other hand, Fortran does not have a standard ABI and modules must be generated for every compiler vendor, and possibly also multiple (major) versions of the compiler from a single vendor.

Suggested implementation

1. Allow the user to specify FCn for n=1,2,... as an alternative to FC. The build system would then generate MPI Fortran scripts named mpi$FC1, mpi$FC2, etc., assuming FCn do not have the full paths, otherwise they will be stripped.
2. -enable-fortran-compilers=$FC1,$FC2. This is just different syntax for option 1.
3. If it is too hard to do it simply, require a configuration file that describes the settings for each Fortran compiler explicitly.

Ideally, the user can do something like $PATH_TO_OMPI/configure FC1=ifort FC2=gfortran and get $OMPI_PREFIX/bin/mpiifort, $OMPI_PREFIX/bin/mpigfortran, $OMPI_PREFIX/include/ifort/mpi.mod, $OMPI_PREFIX/include/gfortran/mpi.mod, $OMPI_PREFIX/include/ifort/mpi_f08.mod, $OMPI_PREFIX/include/gfortran/mpi_f08.mod, and any companion files.

Prior Engineering / Proof-of-Principle

Intel MPI

Intel MPI packages the wrappers and modules for Intel Fortran and GNU Fortran.

$ mpiifort -show
ifort -I/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include 
-I/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include 
-L/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib/release_mt 
-L/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib -Xlinker 
--enable-new-dtags -Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib/release_mt 
-Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib 
-Xlinker -rpath -Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt -Xlinker -rpath 
-Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib -lmpifort -lmpi -lmpigi -ldl -lpthread

$ mpifc -show
gfortran -I/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include/gfortran/5.1.0 
-I/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include 
-L/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib/release_mt 
-L/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib -Xlinker 
--enable-new-dtags -Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib/release_mt 
-Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib 
-Xlinker -rpath -Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt -Xlinker -rpath 
-Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib -lmpifort -lmpi -lmpigi -ldl -lpthread

$ mpif90 -show
gfortran -I/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include/gfortran/5.1.0 
-I/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include 
-L/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib/release_mt 
-L/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib -Xlinker 
--enable-new-dtags -Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib/release_mt 
-Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib 
-Xlinker -rpath -Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt -Xlinker -rpath 
-Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib -lmpifort -lmpi -lmpigi -ldl -lpthread

$ mpif77 -show
gfortran -I/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include 
-L/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib/release_mt 
-L/opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib -Xlinker 
--enable-new-dtags -Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib/release_mt 
-Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/lib 
-Xlinker -rpath -Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt -Xlinker -rpath 
-Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib -lmpifort -lmpi -lmpigi -ldl -lpthread

$ ll /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include
total 2.0M
drwxr-xr-x 11 jrhammon root 4.0K Mar 16 16:41 gfortran
drwxr-xr-x  3 jrhammon root 4.0K Mar 16 16:41 ilp64
-rw-r--r--  1 jrhammon root 5.3K Mar  2  2016 i_malloc.h
-rw-r--r--  1 jrhammon root 116K Mar  2  2016 mpi_base.mod
-rw-r--r--  1 jrhammon root  51K Mar  2  2016 mpi_constants.mod
-rw-r--r--  1 jrhammon root 103K Mar  2  2016 mpicxx.h
-rw-r--r--  1 jrhammon root  32K Mar  2  2016 mpi_f08_callbacks.mod
-rw-r--r--  1 jrhammon root  81K Mar  2  2016 mpi_f08_compile_constants.mod
-rw-r--r--  1 jrhammon root 8.6K Mar  2  2016 mpi_f08_link_constants.mod
-rw-r--r--  1 jrhammon root 645K Mar  2  2016 mpi_f08.mod
-rw-r--r--  1 jrhammon root  49K Mar  2  2016 mpi_f08_types.mod
-rw-r--r--  1 jrhammon root  23K Mar  2  2016 mpif.h
-rw-r--r--  1 jrhammon root 131K Mar  2  2016 mpi.h
-rw-r--r--  1 jrhammon root  11K Mar  2  2016 mpi.mod
-rw-r--r--  1 jrhammon root 4.1K Mar  2  2016 mpiof.h
-rw-r--r--  1 jrhammon root  26K Mar  2  2016 mpio.h
-rw-r--r--  1 jrhammon root 8.8K Mar  2  2016 mpi_sizeofs.mod
-rw-r--r--  1 jrhammon root 656K Mar  2  2016 pmpi_f08.mod

$ ll /opt/intel/compilers_and_libraries_2017.0.036/linux/mpi/intel64/include/gfortran/5.1.0
total 52K
-rw-r--r-- 1 jrhammon root  13K Mar  2  2016 mpi_base.mod
-rw-r--r-- 1 jrhammon root 7.9K Mar  2  2016 mpi_constants.mod
-rw-r--r-- 1 jrhammon root  23K Mar  2  2016 mpi.mod
-rw-r--r-- 1 jrhammon root 1.2K Mar  2  2016 mpi_sizeofs.mod

IBM Blue Gene/Q

IBM supported this on Blue Gene/Q. I do not know the implementation details but can demonstrate the results if anyone needs to see it.

[kawashima-fj]

Fujitsu MPI, which is based on Open MPI, also provides multiple compiler support unofficially (not general product).

The issue is not specific to Fortran (and removed C++). Even for C, OpenMP support is not ABI compatible. So Fujitsu MPI provides mpifcc/mpiFCC/mpifrt for Fujitsu compiler, mpigcc/mpig++/mpigfortran for GCC, and mpiicc/mpicpc/mpiifort for Intel compiler.

The implementation is not same as the one which @jeffhammond suggested. But if someone need this, I can contribute it to Open MPI.

User's view

All MPI wrapper compilers are installed in the same directory (/opt/FJSVpclang/bin). Users don't need to switch the PATH environment variable but need to switch command names.

All shared libraries have distinct names (e.g. libmpi_gnu.so) and installed in the same directory (/opt/FJSVpclang/lib64). Include files and module files are installed separately with the same file names (e.g. /opt/FJSVpclang/include/mpi/gnu/mpi.h). All are hidden by the wrapper compilers and users don't need to know the file names or directory paths.

mpiexec and other files are shared. When user executes a MPI program using mpiexec, appropriate libraries are linked automatically.

Developer's view

Developers need to run configure and make all install for each compiler product. On each configure run, --enable-extra-named-lib=COMPILER_NAME option must be supplied and CC/CXX/FC must be set appropriately. The --prefix option must be same.

Though this does not save sysadmin's time, but saves filesystem metadata.

[jeffhammond]

@kawashima-fj It sounds like Fujitsu MPI does what users are doing already, meaning recompiling the full library for every compiler toolchain. Did I misunderstand you?

I was hoping for an implementation that would only compile the Fortran interface faces for every Fortran compiler, but reuse the C object codes for all Fortran compilers, for a single C compiler.

OpenMP might break C ABI compatibility when used to build object files but this should not affect the compilation of C source files in Open-MPI because it does not use OpenMP constructs, nor does it include omp.h in any source file. Thus, one can build the vast majority of the Open-MPI object files with the system C compiler and end up with a C library that can be used with any compiler.

[kawashima-fj]

@jeffhammond You understood correctly. The mechanism I described is not a solution for you. It's slightly off topic. The merit of the mechanism is mainly for end users, not sysadmins.

My OpenMP description was not related to this topic, sorry. Please ignore it.

[ggouaillardet]

@jeffhammond i think we should decouple end user experience and sysadmin experience.

sysadmins can script openmpi build/install, and run several builds in parallel.
nowadays SSD can store multi TB of data, so as far as i am concerned, i'd rather focus on enduser experience.

an other nice feature to have is the use of a single fortran wrapper that picks the "right" modules and libs.

for example
mpifort foo.f90 picks the "standard" modules/libs
whereas the infamous
mpifort -i8 -d8 foo.f90 would rather pick the modules/libs compiled with 8 bytes integer and 8 bytes real
also, an option could switch between the libs that support MPI_THREAD__MULTIPLE and the libs that do not.

iirc, IntelMPI wrappers do a great job at that

about saving disk space, i am afraid you assume all fortran compilers work just fine with the GNU C compiler. though that is very likely true for the Intel Fortran compilers, PGI Fortran compilers do not work out of the box with GNU C compiler (and you can consider this is an Open MPI bug)
i am also skeptical about Intel C and PGI Fortran (and no, i do not have any plan to do this in the real life)

last but not least, an other nice to have feature is the ability to generate wrappers for both the native compiler and the (several ?) cross compilers

[jeffhammond]

@ggouaillardet It is a user experience issue if the sysadmin users experience is such that the latter does not build every case. Some sysadmins believe in validating libraries before exposing them to users. Because we have both observed that C correctness can depend on Fortran, at least when PGI is involved (e.g. http://trac.mpich.org/projects/mpich/ticket/1869), sysadmins need to validate the entire build, so building the entire library for N Fortran compilers forces the sysadmin to run N times as many tests. At least in the case where the C library correctness is not dependent on the Fortran compiler, my proposal reduces sysadmin effort.

You are right that switching between 32b and 64b INTEGER is a great feature for Fortran users, even though that feature of Fortran is itself horrible beyond imagination.

I don't see any reason for the wrapper to need to know about MPI thread support. That should be selectable at runtime the same as choosing a BTL/MTL.

It is great that all of your filesystems are infinitely large. I use both laptops and supercomputers where this is not the case.

[ggouaillardet]

if you might need MPI_THREAD_MULTIPLE, then you have to configure with --enable-mpi-thread-multiple, otherwise Open MPI is configured with default --disable-mpi-thread-multiple.
the latter library is a bit faster because some thread related stuff is #ifdef'ed out.
/* and there is some work to make the former library nearly as fast as the latter library */

iirc, Intel MPI does the same thing under the hood (well, if you use Open MP, then the MPI_THREAD_MULTIPLE library is selected, even if your app does not require MPI_THREAD_MULTIPLE)

we previously had a discussion of how Open MPI (or any MPI implementaion) should behave when configure'd with --disable-mpi-fortran with respect to Fortran predefined datatypes.
iirc, we kind of agreed that was a (dark) grey area since some assumptions had to be made
(e.g. do we simply assume MPI_INT is equivalent to MPI_INTEGER ?,
and as we all know, bad things happen when we start assuming things).
having said that, i do not think it is realistic to build the C libs with --disable-mpi-fortran and then build one or several Fortran lib(s) on top of that.
my point is if we build with one C compiler and several Fortran compilers, then we would have to check all Fortran compilers have the same "characteristics" from the Open MPI C library point of view.
an obvious mistake would be to run gfortran as is and ifort -i8 -d8, but there might be less obvious source of errors in the real life.

below are two additional though weaker reasons why i'd rather have multiple Open MPI install dir

the install dir of the latest master, built with -g and without --disable-dlopen (needless to say this is far from optimal from a disk usage point of view) takes 48 MB, which is several orders of magnitude lower than infinitely large or even a laptop SSD.

if i assume a sysadmin has already scripted the installation and validation process of Open MPI, then running several instances (one for each C&Fortran compiler) in parallel might require a shorter elapsed time than running one instance with all the Fortran compilers.

[ggouaillardet]

i made a proof-of-concept at #2070

here is how it can be used

./configure --with-mpi-fortran-dir=/intel --disable-oshmem CC=gcc CXX=g++ FC=ifort
make
make install
mv /usr/local/share/openmpi/mpifort-wrapper-data.txt /usr/local/share/openmpi/mpiifort-wrapper-data.txt
ln -s opal_wrapper /usr/local/bin/mpiifort
./configure --with-mpi-fortran-dir=/intel --disable-oshmem CC=gcc CXX=g++ FC=gfortran
make
make install

and then you can use mpifort if you want to use the GNU compiler, and mpiifort if you want to use the Intel compiler

note

• for the time being, you have to configure with --disable-oshmem because there is a single oshmem lib (and not two for C and Fortran)
• yes, you have to build twice (``configureandmake` can run in parallel if you use `VPATH` with two directories, just

make install
mv /usr/local/share/openmpi/mpifort-wrapper-data.txt /usr/local/share/openmpi/mpiifort-wrapper-data.txt
ln -s opal_wrapper /usr/local/bin/mpiifort

the Intel built library before you

make install

the GNU build library

• last but not least, it is your responsability to ensure GNU and Intel Fortran compilers are identical with respect to how Open MPI collects information in order to build the C library
• that mechanism cannot be used to automagically handle the infamous -i8 -d8 like option

Under the hood, when configured with --with-mpi-fortran-dir=/intel Fortran headers will be installed in /usr/local/include/intel and libs and modules will be installed in /usr/local/lib/intel

[ggouaillardet]

@jsquyres @gpaulsen would you mind raising this issue during today's telcon ?
Ideally, I'd like some more folks think about it and provide some directions on how to move forward.

[jjhursey]

As a note, IBM's Spectrum MPI provides GNU, PGI, and XL Fortran modules that work with the same core MPI library:

• https://www.ibm.com/docs/en/smpi/10.4?topic=applications-compiling