OpenMPI may pick up an old version of mpi.mod from an existing installation for building

[marmistrz]

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

8b424c3

Please describe the system on which you are running

• Operating system/version: Ubuntu 18.04
• GCC/GFortran: 7.4.0

---

Details of the problem

OpenMPI was configured as follows:

./autogen.pl
./configure --enable-hwloc-pci --with-verbs --with-ucx

Then the compilation fails with:

  PPFC     cancel_f08.lo
  PPFC     cart_coords_f08.lo
  PPFC     cart_create_f08.lo
  PPFC     cartdim_get_f08.lo
cart_create_f08.F90:12:21:

    use :: mpi, only : PMPI_Cart_create
                     1
Error: Symbol ‘pmpi_cart_create’ referenced at (1) not found in module ‘mpi’
Makefile:4362: recipe for target 'cart_create_f08.lo' failed
make[2]: *** [cart_create_f08.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/home/marcin/build/openmpi/ompi/mpi/fortran/use-mpi-f08'
Makefile:3496: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/marcin/build/openmpi/ompi'
Makefile:1843: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

the configuration details printed by configure:

This happened because there was still a leftover from the previous installation, /usr/local/include/mpi.mod. It was picked up by make and used for the build of the new version. This is probably a bug, after purging this file the compilation succeeded.

[jsquyres]

What should the Open MPI build do differently, if an mpi.mod is in the compiler/linker default search path?

I'm open to suggestions.

[marmistrz]

In C/C++ there's #include "..." vs #include <...>, where the former prefers local includes over include path. I don't know if there's anything like that for Fortran. Or possibly the local path could be prepended to the module search path, if something like that exists in Fortran.

[jeffhammond]

Here are some relevant ifort docs:
https://software.intel.com/en-us/fortran-compiler-developer-guide-and-reference-using-include-files

You can use compiler option I to indicate the location of include files (and also module files).
To prevent the compiler from searching the default path specified by the CPATH or the INCLUDE environment variable, use compiler option -X or /noinclude.
You can specify these options in the configuration file, ifort.cfg, or on the command line.

However, these do not help if Open-MPI explicitly looks in /usr/local/include, as it does for me:

ifort -DHAVE_CONFIG_H -I. -I../../../../../ompi/mpi/fortran/mpiext-use-mpi \
-I../../../../opal/include -I../../../../ompi/include -I../../../../oshmem/include \
-I../../../../opal/mca/hwloc/hwloc201/hwloc/include/private/autogen \
-I../../../../opal/mca/hwloc/hwloc201/hwloc/include/hwloc/autogen \
-I../../../../ompi/mpiext/cuda/c   -I../../../../.. -I../../../.. -I../../../../../opal/include \
-I../../../../../orte/include -I../../../../orte/include -I../../../../../ompi/include \
-I../../../../../oshmem/include -I/Users/jrhammon/Work/MPI/OMPI/github/build/opal/mca/event/libevent2022/libevent/include \
-I/Users/jrhammon/Work/MPI/OMPI/github/opal/mca/event/libevent2022/libevent \
-I/Users/jrhammon/Work/MPI/OMPI/github/opal/mca/event/libevent2022/libevent/include \
-I/Users/jrhammon/Work/MPI/OMPI/github/build/opal/mca/hwloc/hwloc201/hwloc/include \
-I/Users/jrhammon/Work/MPI/OMPI/github/opal/mca/hwloc/hwloc201/hwloc/include   \
-I/usr/local/include -I/usr/local/include -I../../../../ompi/include -I../../../../../ompi/include \
-I../../../../ompi/mpi/fortran/base -I../../../../ompi/mpi/fortran/use-mpi-ignore-tkr \
-I../../../../.. -X  -c -o mpi-ext-module.lo mpi-ext-module.F90

I do not know why Open-MPI needs to add this include path for Fortran since there is nothing in this folder that should ever be required for compiler Fortran code. All the build-in modules are already known to the Fortran compiler and the third-party modules are, of course, the ones one absolutely should not get from such a path.

A solution that may be less intrusive is to place -I../../../../ompi/mpi/fortran/use-mpi-ignore-tkr before -I/usr/local/include when the compiler is invoked. This is working for me. I can get the same effect by removing -I/usr/local/include from CPPFLAGS in all the makefiles in the Fortran directories. Why the Fortran compiler is invoked with CPPFLAGS is beyond me. The C in CPPFLAGS stands for C.

[jeffhammond]

@jsquyres In short, Open-MPI should modify CPPFLAGS to not include the system header path when compiling Fortran, or it should not use CPPFLAGS at all and create a new variable like FPPFLAGS that only includes the right paths (ones inside the Open-MPI source tree).

[jsquyres]

@jeffhammond A few things:

1. We have logic for not adding /usr/include to include directories, but I don't think we have that same kind of login for /usr/local/include. More specifically: we don't have logic looking for the specific default include paths for the compiler being used -- just not adding /usr/include (and /usr/lib[64]) is the heuristic that we use.
2. I can reproduce your issue.
3. Looking deeper, it looks like the Automake definition for compiling Fortran includes the use of CPPFLAGS:

PPFCCOMPILE = $(FC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
        $(AM_CPPFLAGS) $(CPPFLAGS) $(AM_FCFLAGS) $(FCFLAGS)
LTPPFCCOMPILE = $(LIBTOOL) $(AM_V_lt) --tag=FC $(AM_LIBTOOLFLAGS) \
        $(LIBTOOLFLAGS) --mode=compile $(FC) $(DEFS) \
        $(DEFAULT_INCLUDES) $(INCLUDES) $(AM_CPPFLAGS) $(CPPFLAGS) \
        $(AM_FCFLAGS) $(FCFLAGS)

I wonder why that is...? 🤷‍♂

@jeffhammond @marmistrz That being said, does this patch obviate the issue for you:

diff --git a/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am b/ompi/mpi/fortran/mpiext-use-mpi
index b8318ce93c..9cec237718 100644
--- a/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am
+++ b/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am
@@ -15,6 +15,9 @@
 # the mpi ext modules.
 #
 
+CPPFLAGS =
+AM_CPPFLAGS =
+
 if OMPI_BUILD_FORTRAN_USEMPI_OR_USEMPIF08_EXT
 
 # Setup

[jeffhammond]

@jsquyres That patch fixes this bug.

Edit: it might create a new one. Please stand by for details.

[jeffhammond]

With the patch, I get this error:

Making all in mpi/fortran/use-mpi-ignore-tkr
  GENERATE mpi-ignore-tkr-sizeof.h
  GENERATE mpi-ignore-tkr-sizeof.f90
  PPFC     mpi-ignore-tkr.lo
  FC       mpi-ignore-tkr-sizeof.lo
  FCLD     libmpi_usempi_ignore_tkr.la
Making all in mpi/fortran/mpiext-use-mpi
  PPFC     mpi-ext-module.lo
mpi-ext-module.F90(7): #error: can't find include file: ompi/mpi/fortran/configure-fortran-output.h
mpi-ext-module.F90(32): #error: can't find include file: ../ompi/mpiext/pcollreq/mpif-h/mpiext_pcollreq_mpifh.h
mpi-ext-module.F90(33): #error: can't find include file: ../ompi/mpiext/pcollreq/use-mpi/mpiext_pcollreq_usempi.h
mpi-ext-module.F90(38): #error: can't find include file: ompi/mpiext/shortfloat/mpif-h/mpiext_shortfloat_mpifh.h
mpi-ext-module.F90(39): #error: can't find include file: ../ompi/mpiext/shortfloat/use-mpi/mpiext_shortfloat_usempi.h
make[2]: *** [mpi-ext-module.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

[jeffhammond]

@jsquyres This fixes the issue. I doubt it is the minimal change solution.

diff --git a/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am b/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am
index 9cec237718..9585c019c9 100644
--- a/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am
+++ b/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am
@@ -15,8 +15,8 @@
 # the mpi ext modules.
 #
 
-CPPFLAGS =
-AM_CPPFLAGS =
+CPPFLAGS = -I$(top_srcdir) -I$(top_builddir) -I$(top_srcdir)/opal/include -I$(top_srcdir)/orte/include -I$(top_builddir)/orte/include -I$(top_srcdir)/ompi/include
+AM_CPPFLAGS = -I$(top_srcdir) -I$(top_builddir) -I$(top_srcdir)/opal/include -I$(top_srcdir)/orte/include -I$(top_builddir)/orte/include -I$(top_srcdir)/ompi/include
 
 if OMPI_BUILD_FORTRAN_USEMPI_OR_USEMPIF08_EXT
 

The problem now appears in the MPI F08 module, so a similar solution will need to be applied there.

[jeffhammond]

I do not understand this one.

Input

$ git diff master
diff --git a/ompi/mpi/fortran/mpiext-use-mpi-f08/Makefile.am b/ompi/mpi/fortran/mpiext-use-mpi-f08/Makefile.am
index 616982611a..084784175e 100644
--- a/ompi/mpi/fortran/mpiext-use-mpi-f08/Makefile.am
+++ b/ompi/mpi/fortran/mpiext-use-mpi-f08/Makefile.am
@@ -15,6 +15,9 @@
 # the mpi_f08 ext modules.
 #
 
+CPPFLAGS = -I$(top_srcdir) -I$(top_builddir) -I$(top_srcdir)/opal/include -I$(top_srcdir)/orte/include -I$(top_builddir)/orte/include -I$(top_srcdir)/ompi/include
+AM_CPPFLAGS = -I$(top_srcdir) -I$(top_builddir) -I$(top_srcdir)/opal/include -I$(top_srcdir)/orte/include -I$(top_builddir)/orte/include -I$(top_srcdir)/ompi/include
+
 if OMPI_BUILD_FORTRAN_USEMPI_OR_USEMPIF08_EXT
 
 # Setup
diff --git a/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am b/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am
index b8318ce93c..9585c019c9 100644
--- a/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am
+++ b/ompi/mpi/fortran/mpiext-use-mpi/Makefile.am
@@ -15,6 +15,9 @@
 # the mpi ext modules.
 #
 
+CPPFLAGS = -I$(top_srcdir) -I$(top_builddir) -I$(top_srcdir)/opal/include -I$(top_srcdir)/orte/include -I$(top_builddir)/orte/include -I$(top_srcdir)/ompi/include
+AM_CPPFLAGS = -I$(top_srcdir) -I$(top_builddir) -I$(top_srcdir)/opal/include -I$(top_srcdir)/orte/include -I$(top_builddir)/orte/include -I$(top_srcdir)/ompi/include
+
 if OMPI_BUILD_FORTRAN_USEMPI_OR_USEMPIF08_EXT
 
 # Setup

$ git clean -dfx && ./autogen.pl && mkdir build && cd build && ../configure FC=ifort CC=clang CXX=clang++ --enable-mpi-fortran=all && make

Output

Making all in mpi/fortran/use-mpi-f08/bindings
  PPFC     ompi-mpifh-bindings.lo
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(216): error #7013: This module file was not generated by any release of this compiler.   [MPI_F08_TYPES]
   use :: mpi_f08_types, only : MPI_Status
----------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(218): error #6406: Conflicting attributes or multiple declaration of name.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(IN) :: status
--------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(214): error #6404: This name does not have a type, and must have an explicit type.   [STATUS]
subroutine ompi_get_count_f(status,datatype,count,ierror) &
----------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(216): error #6580: Name in only-list does not exist or is not accessible.   [MPI_STATUS]
   use :: mpi_f08_types, only : MPI_Status
--------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(281): error #6580: Name in only-list does not exist or is not accessible.   [MPI_STATUS]
   use :: mpi_f08_types, only : MPI_Status
--------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(285): error #6457: This derived type name has not been declared.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(OUT) :: status
--------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(279): error #6404: This name does not have a type, and must have an explicit type.   [STATUS]
subroutine ompi_probe_f(source,tag,comm,status,ierror) &
----------------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(291): error #7013: This module file was not generated by any release of this compiler.   [MPI_F08_TYPES]
   use :: mpi_f08_types, only : MPI_Status
----------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(297): error #6406: Conflicting attributes or multiple declaration of name.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(OUT) :: status
--------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(289): error #6404: This name does not have a type, and must have an explicit type.   [STATUS]
subroutine ompi_recv_f(buf,count,datatype,source,tag,comm,status,ierror) &
----------------------------------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(291): error #6580: Name in only-list does not exist or is not accessible.   [MPI_STATUS]
   use :: mpi_f08_types, only : MPI_Status
--------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(353): error #6580: Name in only-list does not exist or is not accessible.   [MPI_STATUS]
   use :: mpi_f08_types, only : MPI_Status
--------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(361): error #6457: This derived type name has not been declared.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(OUT) :: status
--------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(351): error #6404: This name does not have a type, and must have an explicit type.   [STATUS]
                           recvcount,recvtype,source,recvtag,comm,status,ierror) &
------------------------------------------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(368): error #7013: This module file was not generated by any release of this compiler.   [MPI_F08_TYPES]
   use :: mpi_f08_types, only : MPI_Status
----------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(374): error #6406: Conflicting attributes or multiple declaration of name.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(OUT) :: status
--------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(366): error #6404: This name does not have a type, and must have an explicit type.   [STATUS]
                                   recvtag,comm,status,ierror) &
------------------------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(368): error #6580: Name in only-list does not exist or is not accessible.   [MPI_STATUS]
   use :: mpi_f08_types, only : MPI_Status
--------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(427): error #6580: Name in only-list does not exist or is not accessible.   [MPI_STATUS]
   use :: mpi_f08_types, only : MPI_Status
--------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(430): error #6457: This derived type name has not been declared.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(OUT) :: status
--------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(425): error #6404: This name does not have a type, and must have an explicit type.   [STATUS]
subroutine ompi_wait_f(request,status,ierror) &
-------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(436): error #7013: This module file was not generated by any release of this compiler.   [MPI_F08_TYPES]
   use :: mpi_f08_types, only : MPI_Status
----------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(440): error #6406: Conflicting attributes or multiple declaration of name.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(OUT) :: array_of_statuses(count)
--------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(436): error #6580: Name in only-list does not exist or is not accessible.   [MPI_STATUS]
   use :: mpi_f08_types, only : MPI_Status
--------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(434): error #6404: This name does not have a type, and must have an explicit type.   [ARRAY_OF_STATUSES]
subroutine ompi_waitall_f(count,array_of_requests,array_of_statuses,ierror) &
--------------------------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(446): error #6580: Name in only-list does not exist or is not accessible.   [MPI_STATUS]
   use :: mpi_f08_types, only : MPI_Status
--------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(451): error #6457: This derived type name has not been declared.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(OUT) :: status
--------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(444): error #6404: This name does not have a type, and must have an explicit type.   [STATUS]
subroutine ompi_waitany_f(count,array_of_requests,index,status,ierror) &
--------------------------------------------------------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(458): error #7013: This module file was not generated by any release of this compiler.   [MPI_F08_TYPES]
   use :: mpi_f08_types, only : MPI_Status
----------^
../../../../../../ompi/mpi/fortran/use-mpi-f08/bindings/mpi-f-interfaces-bind.h(464): error #6406: Conflicting attributes or multiple declaration of name.   [MPI_STATUS]
   TYPE(MPI_Status), INTENT(OUT) :: array_of_statuses(*)
--------^

[ggouaillardet]

@jeffhammond This is likely the same issue.

mpi_f08_types.mod is likely picked from /usr/local/include instead of $(top_builddir)/ompi/mpi/fortran/use-mpi-f08/mod/mpi_f08_types.mod

fwiw, here is the list of F08 modules generated by Open MPI (and that should be picked under $(top_builddir) instead of /usr/local
so your next step is to patch ompi/mpi/fortran/use-mpi-f08/bindings/Makefile.am

[jsquyres]

I filed a complete PR in #7265 so that we can properly talk about the code without posting a series of diffs.

@jeffhammond GitHub Pro Tip: if you put the word patch after the three single tick marks that indicate the beginning of a GitHub verbatim section, it'll syntax hilight the diff.