problem linking mpi memory routines from C++

[floquet-cxx]

Thank you for taking the time to submit an issue!

Background information

I am compiling an application (my own) which has C++ as the top level but also uses some C and F77 libraries. The build system is cmake on OSX via macports. The application is about 2 decades old now, and has successfully used various versions of MPI-1 for most of that time. Recently I added some new MPI memory and IO routines to do with Cartesian communicators and parallel IO, and have created a small set of linkage problems if using openmpi (they can however be linked successfully if I use mpich). The problem has persisted across two OS X versions, so I suspect the problem lies in openmpi code base. It has the flavour of a misplaced "extern" declaration in a header file.

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

4.1.6

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Installed via macports. (Of course this leaves open the possibility that the problem lies there...)

sudo port -N install  openmpi +gfortran
sudo port select --set mpi openmpi-mp-fortran

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version: MacOS Sonoma (14.2.1) + Xcode15
• Computer hardware: Mac Studio and Macbook Pro (ARM M2)
• Network type: SMP/single machine

---

Details of the problem

The build fails at the linkage stage, unable to find 4 low-level (not API-level) MPI routines called from various object files. Below is a clip of the relevant messages.

[ 53%] Linking CXX executable elliptic_mp
Undefined symbols for architecture arm64:
  "__ZN3MPI3Win4FreeEv", referenced from:
      __ZTVN3MPI3WinE in auxfield.cpp.o
      __ZTVN3MPI3WinE in data2df.cpp.o
      __ZTVN3MPI3WinE in domain.cpp.o
      __ZTVN3MPI3WinE in geometry.cpp.o
      __ZTVN3MPI3WinE in mesh.cpp.o
      __ZTVN3MPI3WinE in message.cpp.o
      __ZTVN3MPI3WinE in helmholtz.cpp.o
      ...
  "__ZN3MPI4CommC2Ev", referenced from:
      __ZNK3MPI9Intracomm5CloneEv in auxfield.cpp.o
      __ZNK3MPI9Graphcomm5CloneEv in auxfield.cpp.o
      __ZNK3MPI8Cartcomm3SubEPKb in auxfield.cpp.o
      __ZNK3MPI9Intracomm12Create_graphEiPKiS2_b in auxfield.cpp.o
      __ZNK3MPI8Cartcomm5CloneEv in auxfield.cpp.o
      __ZNK3MPI9Intracomm11Create_cartEiPKiPKbb in auxfield.cpp.o
      __ZNK3MPI9Intercomm5MergeEb in auxfield.cpp.o
      ...
  "__ZN3MPI8Datatype4FreeEv", referenced from:
      __ZTVN3MPI8DatatypeE in auxfield.cpp.o
      __ZTVN3MPI8DatatypeE in data2df.cpp.o
      __ZTVN3MPI8DatatypeE in domain.cpp.o
      __ZTVN3MPI8DatatypeE in geometry.cpp.o
      __ZTVN3MPI8DatatypeE in mesh.cpp.o
      __ZTVN3MPI8DatatypeE in message.cpp.o
      __ZTVN3MPI8DatatypeE in helmholtz.cpp.o
      ...
  "_ompi_mpi_cxx_op_intercept", referenced from:
      __ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb in auxfield.cpp.o
      __ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb in data2df.cpp.o
      __ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb in domain.cpp.o
      __ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb in geometry.cpp.o
      __ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb in mesh.cpp.o
      __ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb in message.cpp.o
      __ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb in helmholtz.cpp.o
      ...
ld: symbol(s) not found for architecture arm64
collect2: error: ld returned 1 exit status
make[2]: *** [elliptic_mp] Error 1
make[1]: *** [CMakeFiles/elliptic_mp.dir/all] Error 2
make: *** [all] Error 2

These linkage errors do not occur if I use mpich instead of openmpi (but right now I have issues with mpich not working on Sonoma). As I said, this looks like it could be an issue with C++ mangling routine names as a result of a misplaced "extern" declaration in a header.

An older code version which used a smaller subset of MPI continues to compile and run fine.

[ggouaillardet]

Your application is using the MPI C++ bindings that have been removed from the MPI standard more than a decade ago.

The right fix is to modernize your code and stop using them.

Meanwhile, you can rebuild Open MPI by passing --enable-mpi-cxx to the configure command line
(note that won't be possible any more with Open MPI 5)

[floquet-cxx]

Thanks - now could you please suggest a simple way to revert to using C bindings? It seems, I can't just put extern "C" { #include <mpi.h>}. Perhaps I can put extern "C" {} around each and every instance of MPI use, but I don't think so.

An alternative may be to pull them all the routines I've used out of C++ code and put them into a C code file; effectively, to re-wrap them - that used to be my approach when I was only calling a small set of MPI routines. (Forgive me but that somehow doesn't seem like "modernization"!)

[ggouaillardet]

This is not straight forward, but this is not rocket science either.

For example, you can compare examples/ring_c.c and examples/ring_cxx.cc to get an idea of what has to be changed.

Stopping using an API that has been removed from the MPI standard more than a decade ago is indeed modernization.
But if you believe/expect something more C++ish is the way to modernize your code, then feel free to do your own research starting for example with Boost.MPI or Elementals (do not ask me to help you though)

[rhc54]

Another, perhaps simpler, option might be to just use OMPI v3.x - IIRC, the C++ bindings were still supported at that time. Unless you really need something in the 4.x series, you can probably find an older version that works for you.

[floquet-cxx]

Another question occurs to me though: why don't I get the same issues when I call a more restricted set of MPI routines directly from C++? That approach seems to work fine with openmpi. I would have thought that I should get many more instances of linkage failure if the problem is related to removal of C++ bindings.

[ggouaillardet]

you are probably using the C bindings.

[floquet-cxx]

But, a top-level instance of #include <mpi.h> is used in every C++ file I have. So how could I be using the C bindings? Maybe because testing for CXX is not uniform/consistent across openmpi include files? That seems a bit unlikely.

However, I'm happy to believe you that I should be using C bindings now, and make the adjustments if I have to.

[ggouaillardet]

there is only one include file, and it is mpi.h

For example you use the C bindings if you do something like MPI_Send(..., MPI_COMM_WORLD), and you use the C++ bindings if you MPI::COMM_WORLD.Send(...)

That being said, maybe the C++ bindings have been built and your link command is missing -lmpi_cxx
That can happen if you link manually or if you link with mpicc instead of mpicxx

[floquet-cxx]

Oh for sure, I am just using the C bindings within my C++ code. For example

MPI_File_set_view
  (fh, skip, Geometry::contig(), Geometry::fileview(),
   "native", MPI_INFO_NULL);

Thank you, changing the link command is a good idea. I don't have such easy direct control over link flags because I'm using macports and cmake. I will have a go though, and also feed that back to the macports maintainers.

Also, I can't find ring_cxx.cc in the examples...

[floquet-cxx]

Hmmmm. I should just be able to use the C bindings directly as I am doing though, right? Again, I am wondering if there is a missing extern "C" in some header file. I see a few remarks about this when I look through the headers.

[ggouaillardet]

That's a C binding, but not the one the error message is about.

Try to grep Clone and see what you got

the example can be seen at https://github.com/open-mpi/ompi/blob/v4.1.x/examples/ring_cxx.cc

[floquet-cxx]

Hmmm. No Clone, but MPI_Datatype does certainly get used. And MPI_Type_free(). All in one simple base file. Both these things come up on the errors. Perhaps I'm mis-using something, let me review that file. (Why that would have worked with mpich, I don't know...). Thanks again for your suggestions.

[ggouaillardet]

Since you are not using the MPI C++ bindings, you should be able to compile with -DOMPI_SKIP_MPICXX
By doing so, no MPI C++ headers will be pulled.
But if the compilation fails, that would suggests your code is trying to use them.