Crash with Open MPI 4.1.0 on macOS, but not 4.0.5: Symbol not found: _preadv

[mathomp4]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

I am using v4.0.5 and v4.1.0 for this.

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

In each case, the stack was built with GCC 10.2.0 built from scratch. I then used the release tarballs from Open MPI. The configure line for each is:

$ ./configure --disable-wrapper-rpath --disable-wrapper-runpath \
CC=gcc CXX=g++ FC=gfortran \
--prefix=$HOME/installed/Compiler/gcc-gfortran-10.2.0/openmpi/4.0.5

where only the last "4.0.5" is changed to "4.1.0" for 4.1.0.

Please describe the system on which you are running

• Operating system/version: macOS 10.15.7
• Computer hardware: Intel MacBook Pro 2019
• Network type: Uh...internal? It's not a cluster, just a laptop running with ye olde --oversubscribe

---

Details of the problem

I'm still trying to see if I can figure out a reproducer for this (Hello World works just fine). But I want to try and describe the issue.

I help maintain an earth system model and we like to have the ability to run on a local laptop instead of our usual supercomputers for ease of development, etc. So, on my laptop, I build GCC and Open MPI by hand to have better control. I then build some base libraries and then the model.

In this case, everything is identical except for the Open MPI version. In every case a 4.0.5 becomes a 4.1.0. With 4.0.5, the model runs just fine:

...
 Character Resource Parameter: RATS_PROVIDER:PCHEM
 Character Resource Parameter: pchem_clim:species.data
 Integer*4 Resource Parameter: pchem_clim_years:39
 GOCART::Initialize_: Starting...

 Real*4 Resource Parameter: GOCART_DT:900.000000
 Integer*4 Resource Parameter: GOCART_REFERENCE_TIME:-999
 Character Resource Parameter: GOCART_IMPORT_RESTART_FILE:gocart_import_rst
 Bootstrapping gocart_import_rst
...

But when I move to 4.1.0, I get this:

...
 Character Resource Parameter: RATS_PROVIDER:PCHEM
 Character Resource Parameter: pchem_clim:species.data
 Integer*4 Resource Parameter: pchem_clim_years:39
dyld: lazy symbol binding failed: Symbol not found: _preadv
  Referenced from: /Users/mathomp4/installed/Compiler/gcc-gfortran-10.2.0/openmpi/4.1.0/lib/openmpi/mca_fbtl_posix.so
  Expected in: flat namespace

dyld: Symbol not found: _preadv
  Referenced from: /Users/mathomp4/installed/Compiler/gcc-gfortran-10.2.0/openmpi/4.1.0/lib/openmpi/mca_fbtl_posix.so
  Expected in: flat namespace


Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0  0x12c983d3d
#1  0x12c98316d
#2  0x7fff6a9055fc

As an additional test, I built our model with Intel Fortran/Clang + Open MPI 4.1.0, and the model runs.I notice in the config.log files for 4.1.0:

❯ rg ac_cv_func_preadv *.log
config.gcc-gfortran-10.2.0.log
188849:ac_cv_func_preadv=yes

config.intel-clang-2020.4.301.log
188184:ac_cv_func_preadv=no

So GNU has preadv, but Intel-Clang does not?

Now, I did a bit of diffing between 4.0.5 and 4.1.0 and I do see changes in files with the word preadv like ompi/mca/fbtl/posix/fbtl_posix_preadv.c. Indeed, a big change! But...maybe that doesn't matter?

Any help would be much appreciated.

[edgargabriel]

@mathomp4 thank you for report this issue. You are right that there were significant changes in the fbtl posix component, I am however confused since the part that you are referring to (namely the utilization of preadv) is identical in the 4.1 and 4.0.x series. The function is protected by an #if macro,e.g.

#if defined(HAVE_PREADV)
        ret_code = preadv (fh->fd, iov, iov_count, iov_offset);
#else
        if (-1 == lseek (fh->fd, iov_offset, SEEK_SET)) {
            opal_output(1, "mca_fbtl_posix_preadv: error in lseek:%s", strerror(errno));
            free(iov);
            mca_fbtl_posix_unlock ( &lock, fh );
            return OMPI_ERROR;
        }
        ret_code = readv (fh->fd, iov, iov_count);
#endif

i.e. if the configure step does not find the preadv function with a given compiler, it should not need the preadv function. I do not see (at least in fbtl_posix_preadv.c) any utilization of preadv outside of the if macro.

I will try to get access to a Mac for testing this out as well, will come back to you later today/tomorrow.
Short question though: it only complains about preadv, but not about pwritev (which is handled in exactly the same manner) ?

[mathomp4]

@edgargabriel Well, it might just be where I am in the program. It's dying in the initialization phase of our model which is a looooooot of file reads. I'd have to consult with the experts, but it's possible this could be the first MPI IO read? Our model is a mix of plain fortran read(), a lot of netCDF Fortran, and some scatter MPI I/O.

Do you know of a "simple" MPI program I could whip up that would exercise this code? My usual "test" MPI programs are things like helloWorld or a Pi Calculator, so not very fancy.

[edgargabriel]

for this particular purpose, something as simple as the following test would do:

#include <stdio.h>
#include "mpi.h"

int main ( int argc, char ** argv)
{
    int rank, size, ret;
    int val;
    MPI_File fh;
    MPI_Status status;

    MPI_Init ( &argc, &argv );
    MPI_Comm_rank ( MPI_COMM_WORLD, &rank );
    MPI_Comm_size ( MPI_COMM_WORLD, &size );

    MPI_File_open (MPI_COMM_WORLD, "test.out", MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);
    ret = MPI_File_read (fh, &val, 1, MPI_INT, &status );
    if ( ret != MPI_SUCCESS ) {
        printf("Error while writing file. Aborting\n");
        MPI_Abort ( MPI_COMM_WORLD, 1 );
    }

    MPI_File_close ( &fh);
    MPI_Finalize ();
    return 0;
}

just compile this file using mpicc, create a testfile called test.out with an integer value in it, and if you run it on a single process using you should see the same error message,

[edgargabriel]

actually, looking back at your error message, the issue is maybe slightly different:

you use Users/mathomp4/installed/Compiler/gcc-gfortran-10.2.0/openmpi/4.1.0/lib/openmpi/mca_fbtl_posix.so which is I assume using gcc-gfortran-10.2.0. According to some later output that you showed, gcc-gfortran has ac_cv_func_preadv=yes (which means the configure script found preadv), and that's why its using the preadv function (but can't find it at load time).

looking at the differences between the fbtl_posix_preadv and fbtl_posix_pwritev.v files, the pwritev.c file has one additional include statement (#include <sys/uio.h>), I am wandering whether that is the issue here. Will give it a go tonight once I have access to the Mac of my wife:-) I am not just not sure why things worked with the 4.0.x series in that case, since it was the same code in that series already as well.

[mathomp4]

Well, sadly your example didn't cause a crash. I even went looking through our source code and added some of the info's we set:

9a10
>     MPI_Info info;
15c16,19
<     MPI_File_open (MPI_COMM_WORLD, "test.out", MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);
---
>     MPI_Info_create ( &info );
>     MPI_Info_set ( info, "romio_cb_read", "automatic" );
>     MPI_Info_set ( info, "cb_buffer_size", "1048576" );
>     MPI_File_open (MPI_COMM_WORLD, "test.out", MPI_MODE_RDONLY, info, &fh);
22a27
>     MPI_Info_free ( &info );

but it still worked.

[mathomp4]

I might try out adding:

#include <sys/uio.h>

to a 4.1.0 build and see what happens. Maybe it will do something!

[edgargabriel]

did you compile netcdf with MPI support, and if yes, did you recompile it separately with gcc vs. clang as well? I am wandering whether the gcc compiled fbtl component could be sucked into the clang execution, and this is causing the issue.

[mathomp4]

@edgargabriel Yes, I compiled netCDF with MPI support. To be "safe" I'm recompiling Open MPI, then I'll do the baselibs, and then I'll build the model and run it.

As for compiling, in this case, I don't let clang get near it as far as I can. My library build system sees only gcc/mpicc (where mpicc = gcc), etc. My lua modules are setup to make sure I prepend paths and I tell the build system "Use gcc, not clang". I'll report back...probably tomorrow. it's going to be a lot of building on a laptop!

[mathomp4]

I can report that adding:

#include <sys/uio.h>

to ompi/mca/fbtl/posix/fbtl_posix_preadv.c didn't help. Still get the preadv crash. Ah well. A valiant effort!

[edgargabriel]

I just compiled openmpi-4.1.0 on the Mac of my wife, it uses Apple clang version 11.0, and I am afraid that I could not reproduce the problem that you mentioned. Configure correctly detected that preadv and pwritev are not available (ac_cv_func_preadv=no) and the code uses because of that the workaround (i.e. lseek/pread). I could run tests without issues. So not entirely sure how to follow up on this.

That being said, when I ran my testsuite, there was a testcase that triggered an issue with aio_read()/aio_write(). I don't think it is related to the issue that you reported on this ticket, but I will look into it anyway.

As a workaround, you could try to run your code enforcing using the romio321 component instead of ompio, e.g.

mpirun --mca io romio321 -np x ...

[mathomp4]

As a workaround, you could try to run your code enforcing using the romio321 component instead of ompio, e.g.

mpirun --mca io romio321 -np x ...

Well, guess what worked? 👍🏼

I'm going to blow away my model build and rebuild from scratch to confirm.

[mathomp4]

Yep. I can confirm it works for me. I can even add:

setenv("OMPI_MCA_io","romio321")

in my Open MPI 4.1.0 modulefile and it works.

So, is this "safe"? Or is there something maybe I can do at configure time?

[edgargabriel]

I am the main developer of ompio, the default parallel I/O component in Open MPI, so I will not advocate to switch to romio, the competition 😉. You probably also don't want a detailed comparison between ompio and romio. I would look at it fairly pragmatic at this point: as long as we are not able to identify the problem on your laptop with your application, this is viable option for you.

[mathomp4]

Ah. Indeed. 😄

Was romio the default in Open MPI 4.0.x and that's why it worked for me? Or did something change in ompio?