Crash in with Open MPI v4.1.x in MPI_Win_lock_all when used with libfabric < 1.12

[Flamefire]

Background information

We were using OpenMPI 4.0.5 with libfabric 1.11.0 for MPI one-sided communication. When upgrading to OpenMPI 4.1 MPI_Win_lock_all crashes.

Using libfabric 1.12.x works. However as OMPI 4.0 works with libfabric 1.11 this rather looks like an OMPI bug and upgrading libfabric may not be easily possible

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

4.0.x (specifically 4.0.5) works, 4.1.0 & 4.1.1 crashes

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Source

Please describe the system on which you are running

• Operating system/version: RHEL 7.9
• Computer hardware: Intel x86
• Network type: Infiniband

---

Details of the problem

The following program crashes when compiled and run with mpirun over more than 1 node:
test_mpi2.cpp.txt

Output:

[taurusi6584:18594:0:18594] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x40)
==== backtrace (tid:  18594) ====
 0 0x00000000000234f3 ucs_debug_print_backtrace()  /dev/shm/easybuild-build/UCX/1.9.0/GCCcore-10.2.0/ucx-1.9.0/src/ucs/debug/debug.c:656
 1 0x00000000000102e1 rxd_start_xfer.cold()  rxd_cq.c:0
 2 0x0000000000073a16 rxd_progress_tx_list()  crtstuff.c:0
 3 0x000000000007547b rxd_handle_recv_comp()  crtstuff.c:0
 4 0x00000000000781a5 rxd_ep_progress()  crtstuff.c:0
 5 0x000000000002ed3d ofi_cq_progress()  crtstuff.c:0
 6 0x000000000002e09e ofi_cq_readfrom()  crtstuff.c:0
 7 0x0000000000006da7 mca_btl_ofi_context_progress()  ???:0
 8 0x0000000000003e8e mca_btl_ofi_component_progress()  btl_ofi_component.c:0
 9 0x00000000000313ab opal_progress()  ???:0
10 0x0000000000018335 ompi_osc_rdma_lock_all_atomic()  ???:0
11 0x000000000009b203 MPI_Win_lock_all()  ???:0
12 0x00000000004017e1 main()  ???:0
13 0x0000000000022555 __libc_start_main()  ???:0
14 0x0000000000401529 _start()  ???:0
=================================
[taurusi6584:18594] *** Process received signal ***
[taurusi6584:18594] Signal: Segmentation fault (11)
[taurusi6584:18594] Signal code:  (-6)
[taurusi6584:18594] Failing at address: 0xf51cf000048a2
[taurusi6584:18594] [ 0] /lib64/libpthread.so.0(+0xf630)[0x2b3dcca1c630]
[taurusi6584:18594] [ 1] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/libfabric/1.11.0-GCCcore-10.2.0/lib/libfabric.so.1(+0x102e1)[0x2b3dcdad92e1]
[taurusi6584:18594] [ 2] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/libfabric/1.11.0-GCCcore-10.2.0/lib/libfabric.so.1(+0x73a16)[0x2b3dcdb3ca16]
[taurusi6584:18594] [ 3] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/libfabric/1.11.0-GCCcore-10.2.0/lib/libfabric.so.1(+0x7547b)[0x2b3dcdb3e47b]
[taurusi6584:18594] [ 4] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/libfabric/1.11.0-GCCcore-10.2.0/lib/libfabric.so.1(+0x781a5)[0x2b3dcdb411a5]
[taurusi6584:18594] [ 5] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/libfabric/1.11.0-GCCcore-10.2.0/lib/libfabric.so.1(+0x2ed3d)[0x2b3dcdaf7d3d]
[taurusi6584:18594] [ 6] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/libfabric/1.11.0-GCCcore-10.2.0/lib/libfabric.so.1(+0x2e09e)[0x2b3dcdaf709e]
[taurusi6584:18594] [ 7] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/OpenMPI/4.1.1-GCC-10.2.0/lib/openmpi/mca_btl_ofi.so(mca_btl_ofi_context_progress+0x57)[0x2b3dcdabfda7]
[taurusi6584:18594] [ 8] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/OpenMPI/4.1.1-GCC-10.2.0/lib/openmpi/mca_btl_ofi.so(+0x3e8e)[0x2b3dcdabce8e]
[taurusi6584:18594] [ 9] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/OpenMPI/4.1.1-GCC-10.2.0/lib/libopen-pal.so.40(opal_progress+0x2b)[0x2b3dcc1693ab]
[taurusi6584:18594] [10] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/OpenMPI/4.1.1-GCC-10.2.0/lib/openmpi/mca_osc_rdma.so(ompi_osc_rdma_lock_all_atomic+0x335)[0x2b3dcfd69335]
[taurusi6584:18594] [11] /beegfs/global0/ws/s3248973-easybuild/openMPINew/software/OpenMPI/4.1.1-GCC-10.2.0/lib/libmpi.so.40(PMPI_Win_lock_all+0xb3)[0x2b3dcb661203]
[taurusi6584:18594] [12] a2.out[0x4017e1]
[taurusi6584:18594] [13] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b3dcbcd0555]
[taurusi6584:18594] [14] a2.out[0x401529]
[taurusi6584:18594] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 18594 on node taurusi6584 exited on signal 11 (Segmentation fault).

  Configure command line: '--build=x86_64-pc-linux-gnu'
                          '--host=x86_64-pc-linux-gnu' '--with-slurm'
                          '--with-pmi=/usr' '--with-pmi-libdir=/usr/lib64'
                          '--with-knem=/opt/knem-1.1.3.90mlnx1'
                          '--enable-mpirun-prefix-by-default'
                          '--enable-shared' '--with-cuda=no'
                          '--with-hwloc=/sw/installed/hwloc/2.2.0-GCCcore-10.2.0'
                          '--with-libevent=/sw/installed/libevent/2.1.12-GCCcore-10.2.0'
                          '--with-ofi=/beegfs/global0/ws/s3248973-easybuild/openMPINew/software/libfabric/1.11.0-GCCcore-10.2.0'
                          '--with-pmix=/sw/installed/PMIx/3.1.5-GCCcore-10.2.0'
                          '--with-ucx=/sw/installed/UCX/1.9.0-GCCcore-10.2.0'
                          '--without-verbs'

[jsquyres]

If you have an infiniband network, is there a reason you're not using the UCX PML?

[Flamefire]

As you can see I build with ucx and ofi. So OMPI can choose, I guess.

[yosefe]

@Flamefire seems like UCX is not selected by default . Can you please run with -mca pml_base_verbose 100 -mca pml_ucx_verbose 100 (args from mpirun) and upload the output?
Also, can you try by adding -mca pml ucx for force UCX selection?

[Flamefire]

The reason we build with libfabric and UCX (as far as I remember) is that the compilation process should be universal so the built package can be used on different systems. In particular this is built with EasyBuild via a "recipe" and run on different HPC systems worldwide. Hence "just use UCX" is not a solution but only a workaround if at all.
So far for context.

The requested logs for without and with -mca pml ucx(both crash):
mpicrash.txt
mpicrash_ucx.txt

[yosefe]

@Flamefire OFI (with osc/rdma layer) is being selected by default since it has higher priority (=101) than osc/ucx (=70).
Can you please add also "-mca osc ucx" to the command line?

@jsquyres on a general note, IMO libfabric components should reduce their priority when running on Mellanox hardware, so that it will be lower than osc/ucx. This is same as UCX components reducing their priority when NOT running on Mellanox hardware. WDYT?

[jsquyres]

@yosefe If the OFI component is being selected incorrectly, then yes, I agree that that is a problem. I don't think it should look for NVIDIA hardware specifically and demote itself in that case, though -- that would be weird (i.e., look for and react to hardware that it wouldn't otherwise look for).

It's quite possible that we're not testing "full build" scenarios well enough (i.e., builds with as many communication libraries as possible, to include Libfabric, UCX, ... etc.):

1. In situations with only OFI-enabled hardware
2. In situations with only NVIDIA-enabled hardware
3. In situations with neither OFI- nor NVIDIA-enabled hardware
4. In situations with both OFI- and NVIDIA-enabled hardware

[yosefe]

I don't think it should look for NVIDIA hardware specifically and demote itself in that case, though

So should OFI check for OFI-specific hardware?

It's quite possible that we're not testing "full build" scenarios well enough (i.e., builds with as many communication libraries as possible, to include Libfabric, UCX, ... etc.):

For case (4), IMO the user should select explicitly (via -mca param) to get the right library to work. And for case (3), probably neither OFI nor UCX should be high priority.

[jsquyres]

So should OFI check for OFI-specific hardware?

I'm not sure what the right answer is here. I agree that if there's only NVIDIA hardware and UCX is available, the UCX components should be auto-selected over the OFI components. I'm not sure of the exact mechanism.

FYI: @open-mpi/ofi this requires some discussion. See the original description, above, for version/environment information.

[rhc54]

FYI: @open-mpi/ofi this requires some discussion. See the original description, above, for version/environment information.

Sounds like a great topic for the proposed developer's meetings, yes? https://github.com/open-mpi/ompi/wiki/Meeting-2021-07

[mwheinz]

ucs_debug_print_backtrace() means that the crash occurred inside the UCX library itself. I would really dislike UCX being the default because it already mistakes OPA hardware for Mellanox hardware and generates warning or errors on OPA fabrics if it isn't disabled.

[acgoldma]

OMPI v4.1 added in OFI BTL support (used to just be only MTL in v4.0). If for whatever reason you did not need ofi btl, then the adding this can disable it: --mca btl ^ofi

Please rerun and capture output with ofi logging enabled: -x FI_LOG_LEVEL=info
This appears to be a failure in rxd that is getting caught by a signal handler in UCX (I think)

Some ofi providers layer what are called util providers (ofi_rxd, ofi_rxm, etc.) on top of core providers.
There is no easy way to disable this 'feature' (if you wanted) other than excluding all providers but the one you want:

MY_PROVIDER=psm3
FI_PROVIDER="^$(fi_info | sed -n 's/provider: //p'  | sort -u | grep -v ^${MY_PROVIDER}$ | tr '\n' ',' | sed 's/,$//')"

mpirun ... -x FI_PROVIDER=^UDP,UDP;ofi_rxd,psm2,psm2;ofi_rxd,psm3;ofi_rxd,shm,sockets,tcp,tcp;ofi_rxm,verbs,verbs;ofi_rxd,verbs;ofi_rxm  ./my_mpi_app

Note: psm3 is an example as this is what I work on :). It was also added to libfabric v1.12.x

Notice above psm3;ofi_rxd is there but psm3 was the only one not excluded.

[yosefe]

ucs_debug_print_backtrace() means that the crash occurred inside the UCX library itself. I would really dislike UCX being the default because it already mistakes OPA hardware for Mellanox hardware and generates warning or errors on OPA fabrics if it isn't disabled.

The error happened at 1 0x00000000000102e1 rxd_start_xfer.cold() rxd_cq.c:0; ucs_debug_print_backtrace is merely the signal handler. Anyway, latest versions of UCX do not install SIGSEGV handler by default.
Also, we changed pml/ucx to reduce UCX priority if no "mlnx" hardware is found, so it should not be selected on OPA fabrics. If you observe otherwise, pls LMK and we will fix it.