OMPI + UCX segfault during MPI_Finalize

[bosilca]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

Master with a full set of UCX flavors: 1.7, 1.8 1.9 and master. All UCX versions were configured with --with-cuda --with-avx --enable-mt

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git pull
configure --prefix=... --enable-picky --enable-debug --disable-heterogeneous --enable-visibility \
          --enable-mpirun-prefix-by-default --with-pmix=internal --with-ucx=/opt/ucx/XXX/fast

Please describe the system on which you are running

Scientific Linux release 7.4 (Nitrogen)
Cluster of Mellanox 56gbps

hca_id:	mthca0
	transport:			InfiniBand (0)
	fw_ver:				4.8.200

Details of the problem

Any app run on 2 nodes, including the most trivial benchmark, results in a failure during MPI_Finalize. Everything seems to run fine on a single node,

 0 0x0000000000053b83 ucs_debug_print_backtrace()  src/ucs/debug/debug.c:653
 1 0x000000000001cb8b uct_rc_verbs_handle_failure()  src/uct/ib/rc/verbs/rc_verbs_iface.c:63
 2 0x000000000001d195 uct_rc_verbs_iface_poll_tx()  src/uct/ib/rc/verbs/rc_verbs_iface.c:104
 3 0x000000000001d195 uct_rc_verbs_iface_progress()  src/uct/ib/rc/verbs/rc_verbs_iface.c:132
 4 0x000000000002a79a ucs_callbackq_dispatch()  src/ucs/datastruct/callbackq.h:211
 5 0x000000000002a79a uct_worker_progress()  src/uct/api/uct.h:2221
 6 0x000000000002a79a ucp_worker_progress()  src/ucp/core/ucp_worker.c:1951
 7 0x0000000000002d0c opal_common_ucx_wait_request()  ompi/build/debug/opal/mca/common/ucx/../../../../../../opal/mca/common/ucx/common_ucx.h:182
 8 0x00000000000032f8 opal_common_ucx_wait_all_requests()  ompi/build/debug/opal/mca/common/ucx/../../../../../../opal/mca/common/ucx/common_ucx.c:210
 9 0x0000000000003574 opal_common_ucx_del_procs_nofence()  ompi/build/debug/opal/mca/common/ucx/../../../../../../opal/mca/common/ucx/common_ucx.c:254
10 0x00000000000035fb opal_common_ucx_del_procs()  ompi/build/debug/opal/mca/common/ucx/../../../../../../opal/mca/common/ucx/common_ucx.c:272
11 0x0000000000006877 mca_pml_ucx_del_procs()  ompi/build/debug/ompi/mca/pml/ucx/../../../../../../ompi/mca/pml/ucx/pml_ucx.c:481
12 0x0000000000071510 ompi_mpi_finalize()  ompi/build/debug/ompi/../../../ompi/runtime/ompi_mpi_finalize.c:326
13 0x00000000000b7cb1 PMPI_Finalize()  ompi/build/debug/ompi/mpi/c/profile/pfinalize.c:54
14 0x0000000000400b35 main()  osu/osu_latency.c:305
15 0x00000000000223d5 __libc_start_main()  ???:0
16 0x0000000000400db2 _start()  ???:0

[yosefe]

"mthca0" devices are not supported by UCX
pls see more details in openucx/ucx#5318

[bosilca]

Hold on, what exactly “wont fix” means ? I am not asking you to make UCX support old generation hardware, but you can’t leave OMPI segfaulting due to a choice made in your component. You have a choice either to implement a way to prevent UCX from selecting itself when it cannot run and/or fix the segfault after the run but we need a way to have OMPI run everywhere.

[yosefe]

@bosilca can you pls provide the full error log and full output of ibstat/ibv_devinfo?

[bosilca]

$ ibstat
CA 'mthca0'
	CA type: MT25208 (MT23108 compat mode)
	Number of ports: 2
	Firmware version: 4.8.200
	Hardware version: a0
	Node GUID: 0x00066a0098006c03
	System image GUID: 0x00066a0098006c03
	Port 1:
		State: Down
		Physical state: Polling
		Rate: 10
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x02590a68
		Port GUID: 0x00066a00a0006c03
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 10
		Base lid: 13
		LMC: 0
		SM lid: 1
		Capability mask: 0x02590a6a
		Port GUID: 0x00066a01a0006c03
		Link layer: InfiniBand

[bosilca]

For the error log it's a little more tricky, there is a lot of info from each process. I made a selection below:

local QP operation err (QPN 47040e, WQE @ 00007b03, CQN 5300b0, index 380)
  [ 0] 0047040e
  [ 4] 00000000
  [ 8] 00000000
  [ c] 00000000
  [10] 02700000
  [14] 00000000
  [18] 00007b03
  [1c] ff100000
[c05:8918 :0:8918] rc_verbs_iface.c:63   send completion with error: local QP operation error qpn 0x470411 wrid 0x29 vendor_err 0x70

      ...
       60     }
       61
       62     ucs_log(log_lvl,
==>    63             "send completion with error: %s qpn 0x%x wrid 0x%lx vendor_err 0x%x",
       64             ibv_wc_status_str(wc->status), wc->qp_num, wc->wr_id, wc->vendor_err);
       65 }
       66

[yosefe]

@bosilca openucx/ucx#5488 should fix the issue, can you pls try it?