COMM_SPAWN is broken in 2.1.x HEAD

[jsquyres]

Recent Cisco and NVIDIA MTT results are showing COMM_SPAWN and related dynamic operations are failing.

We need to find out if it was broken in v2.0.0 and see if this is a regression. This will determine whether it's a v2.0.1 blocker or not (because we'd really like to release the other important fixes in v2.0.1 ASAP, and start working on the short timeline for v2.1.0 and v2.0.2, etc.).

@sjeaugey is going to test on his tree and see if the recent v2.x commit(s) about bringing over PMIx 1.1.5 are the root of this particular problem.

[sjeaugey]

Strangely enough, IBM MTT shows the issue on master but not on v2.x.

Also, I looked at the 1.1.5rc commit and it is far from being a significant change. If this commit is the issue, it should be easy to fix.

[sjeaugey]

I'm currently unable to reproduce the problem on v2.x outside of MTT. On MTT though, it failed both with tcp and smcuda, and the error is a clear segmentation fault in orted (not the random pmix failure when there is a remaining socket in /tmp).

[jsquyres]

@sjeaugey Can you provide a backtrace?

[jsquyres]

@sjeaugey Also, can you force your MTT to use a tarball or git clone with that PMIx commit reverted?

[sjeaugey]

@jsquyres I still need to be able to reproduce the problem. I'm trying to launch MTT building only v2.x and only IBM dynamic/spawn, but maybe it only fails when spawn is launched after in the middle of all other tests.

[jsquyres]

@sjeaugey You might be able to run this indirectly via MTT. E.g. (I didn't look at your MTT results; I'm guessing/assuming it's the ibm section where you're seeing failures, and fill in ... below for your relevant .ini file and scratch directory):

$ client/mtt --file ... --scratch ... --verbose --mpi-get
$ client/mtt --file ... --scratch ... --verbose --mpi-install
$ client/mtt --file ... --scratch ... --verbose --test-get --section ibm
$ client/mtt --file ... --scratch ... --verbose --test-build --section ibm
$ client/mtt --file ... --scratch ... --verbose --test-run --section ibm-run-the-bad-test

I.e., make a test run section named ibm-run-the-bad-test that just runs the one test that is failing. That way you can run the test via MTT and hopefully be able to reproduce the error.

Make sense?

[sjeaugey]

@jsquyres Yes it makes sense. Still, no luck. When I only run dynamic/spawn, the tests pass.

@rhc54 I'm starting to wonder if I were just out of luck and hit the pmix problem (when the socket is already in /tmp), only that in the case of spawns, instead of the usual "Error in file orted/pmix/pmix_server.c at line 254" message, I get a segmentation fault.

[sjeaugey]

I added tmp cleanup before my nightly MTT run. We'll see tomorrow if it solved the issue.

[rhc54]

@sjeaugey Looks like you ran clean last night - can you clarify exactly what you did? Did you just clean the tmp? Or did you also roll back the PMIx 1.1.5 commit?

[sjeaugey]

@rhc54 I wouldn't say that. I got rid of some failures (caused by remaining sockets in tmp) but only 2 failures remain : spawn and spawn_with_env_vars. And spawn_multiple timed out too.

I'm still unable to reproduce it outside of the nightly runs however. Working on it.

[rhc54]

how strange...the nightly summary didn't show those failures. Is there a core file you can look at? A line number where those daemons are crashing would really help.

[sjeaugey]

Some new interesting fact : on my second node, the spawn process are still there, stuck :

[sjeaugey@drossetti-ivy5 ~]$ ps -ef | grep spawn
sjeaugey 18626     1 99 Aug29 ?        2-04:35:50 dynamic/spawn this is argv 1 this is argv 2
sjeaugey 22338     1  3 Aug30 ?        00:58:16 dynamic/spawn this is argv 1 this is argv 2
sjeaugey 22447     1 99 Aug30 ?        1-04:08:22 dynamic/spawn_with_env_vars
sjeaugey 23013     1  3 06:02 ?        00:06:57 dynamic/spawn this is argv 1 this is argv 2
sjeaugey 23064     1 99 06:02 ?        03:35:28 dynamic/spawn_with_env_vars
sjeaugey 23122     1 99 06:02 ?        03:35:31 dynamic/spawn_with_env_vars

I tried to gstack them. That killed some of the processes, but some others survived and gave me a stack trace :

[sjeaugey@drossetti-ivy5 ~]$ gstack 18626
Thread 3 (Thread 0x7fbb3b437700 (LWP 18628)):
#0  0x00007fbb3d048173 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fbb3c9ea998 in epoll_dispatch () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libopen-pal.so.20
#2  0x00007fbb3c9eddfe in opal_libevent2022_event_base_loop () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libopen-pal.so.20
#3  0x00007fbb3b4650bd in progress_engine () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/openmpi/mca_pmix_pmix112.so
#4  0x00007fbb3d2fa9d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fbb3d047b7d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fbb3a436700 (LWP 18631)):
#0  0x00007fbb3d00bced in nanosleep () from /lib64/libc.so.6
#1  0x00007fbb3d040e64 in usleep () from /lib64/libc.so.6
#2  0x00007fbb3b46d715 in OPAL_PMIX_PMIX112_PMIx_Abort () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/openmpi/mca_pmix_pmix112.so
#3  0x00007fbb3b44202a in pmix1_abort () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/openmpi/mca_pmix_pmix112.so
#4  0x00007fbb3b6b7bf0 in rte_abort () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/openmpi/mca_ess_pmi.so
#5  0x00007fbb3d608157 in ompi_rte_abort () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#6  0x00007fbb3d566210 in ompi_mpi_abort () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#7  0x00007fbb3d54ce35 in ompi_errhandler_runtime_callback () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#8  0x00007fbb3cce7bdd in orte_errmgr_base_execute_error_callbacks () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libopen-rte.so.20
#9  0x00007fbb38a23972 in proc_errors () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/openmpi/mca_errmgr_default_app.so
#10 0x00007fbb3c9ee851 in opal_libevent2022_event_base_loop () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libopen-pal.so.20
#11 0x00007fbb3c995b22 in progress_engine () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libopen-pal.so.20
#12 0x00007fbb3d2fa9d1 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fbb3d047b7d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fbb3da91700 (LWP 18626)):
#0  0x00007fbb30c5cf61 in sm_fifo_read () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/openmpi/mca_btl_smcuda.so
#1  0x00007fbb30c5f111 in mca_btl_smcuda_component_progress () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/openmpi/mca_btl_smcuda.so
#2  0x00007fbb3c9903ae in opal_progress () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libopen-pal.so.20
#3  0x00007fbb3d564c46 in ompi_request_wait_completion () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#4  0x00007fbb3d564c80 in ompi_request_default_wait () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#5  0x00007fbb3d5e059b in ompi_coll_base_bcast_intra_generic () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#6  0x00007fbb3d5e0c7f in ompi_coll_base_bcast_intra_binomial () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#7  0x00007fbb2aa05238 in ompi_coll_tuned_bcast_intra_dec_fixed () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/openmpi/mca_coll_tuned.so
#8  0x00007fbb3d543706 in ompi_comm_allreduce_intra_pmix () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#9  0x00007fbb3d540e13 in ompi_comm_nextcid () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#10 0x00007fbb3d5492bd in ompi_dpm_connect_accept () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#11 0x00007fbb3d54b793 in ompi_dpm_dyn_init () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#12 0x00007fbb3d567174 in ompi_mpi_init () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#13 0x00007fbb3d59a00c in PMPI_Init () from /ivylogin/home/sjeaugey/tests/mtt/scratches/mtt-scratch-1/installs/EDVY/install/lib/libmpi.so.20
#14 0x0000000000401652 in main ()

I'll kill those remaining processes eventually (they could be the cause of the failures), but if you want me to look at something in particular, please tell me.