error launching/attaching LaunchMON debugger with OpenMPI 2.1.1

[lee218llnl]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

v2.1.1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

from source tarball

Please describe the system on which you are running

• Operating system/version: RHEL7
• Computer hardware: x86-64
• Network type: infiniband

---

Details of the problem

I am having trouble attaching LaunchMON when using OpenMPI 2.1.1.

[LMON_FE] launching the job/daemons via /usr/workspace/wsrzd/lee218/install/toss_3_x86_64_ib/ompi-2.1.1/bin/orterun

[LMON FE] 6 RM types are supported
[warn] Epoll ADD(4) on fd 38 failed.  Old events were 0; read change was 0 (none); write change was 1 (add): Bad file descriptor
[warn] Epoll ADD(4) on fd 35 failed.  Old events were 0; read change was 0 (none); write change was 1 (add): Bad file descriptor
[rzoz1:132008] [[21665,0],0] usock_peer_send_blocking: send() to socket 36 failed: Broken pipe (32)
[rzoz1:132008] [[21665,0],0] ORTE_ERROR_LOG: Unreachable in file oob_usock_connection.c at line 316
[rzoz1:132008] [[21665,0],0]-[[21665,1],1] usock_peer_accept: usock_peer_send_connect_ack failed
[rzoz1:132008] [[21665,0],0] usock_peer_send_blocking: send() to socket 40 failed: Broken pipe (32)
[rzoz1:132008] [[21665,0],0] ORTE_ERROR_LOG: Unreachable in file oob_usock_connection.c at line 316
[rzoz1:132008] [[21665,0],0]-[[21665,1],0] usock_peer_accept: usock_peer_send_connect_ack failed
--------------------------------------------------------------------------
orterun was unable to start the specified application as it encountered an
error:

Error name: Not supported
Node: rzoz1

when attempting to start process rank 0.
--------------------------------------------------------------------------
<Jun 06 11:51:50> <LMON FE API> (INFO): FE-ENGINE connection timed out: 120
[LMON FE] FAILED

Here's how you can reproduce (modify your PATH and the path to mpirun):

git clone https://github.com/llnl/launchmon.git
cd launchmon/
export PATH=/collab/usr/global/tools/openmpi/toss_3_x86_64_ib/openmpi-2.1.1/bin:$PATH
CFLAGS="-g -O0" CXXFLAGS="-g -O0" ./configure --prefix=/nfs/tmp2/lee218/prefix/launchmon-1.0.3b --with-test-rm=orte --with-test-rm-launcher=/collab/usr/global/tools/openmpi/toss_3_x86_64_ib/openmpi-2.1.1/bin/mpirun --with-test-installed --with-test-nnodes=1 && make clean && make -j 8 install && make -j 8 check
cd test/src
./test.launch_1

In addition, the LaunchMON "test.attach_1" test hangs when trying to attach. @rhc54 had previously helped me with various debugger attach issues and we had a working commit. I don't know if that made it into the release or if this is a new issue. It would be nice if LaunchMON tests could be integrated as part of the release testing.

[lee218llnl]

FWIW, I can launch/attach via LaunchMON with OpenMPI 2.0.3

[rhc54]

I won't be able to get to this right away - I'm surprised this is still using the usock component, though. I'll have to check what version of pmix is being used.

So far as I can tell, all the PR's were taken. However, it is possible that I forgot to file one for the 2.1 series. I'll check that too.

Be advised that I plan to gradually move to supporting only PMIx attach methods as we go forward. Not sure if someone else will pickup the non-PMIx methods. Won't be until after 3.0, though, so nothing imminent - just something to plan for the future.

[rhc54]

Hmmm...well, the code supporting the debuggers has clearly not been updated. Not entirely sure why - probably my fault (most likely forgot to file, I suppose). Anyway, I'll create a patch.

[rhc54]

See the referenced PR for fix - please let us know if it resolves the problem. I tested it with both launch_1 and attach_1.

[lee218llnl]

This looks good to me, thanks!

[rhc54]

This apparently isn't fully fixed yet, according to folks at Allinea:

I've tested OpenMPI 2.1.2rc2 and I still hit the same problem reported in #3660 with Allinea MAP and DDT, but intermittently:
[e112475-lin:09338] [[1848,0],0] usock_peer_send_blocking: send() to socket 44 failed: Broken pipe (32)
[e112475-lin:09338] [[1848,0],0] ORTE_ERROR_LOG: Unreachable in file oob_usock_connection.c at line 316
[e112475-lin:09338] [[1848,0],0]-[[1848,0],0] usock_peer_accept: usock_peer_send_connect_ack failed

It is an intermittent issue and it usually appears when I use between 2 and 6 processes. I've used two completely different machines to test OpenMPI 2.1.2, my laptop (Intel i7) with Ubuntu 16 and an ARMv8 machine (ThunderX) with SUSE 12.
OpenMPI 3.0.0rc2 always work fine in my laptop (I did a lot of executions).

This is the problem with 2.1.2, it works using 8 procs, but it crashes using 2:
[xavoro01]:~/tmp/allinea-forge-7.0.6-Ubuntu-14.04-x86_64/examples$ ../bin/map --profile mpirun -n 8 wave_c 5
Waiting for licence
Allinea Forge 7.0.6 - Allinea MAP

Profiling             : mpirun -n 8 wave_c 5
Allinea sampler       : preload (Express Launch)
MPI implementation    : Auto-Detect (Open MPI)
* number of processes : 8
* number of nodes     : 1
* Allinea MPI wrapper : preload (JIT compiled) (Express Launch)

Wave solution with 8 processes
points = 1000000, running for 5 seconds

               min    mean    max
All time (ms): 5001    5009    5014
CPU time (ms): 2090    2179    2409
MPI time (ms): 2605    2831    2921

CPU time (%) : 42    43    48
MPI time (%) : 52    56    58

Iterations   : 1550    1550    1550
points/second: 309.3M (38.7M per process)
wave finished

MAP analysing program...
MAP gathering samples...
MAP generated /home/xavoro01/tmp/allinea-forge-7.0.6-Ubuntu-14.04-x86_64/examples/wave_c_8p_1n_2017-08-24_12-39.map

[xavoro01]:~/tmp/allinea-forge-7.0.6-Ubuntu-14.04-x86_64/examples$ ../bin/map --profile mpirun -n 2 wave_c 5
Waiting for licence
Allinea Forge 7.0.6 - Allinea MAP

[e112475-lin:09338] [[1848,0],0] usock_peer_send_blocking: send() to socket 44 failed: Broken pipe (32)
[e112475-lin:09338] [[1848,0],0] ORTE_ERROR_LOG: Unreachable in file oob_usock_connection.c at line 316
[e112475-lin:09338] [[1848,0],0]-[[1848,0],0] usock_peer_accept: usock_peer_send_connect_ack failed
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------

I don't have time to deal with this one, so I'm assigning it others.

[rhc54]

One further piece of info:

Yes, it only happens under debugger control.
"mpirun -n ......" always work.
All the procs are on the same machine.