grpcomm errors when launching on RHEL 7.2/ssh

[jsquyres]

I'm seeing odd behavior when trying to launch small MPI jobs on master (as of Sun 13 Dec 2015, after @rhc54's update to pmix 1.1.2).

Here's the specs:

• RHEL 7.2
• TCP BTL
• ssh launcher (no SLURM or any other scheduler)
• (mostly) Default master build: ./configure --prefix=/home/jsquyres/bogus --with-libfabric=/home/jsquyres/bogus --with-usnic --disable-vt --disable-mpi-fortran
  • Yes, I built with libfabric/usnic, but I'm intentionally testing with the TCP BTL just to ensure something isn't wrong with the usnic BTL -- but I'm seeing the same behavior regardless of BTL selection

Here's what I'm launching:

$ mpirun --mca pml ob1 --mca btl tcp,vader,self --hostfile hosts -np 40 ring_c

The hostfile contains a bunch of lines like this: hostname slots=16

Sometimes that runs fine, sometimes it results in the following:

$ mpirun --mca pml ob1 --mca btl tcp,vader,self --hostfile hosts -np 40 ring_c
[pacini014.arcetri.cisco.com:08929] [[4261,0],3] ORTE_ERROR_LOG: Not found in file base/grpcomm_base_stubs.c at line 294
[pacini014.arcetri.cisco.com:08929] [[4261,0],3] ORTE_ERROR_LOG: Not found in file base/grpcomm_base_stubs.c at line 254  
[pacini014.arcetri.cisco.com:08929] [[4261,0],3] ORTE_ERROR_LOG: Not found in file grpcomm_brks.c at line 241  
malloc debug: Request for 4 zeroed elements of size -1 failed (grpcomm_brks.c, 92)
[pacini014.arcetri.cisco.com:08929] [[4261,0],3] ORTE_ERROR_LOG: Not found in file grpcomm_brks.c at line 170

FWIW, I observed this same behavior this past Thursday (i.e., before the pmix 1.1.2 update), but didn't have the time to file a proper bug report. This suggests that the problem might be unrelated to the old-vs.-new PMIX...?

Here's a gist of a failed run, but with lots of verbosity, in case it helps. Here's the command line used to launch that run:

$ mpirun \
    --mca ess_base_verbose 100 \
    --mca grpcomm_base_verbose 100 \
    --mca pmix_base_verbose 100 \
    --mca pml ob1 \
    --mca btl tcp,vader,self \
    --hostfile hosts \
    -np 40 \
    ring_c

[rhc54]

out of curiosity - how many hosts are in that hostfile?

[jsquyres]

$ wc hosts
  64  128 1216 hosts
$ head hosts
pacini012 slots=16
pacini013 slots=16
pacini014 slots=16
pacini015 slots=16
pacini016 slots=16
pacini017 slots=16
pacini018 slots=16
pacini019 slots=16
pacini020 slots=16
pacini021 slots=16

The remainder of the file is similar.

[rhc54]

@annu13 has identified the problem and is working on a solution.

@annu13 Can you provide some ETA?

[annu13]

On 12/16/15, 9:20 PM, "rhc54" notifications@github.com wrote:

@annu13 https://github.com/annu13 has identified the problem and is
working on a solution.
@annu13 https://github.com/annu13 Can you provide some ETA?
�
Reply to this email directly or
view it on GitHub
#1215 (comment).

I can have the fix ready next week. I guess this is not a blocking issue,
let me know if its otherwise I will try to fix asap.

[rhc54]

@annu13 That will be fine - it isn't a blocker. Just wanted to give folks some idea of when the fix might become available.

Thanks for tackling it!

[jsquyres]

Perfect, thanks.

[annu13]

@jsquyres could you check if PR 1254 fixes the issue.
#1254
This fix should avoid the race condition thats causing the errors.

[jsquyres]

@rhc54 @annu13 I'm sorry; I'm finally getting around to testing this, and it looks like #1254 (which looks like it was closed and re-submitted/merged as #1255) did not fix the issue when running MPI jobs. Note that running non-MPI jobs, like hostname and uptime works fine.

Here's what I'm seeing with the current master head (70787d1):

# ring_c is the simple ring program from the examples/ dir
$ mpirun -np 400 --hostfile hosts --mca btl tcp,vader,self ring_c
[pacini075.arcetri.cisco.com:13606] [[31700,0],25] ORTE_ERROR_LOG: Not found in file bas
e/grpcomm_base_stubs.c at line 294
[pacini075.arcetri.cisco.com:13606] [[31700,0],25] ORTE_ERROR_LOG: Not found in file bas
e/grpcomm_base_stubs.c at line 254
[pacini075.arcetri.cisco.com:13606] [[31700,0],25] ORTE_ERROR_LOG: Not found in file grp
comm_brks.c at line 304
*** Error in `orted': double free or corruption (out): 0x0000000000dc9c60 ***
======= Backtrace: =========
/usr/lib64/libc.so.6(+0x7cfe1)[0x7fb4d0990fe1]
/home/jsquyres/bogus/lib/libopen-pal.so.0(opal_free+0x1f)[0x7fb4d1ca182a]
/home/jsquyres/bogus/lib/libopen-rte.so.0(+0x66e59)[0x7fb4d1fc4e59]
/home/jsquyres/bogus/lib/openmpi/mca_grpcomm_brks.so(+0x11ac)[0x7fb4ce9bc1ac]
/home/jsquyres/bogus/lib/openmpi/mca_grpcomm_brks.so(+0x2ffb)[0x7fb4ce9bdffb]
/home/jsquyres/bogus/lib/libopen-rte.so.0(orte_rml_base_complete_recv_msg+0x18a)[0x7fb4d
1ffc290]
/home/jsquyres/bogus/lib/libopen-rte.so.0(orte_rml_base_process_msg+0x12f)[0x7fb4d1ffc97
a]
/home/jsquyres/bogus/lib/libopen-pal.so.0(opal_libevent2022_event_base_loop+0x8fc)[0x7fb
4d1cb50dc]
/home/jsquyres/bogus/lib/libopen-rte.so.0(orte_daemon+0x2339)[0x7fb4d1fa8366]
orted[0x400906]
/usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fb4d0935b15]
orted[0x4007b9]
[pacini075:13606] *** Process received signal ***
[pacini075:13606] Signal: Aborted (6)
[pacini075:13606] Signal code:  (-6)
[pacini075:13606] [ 0] /usr/lib64/libpthread.so.0(+0xf100)[0x7fb4d0ce4100]
[pacini075:13606] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x7fb4d09495f7]
[pacini075:13606] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x7fb4d094ace8]
[pacini075:13606] [ 3] /usr/lib64/libc.so.6(+0x75317)[0x7fb4d0989317]
[pacini075:13606] [ 4] /usr/lib64/libc.so.6(+0x7cfe1)[0x7fb4d0990fe1]
[pacini075:13606] [ 5] /home/jsquyres/bogus/lib/libopen-pal.so.0(opal_free+0x1f)[0x7fb4d
1ca182a]
[pacini075:13606] [ 6] /home/jsquyres/bogus/lib/libopen-rte.so.0(+0x66e59)[0x7fb4d1fc4e5
9]
[pacini075:13606] [ 7] /home/jsquyres/bogus/lib/openmpi/mca_grpcomm_brks.so(+0x11ac)[0x7
fb4ce9bc1ac]
[pacini075:13606] [ 8] /home/jsquyres/bogus/lib/openmpi/mca_grpcomm_brks.so(+0x2ffb)[0x7
fb4ce9bdffb]
[pacini075:13606] [ 9] /home/jsquyres/bogus/lib/libopen-rte.so.0(orte_rml_base_complete_
recv_msg+0x18a)[0x7fb4d1ffc290]
[pacini075:13606] [10] /home/jsquyres/bogus/lib/libopen-rte.so.0(orte_rml_base_process_m
sg+0x12f)[0x7fb4d1ffc97a]
[pacini075:13606] [11] /home/jsquyres/bogus/lib/libopen-pal.so.0(opal_libevent2022_event
_base_loop+0x8fc)[0x7fb4d1cb50dc]
[pacini075:13606] [12] /home/jsquyres/bogus/lib/libopen-rte.so.0(orte_daemon+0x2339)[0x7
fb4d1fa8366]
[pacini075:13606] [13] orted[0x400906]
[pacini075:13606] [14] /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fb4d0935b15]
[pacini075:13606] [15] orted[0x4007b9]
[pacini075:13606] *** End of error message ***
--------------------------------------------------------------------------
ORTE has lost communication with its daemon located on node:

  hostname:  pacini075

This is usually due to either a failure of the TCP network
connection to the node, or possibly an internal failure of
the daemon itself. We cannot recover from this failure, and
therefore will terminate the job.
--------------------------------------------------------------------------

I configured Open MPI fairly simply:

$ ./configure --prefix=/home/jsquyres/bogus --with-libfabric=/home/jsquyres/bogus --with-usnic --disable-oshmem --disable-mpi-fortran

(but this run didn't even use the usnic stuff -- just plain/vanilla TCP, to ensure that usnic isn't causing the failure)

[jsquyres]

Some notes I really should have included above:

1. It doesn't always segv like this when running MPI jobs -- sometimes it hangs.
2. This is running without a resource manager (no SLURM, etc.). Just a plain hostfile (same as I pasted above: slots=16 on 25 servers).
3. It feels like a race condition of some kind:
   • When I run with a smaller np (e.g., np=200), it usually works fine -- but still sometimes hangs.
   • np=400 always hangs or segvs.