vader fails IMB-EXT Unidir_Get test

[mathbird]

Hi, I am testing OMPI with Intel IMB test “IMB-EXT Unidir_Get” as following. With “vader”, it gave error. Without “vader” or adding openib as “-mca btl openib,vader,tcp,self”, the test passed well. Does “vader” need special configure option to build? Does it need other options to make it work?

Thanks,

Dahai

mpirun -n 2 \
 -mca btl **vader**,tcp,self \
 imb_src/IMB-EXT Unidir_Get

#bytes #repetitions      t[usec]   Mbytes/sec
            0         1000         0.04         0.00
            4         1000         0.72         5.31
            8         1000         0.89         8.54
           16         1000         0.88        17.39
           32         1000         1.10        27.76
           64         1000         1.15        53.29
          128         1000         0.91       134.17
          256         1000         0.94       260.82
 Read -1, expected 18446744073709547536, errno = 22
 Read -1, expected 18446744073709547536, errno = 22
 Read -1, expected 18446744073709550864, errno = 22
          512         1000         0.95       516.29
         1024         1000         0.56      1731.87
         2048         1000         0.68      2892.14

[jjhursey]

We are also digging into this issue, but any insight/help would be appreciated. We have tracked it down to a handful of commits and are currently bisecting further to narrow it down to one. But it looks like the rcache / mpool rework might be the source.

The current thinking is that it's memory corruption. Adding some additional context to the error reported in vader here shows the size parameter as negative thus the errno of 22 (Invalid argument).

[jjhursey]

I'm using master from yesterday configured with --enable-debug using the IMB 4.1 benchmark. Running on a single machine with -np 2 can reproduce. We have been able to reproduce on both ppc64le and x86_64 platforms.

• Pass: -mca btl sm,self
• Pass: -mca btl tcp,self
• Fail: -mca btl vader,self

Here is the full trace of a failed run:

shell$ cd imb-4.1/src/
shell$  mpirun -np 2 -mca pml ob1 -mca btl vader,self ./IMB-EXT Unidir_get
#------------------------------------------------------------
#    Intel (R) MPI Benchmarks 4.1, MPI-2 part    
#------------------------------------------------------------
# Date                  : Thu Jul  6 10:39:28 2017
# Machine               : ppc64le
# System                : Linux
# Release               : 3.10.0-510.el7.ppc64le
# Version               : #1 SMP Wed Sep 21 14:46:20 EDT 2016
# MPI Version           : 3.1
# MPI Thread Environment: 

# New default behavior from Version 3.2 on:

# the number of iterations per message size is cut down 
# dynamically when a certain run time (per message size sample) 
# is expected to be exceeded. Time limit is defined by variable 
# "SECS_PER_SAMPLE" (=> IMB_settings.h) 
# or through the flag => -time 
  


# Calling sequence was: 

# ./IMB-EXT Unidir_get

# Minimum message length in bytes:   0
# Maximum message length in bytes:   4194304
#
# MPI_Datatype                   :   MPI_BYTE 
# MPI_Datatype for reductions    :   MPI_FLOAT
# MPI_Op                         :   MPI_SUM  
#
#

# List of Benchmarks to run:

# Unidir_Get

#---------------------------------------------------
# Benchmarking Unidir_Get 
# #processes = 2 
#---------------------------------------------------
#
#    MODE: AGGREGATE 
#
       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000         0.22         0.00
            4         1000         3.25         1.17
            8         1000         2.87         2.66
           16         1000         2.79         5.46
           32         1000         2.59        11.80
           64         1000         2.55        23.92
          128         1000         2.53        48.26
          256         1000         2.58        94.66
          512         1000         2.56       190.48
[c712f6n06:131799] Read -1, expected 18446744073709547536, errno = 22
[c712f6n06:131799] Read -1, expected 18446744073709547536, errno = 22
[c712f6n06:131799] Read -1, expected 18446744073709550864, errno = 22
         1024         1000         1.82       535.80
         2048         1000         2.03       963.12

[mathbird]

we found Unidir_Put works well with different options. so how does the vader code handle get and put differently?

[bwbarrett]

@jjhursey, are the patches in question also on the v3.0.x branch? I assume yes, but hoping no...

[jjhursey]

@bwbarrett yeah it impacts the v3.0.x branch. Here is the breakdown for the release branches (as of a build from yesterday):

• Pass v2.0.x
• Fail v2.x
• Fail v3.0.x
• Fail master

The leading suspect commit is the mpool/rcache rework (which is not in the v2.0.x branch):

• d4afb16

Valgrind shows some warnings in the MPI_Alloc_mem and MPI_Free_mem paths that might be related (best I can tell the key field is not initialized):

==85568== Conditional jump or move depends on uninitialised value(s)
==85568==    at 0x47AD480: mca_mpool_base_tree_node_compare (mpool_base_tree.c:62)
==85568==    by 0x46BD513: btree_insert (opal_rb_tree.c:342)
==85568==    by 0x46BCC8F: opal_rb_tree_insert (opal_rb_tree.c:137)
==85568==    by 0x47AD9AF: mca_mpool_base_tree_insert (mpool_base_tree.c:110)
==85568==    by 0x47AC76B: mca_mpool_base_alloc (mpool_base_alloc.c:87)
==85568==    by 0x41599AB: PMPI_Alloc_mem (palloc_mem.c:85)
==85568==    by 0x100055A7: IMB_set_buf (in /home/me/imb-4.1/src/IMB-EXT)
==85568==    by 0x1000631B: IMB_init_buffers_iter (in /home/me/imb-4.1/src/IMB-EXT)
==85568==    by 0x10002123: main (in /home/me/imb-4.1/src/IMB-EXT)
==85568== 

#---------------------------------------------------
# Benchmarking Unidir_Get 
# #processes = 2 
#---------------------------------------------------
#
#    MODE: AGGREGATE 
#
       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000         9.22         0.00
==85568== Conditional jump or move depends on uninitialised value(s)
==85568==    at 0x47AD468: mca_mpool_base_tree_node_compare (mpool_base_tree.c:58)
==85568==    by 0x46BCF1B: opal_rb_tree_find_with (opal_rb_tree.c:191)
==85568==    by 0x47AD41B: opal_rb_tree_find (opal_rb_tree.h:156)
==85568==    by 0x47ADB17: mca_mpool_base_tree_find (mpool_base_tree.c:143)
==85568==    by 0x47AC7DB: mca_mpool_base_free (mpool_base_alloc.c:110)
==85568==    by 0x41812CB: PMPI_Free_mem (pfree_mem.c:53)
==85568==    by 0x1000554B: IMB_set_buf (in /home/me/imb-4.1/src/IMB-EXT)
==85568==    by 0x1000631B: IMB_init_buffers_iter (in /home/me/imb-4.1/src/IMB-EXT)
==85568==    by 0x10002123: main (in /home/me/imb-4.1/src/IMB-EXT)