[mtt] segfault in shmem_verifier

[amaslenn]

Configuration

OMPI: 4.0.1rc1
MOFED: MLNX_OFED_LINUX-4.5-1.0.1.0
Module: hpcx-gcc (2019-02-27)
Test module: none
Nodes: orion x37 (ppn=28(x37), nodelist=clx-orion-[021,026-027,029-034,058-064,066-086])

MTT log: http://e2e-gw.mellanox.com:4080/hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/html/test_stdout_sz_oD0.txt

Cmd (reproduces with np=2):
oshrun -np 1036 --report-bindings -mca btl self --bind-to core -mca oshmem_proc_group_cache_size 10000 -x UCX_NET_DEVICES=mlx5_0:1 -mca sshmem sysv -mca sshmem_sysv_use_hp 0 -mca coll '^hcoll' -mca coll_hcoll_enable 0 -mca spml ucx -mca pml ucx -x UCX_TLS=dc_x -x UCX_IB_SL=1 -x UCX_DC_MLX5_TX_POLICY=rand -x UCX_UNIFIED_MODE=y --map-by node -x SHMEM_SYMMETRIC_HEAP_SIZE=5712M /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/install/bin/oshmem_test exec --no-colour --task=stride

Output:

OpenSHMEM Verification Tool ver."1.2.75"
**********************************
* Log file: none
* Host: clx-orion-021
* Output level: 4
* Log level: 0
**********************************

PASS   stride     short_iget     shmem_short_iget(): Retrieve strided data  of 'short' type.
PASS   stride     int_iget       shmem_int_iget(): Retrieve strided data  of 'int' type.
PASS   stride     long_iget      shmem_long_iget(): Retrieve strided data  of 'long' type.
PASS   stride     float_iget     shmem_float_iget(): Retrieve strided data  of 'float' type.
PASS   stride     double_iget    shmem_double_iget(): Retrieve strided data  of 'double' type.
PASS   stride     longlong_iget  shmem_longlong_iget(): Retrieve strided data  of 'longlong' type.
PASS   stride     longdouble_ige shmem_longdouble_iget(): Retrieve strided data  of 'longdouble' type.
PASS   stride     iget32         shmem_iget32(): Retrieve strided data  with 32-bit element size.
PASS   stride     iget64         shmem_iget64(): Retrieve strided data  with 64-bit element size.
[clx-orion-022:3640 :0:3640] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x20)
[clx-orion-021:22822:0:22822] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x20)
PASS   stride     iget128        shmem_iget128(): Retrieve strided data  with 128-bit element size.
==== backtrace ====
==== backtrace ====
 0 0x000000000000bd16 _dl_relocate_object()  :0
 1 0x000000000001476c dl_open_worker()  dl-open.c:0
 2 0x000000000000f8d4 _dl_catch_error()  :0
 3 0x0000000000013c8b _dl_open()  :0
 4 0x000000000013d012 do_dlopen()  dl-libc.c:0
 5 0x000000000000f8d4 _dl_catch_error()  :0
 0 0x000000000000bd16 _dl_relocate_object()  :0
 1 0x000000000001476c dl_open_worker()  dl-open.c:0
 2 0x000000000000f8d4 _dl_catch_error()  :0
 3 0x0000000000013c8b _dl_open()  :0
 4 0x000000000013d012 do_dlopen()  dl-libc.c:0
 5 0x000000000000f8d4 _dl_catch_error()  :0
 6 0x000000000013d0d2 __GI___libc_dlopen_mode()  :0
 7 0x000000000000fe03 pthread_cancel_init()  :0
 8 0x000000000000caa7 pthread_cancel()  ???:0
 9 0x00000000004b5aea wait_for_completion()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/strided/osh_strided_tc11.c:97
10 0x00000000004b5aea osh_strided_tc11()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/strided/osh_strided_tc11.c:49
 6 0x000000000013d0d2 __GI___libc_dlopen_mode()  :0
 7 0x000000000000fe03 pthread_cancel_init()  :0
 8 0x000000000000caa7 pthread_cancel()  ???:0
 9 0x00000000004b5aea wait_for_completion()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/strided/osh_strided_tc11.c:97
10 0x00000000004b5aea osh_strided_tc11()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/strided/osh_strided_tc11.c:49
11 0x0000000000406a79 __do_exec()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/osh_exec.c:287
11 0x0000000000406a79 __do_exec()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/osh_exec.c:287
12 0x0000000000406a79 proc_mode_exec()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/osh_exec.c:128
13 0x00000000004056fe main()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/osh_main.c:138
14 0x00000000000223d5 __libc_start_main()  ???:0
15 0x0000000000405911 _start()  ???:0

[amaslenn]

UCX emulation job is also affected: http://e2e-gw.mellanox.com:4080/mnt/lustre/users/mtt/scratch/shmem/20190227_232558_23622_317869_clx-orion-017/html/test_stdout_pYm9Eu.txt

OMPI: 4.0.1rc1
MOFED: MLNX_OFED_LINUX-4.5-1.0.1.0
Module: hpcx-gcc (2019-02-27)
Test module: none
Nodes: orion x22 (ppn=28(x22), nodelist=clx-orion-[017,035-045,047,049-057])

[yosefe]

check without -x UCX_DC_MLX5_TX_POLICY=rand -x UCX_UNIFIED_MODE=y

[amaslenn]

Reproduced without -x UCX_DC_MLX5_TX_POLICY=rand -x UCX_UNIFIED_MODE=y.

[yosefe]

Simpler repro:

$ salloc -p orion -N 1
$ module load hpcx-gcc
$ oshrun -np 1 --report-bindings -mca btl self -mca coll ^hcoll -mca spml ucx -x UCX_TLS=self /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/install/bin/oshmem_test exec --no-colour --task=stride:short_iput
[clx-orion-023:25290] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././././././././././././.][./././././././././././././.]
[clx-orion-023:25296:0:25296] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x20)

OpenSHMEM Verification Tool ver."1.2.75"
**********************************
* Log file: none
* Host: clx-orion-023
* Output level: 4
* Log level: 0
**********************************

==== backtrace ====
 0 0x000000000000bd16 _dl_relocate_object()  :0
 1 0x000000000001476c dl_open_worker()  dl-open.c:0
 2 0x000000000000f8d4 _dl_catch_error()  :0
 3 0x0000000000013c8b _dl_open()  :0
 4 0x000000000013d012 do_dlopen()  dl-libc.c:0
 5 0x000000000000f8d4 _dl_catch_error()  :0
 6 0x000000000013d0d2 __GI___libc_dlopen_mode()  :0
 7 0x000000000000fe03 pthread_cancel_init()  :0
 8 0x000000000000caa7 pthread_cancel()  ???:0
 9 0x00000000004b5aea wait_for_completion()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/strided/osh_strided_tc11.c:97
10 0x00000000004b5aea osh_strided_tc11()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/strided/osh_strided_tc11.c:49
11 0x0000000000406a79 __do_exec()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/osh_exec.c:287
12 0x0000000000406a79 proc_mode_exec()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/osh_exec.c:128
13 0x00000000004056fe main()  /hpc/scrap/users/mtt/scratch/shmem/20190228_011330_21138_317954_clx-orion-021/installs/HHi7/tests/verifier/tests-mellanox.git/verifier/osh_main.c:138
14 0x00000000000223d5 __libc_start_main()  ???:0
15 0x0000000000405911 _start()  ???:0
===================
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
oshrun noticed that process rank 0 with PID 25296 on node clx-orion-023 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------