Segfault in mca_io_romio321.so

[amckinstry]

Background information

This is with 4.1.2~rc1 on multiple architectures on Debian unstable (development branch).
It is triggered by the tests in mpi4py:

eg.
https://buildd.debian.org/status/fetch.php?pkg=mpi4py&arch=ppc64el&ver=3.1.1-8&stamp=1632692000&raw=0

The source as installed is at:
https://sources.debian.org/src/openmpi/4.1.2%7Erc1-2/
For configuration information, see the rules file:
https://sources.debian.org/src/openmpi/4.1.2%7Erc1-2/debian/rules/

[edgargabriel]

The romio segfault looks legit, but there is another issue here, namely why was romio used instead of ompio.

[ppc64el-osuosl-01:1381766] mca_base_component_repository_open: unable to open mca_io_ompio: libmca_common_ompio.so.41: cannot open shared object file: No such file or directory (ignored)

Not sure whether to open another ticket for this or not. Will try to reproduce.

[edgargabriel]

At least for the missing libmca_common_ompio problem, the issue seems to be in your scripts:
---snip---(file debian/rules line 275)

if test -f $(DESTDIR)/$(LIBDIR)/openmpi/lib/libmca_common_ompio.so.41.29.1; then
dh_install -p libopenmpi3 $(LIBDIR)/openmpi/lib/libmca_common_ompio.so.41.29.1 $(LIBDIR) ; \

---snip---

the file generated on my system is however libmca_common_ompio.so.41.29.2

[ggouaillardet]

@amckinstry Thanks for the report!

I am unable to reproduce the issue on my CentOS 7 x86_64 VM.

Is this issue specific to ppc64le?

FWIW

[gilles@ws mpi4py-3.1.1]$ python3 test/runtests.py -v -i test_io
[0@ws] Python 3.6 (/usr/bin/python3)
[0@ws] MPI 3.1 (Open MPI 4.1.2)
[0@ws] mpi4py 3.1.1 (build/lib.linux-x86_64-3.6/mpi4py)
testIReadIWrite (test_io.TestIOSelf) ... ok
testIReadIWriteAll (test_io.TestIOSelf) ... ok
testIReadIWriteAt (test_io.TestIOSelf) ... ok
testIReadIWriteAtAll (test_io.TestIOSelf) ... ok
testIReadIWriteShared (test_io.TestIOSelf) ... ok
testReadWrite (test_io.TestIOSelf) ... ok
testReadWriteAll (test_io.TestIOSelf) ... ok
testReadWriteAllBeginEnd (test_io.TestIOSelf) ... ok
testReadWriteAt (test_io.TestIOSelf) ... ok
testReadWriteAtAll (test_io.TestIOSelf) ... ok
testReadWriteAtAllBeginEnd (test_io.TestIOSelf) ... ok
testReadWriteOrdered (test_io.TestIOSelf) ... ok
testReadWriteOrderedBeginEnd (test_io.TestIOSelf) ... ok
testReadWriteShared (test_io.TestIOSelf) ... ok
testIReadIWrite (test_io.TestIOWorld) ... ok
testIReadIWriteAll (test_io.TestIOWorld) ... ok
testIReadIWriteAt (test_io.TestIOWorld) ... ok
testIReadIWriteAtAll (test_io.TestIOWorld) ... ok
testIReadIWriteShared (test_io.TestIOWorld) ... ok
testReadWrite (test_io.TestIOWorld) ... ok
testReadWriteAll (test_io.TestIOWorld) ... ok
testReadWriteAllBeginEnd (test_io.TestIOWorld) ... ok
testReadWriteAt (test_io.TestIOWorld) ... ok
testReadWriteAtAll (test_io.TestIOWorld) ... ok
testReadWriteAtAllBeginEnd (test_io.TestIOWorld) ... ok
testReadWriteOrdered (test_io.TestIOWorld) ... ok
testReadWriteOrderedBeginEnd (test_io.TestIOWorld) ... ok
testReadWriteShared (test_io.TestIOWorld) ... ok

----------------------------------------------------------------------
Ran 28 tests in 40.097s

OK

[drew-parsons]

No, it's affecting nearly all architectures, see https://buildd.debian.org/status/package.php?p=mpi4py
I can reproduce it easily on amd64 with openmpi debian package 4.1.2~rc1-2.

There's some more details at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995150
e.g. gdb backtrace (running test_io.py only)

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007fffad13ea84 in MPIOI_File_iwrite_all (fh=<optimized out>, offset=offset@entry=0, file_ptr_type=file_ptr_type@entry=101, buf=<optimized out>, count=13, datatype=0x7ffff3d27700 <ompi_mpi_signed_char>, 
    myname=0x7fffad17be80 <myname> "MPI_FILE_IWRITE_ALL", request=0x7fffad57ca10) at ../../../../../../../ompi/mca/io/romio321/romio/mpi-io/iwrite_all.c:124
#2  0x00007fffad13ebd3 in mca_io_romio_dist_MPI_File_iwrite_all (fh=<optimized out>, buf=<optimized out>, count=<optimized out>, datatype=<optimized out>, request=<optimized out>)
    at ../../../../../../../ompi/mca/io/romio321/romio/mpi-io/iwrite_all.c:58
#3  0x00007fffad13c4c3 in mca_io_romio321_file_iwrite_all (fh=<optimized out>, buf=<optimized out>, count=<optimized out>, datatype=<optimized out>, request=<optimized out>)
    at ../../../../../../ompi/mca/io/romio321/src/io_romio321_file_write.c:203
#4  0x00007ffff3c86b4c in PMPI_File_iwrite_all (fh=0x1911f40, buf=0x7fffad6072b0, count=13, datatype=<optimized out>, request=request@entry=0x7fffad57ca10) at pfile_iwrite_all.c:83
#5  0x00007ffff3e80d0c in __pyx_pf_6mpi4py_3MPI_4File_62Iwrite_all (__pyx_v_buf=<optimized out>, __pyx_v_self=0x7fffacf8d990) at src/mpi4py.MPI.c:159094
#6  __pyx_pw_6mpi4py_3MPI_4File_63Iwrite_all (__pyx_v_self=<mpi4py.MPI.File at remote 0x7fffacf8d990>, __pyx_args=__pyx_args@entry=(<array.array at remote 0x7fffad57c930>,), __pyx_kwds=__pyx_kwds@entry=0x0)
    at src/mpi4py.MPI.c:27957
#7  0x000000000052b141 in method_vectorcall_VARARGS_KEYWORDS (func=<optimized out>, args=0x1a406b8, nargsf=<optimized out>, kwnames=0x0) at ../Objects/descrobject.c:346
#8  0x00000000005127fb in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x1a406b8, callable=<method_descriptor at remote 0x7ffff3f05450>, tstate=0x968fb0)
    at ../Include/cpython/abstract.h:118
#9  PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x1a406b8, callable=<method_descriptor at remote 0x7ffff3f05450>) at ../Include/cpython/abstract.h:127
#10 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x968fb0) at ../Python/ceval.c:5075
#11 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3504
#12 0x00000000005291c3 in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x1a404e0, for file /projects/python/build/mpi4py/test/test_io.py, line 306, in testIReadIWriteAll (self=<TestIOSelf(_testMethodName='testIReadIWriteAll', _outcome=<_Outcome(expecting_failure=False, result=<TestCaseFunction(keywords=<NodeKeywords(node=<...>, parent=<UnitTestCase(keywords=<NodeKeywords(node=<...>, parent=<Module(keywords=<NodeKeywords(node=<...>, parent=<Session(keywords=<NodeKeywords(node=<...>, parent=None, _markers={'mpi4py': True}) at remote 0x7fffb6a28b80>, own_markers=[], extra_keyword_matches=set(), testsfailed=1, testscollected=28, shouldstop=False, shouldfail=False, trace=<TagTracerSub(root=<TagTracer(_tags2proc={}, _writer=None, indent=0) at remote 0x7ffff7865be0>, tags=('collection',)) at remote 0x7fffb6a28a30>, startdir=<LocalPath(strpath='/projects/python/build/mpi4py/test') at remote 0x7fffb6a28ac0>, _initialpaths=frozenset({<LocalPath(strpath='/projects/python/build/mpi4py/test/test_io.py') at remote 0x7fffb69e7a60>}), _bestrelpathcache=<_best...(truncated), 
    tstate=0x968fb0) at ../Include/internal/pycore_ceval.h:40

Debian unstable has just upgraded to libc6 2.32, though I'm not sure that that's relevant.

[drew-parsons]

for the missing libmca_common_ompio problem, the issue seems to be in your scripts

The missing file shows also in the amd64 log, https://buildd.debian.org/status/fetch.php?pkg=mpi4py&arch=amd64&ver=3.1.1-8&stamp=1632748520&raw=0 , so it sounds like fixing the openmpi debian/rules as you suggested will get mpi4py passing tests again.

[jsquyres]

@amckinstry Should we close this issue here on the Open MPI side, since it looks like the issue is in the debian packaging?

[ggouaillardet]

Thanks @drew-parsons for the backtrace!
I was indeed incorrectly testing ...

Here is what happens:

• because of a Debian packaging error, io/romio321 is selected instead of the default io/ompio
• there is indeed a crash in io/romio321 because ROMIO asynchronous collectives have not been "ported" to Open MPI
• this is not a new bug, but is likely evidenced by the packaging issue and/or new mpi4py tests

From now:

• fixing the Debian packaging issue will have Open MPI use the io/ompio component (unless running on Lustre?) and hence hide the issue
• I currently do not have the bandwidth to implement the missing bits in the v4.1.x series
• I will check the status in the v5.0.x series

So the bug report is legit and the root cause is not in the Debian packaging.

[amckinstry]

There is a bug in the Debian packaging (which I'm currently fixing in the next upload).
I will leave this bug report open for the addition problem @ggouaillardet points out.

I'm happy to see this bug merged with another or closed.

[jsquyres]

@ggouaillardet with the merge of #8371 is this issue now resolved?

[roblatham00]

Huh. Romio is supposed to set some flags at configure time to not use grequest extensions . It should be a one or two line fix to romios "built as part of openmpi" case. I will look more closely at this in the morning

[ggouaillardet]

Thanks @roblatham00

What is the expected behavior if request extensions are not available/used?
Does ROMIO still implements non blocking collectives?

If not, this is more of an Open MPI integration issue: we still have to implement the "frontend" (e.g. MPI_File_iwrite_all()) but should do a better job at supporting a backend (e.g. ROM-IO) that does not implement/support such primitives.
I looked at it a few years ago, so maybe things have changed quite a lot in that area!