List of issues with OMPI master+ULT+PartCom

[janciesko]

This tracks current issues for ULT- and PartCom support in OMPI.

ULT Support:

• Compilation fails on current master @ HEAD when configured to --with-threads={qthreads,argobots} with
  ../../opal/mca/threads/qthreads/threads_qthreads.h:29:10: fatal error: qthread.h: No such file or directory 29 | #include "qthread.h" Configure seems to be not setting -I/PATH_TO_ULT_LIB
• OMPI hangs when configured to --with-threads=argobots and argobots uses multiple streams and --map-by ppr:1:node thus using ucx and libevent. Requires https://github.com/shintaro-iwasaki/libevent/commits/2.0.22-abt in libevent. We need to merge that branch into libevent soon. Check why Qthreads works.
• MPI_Init_thread hangs on current master @ HEAD when configured to --with-threads={qthreads,argobots}. Maybe this is an issue in instance/instance.h after merging sessions-related code changes.
• UCX fails asserts on POW9 and ARM: Discussed this with Howard. An assert is failing in src/ucp/core/ucp_worker.c. I'll add more info once the latest OMPI version compiles.

Partitioned Communication:

• Non equal numbers of send- and receive partitions leads to deadlock. Looking at a backtrace, it seems that the receiving rank waits for the last receive partition. MPI Partix is a reproducer when setting DEFAULT_RECV_SEND_PARTITION_RATIO=1 to another value such as 2.

General improvements:

• Add ULT coverage to CI testing

Reproducers of all above:

git clone https://github.com/sandialabs/MPI-Partix.git
export PATH=$OMPI_INSTALL_PATH/bin
cmake ..  -DCMAKE_CXX_COMPILER=mpicxx -DQthreads_ROOT=$QTHREADS_INSTALL_PATH -DPartix_ENABLE_QTHREADS=ON
mpirun -np 2 --map-by ppr:1:node ./bench1

[jsquyres]

@shintaro-iwasaki Can you have a look at this?

[shintaro-iwasaki]

As I left Argonne a year ago, I might not be the right person for this, but please let me know anything I can help. Jan should have known it.

(Currently, Yanfei Guo is the POC on the Argobots side.)

[jsquyres]

@shintaro-iwasaki Gotcha. What's Yanfei Guo's github ID?

[shintaro-iwasaki]

@yfguo He is leading the MPICH project.

[jsquyres]

@yfguo It looks like there are some errors with the qthreads implementation in Open MPI. @shintaro-iwasaki used to be the maintainer of this, but now -- perhaps by default? -- the ownership has apparently transferred to you...? We're having some compile issues as listed in this issue. Can you investigate?

[shintaro-iwasaki]

Sorry for interruption. I believe @janciesko owns the Qthreads/Open MPI part. For example, #8479 and #8500

I think Jan posted this issue to fix it later.

[jsquyres]

Ah, ok. @janciesko Can you also update the owner.txt files so that we don't ping the wrong people in the future? 😄

[awlauria]

@janciesko is there any update on this? We may look to disable these for the v5.0.0 release if not completed in time, since they are new features it would not be a regression. That said, we would love to have them. We're targeting an end of summer release, so end of August/September timeframe. Thanks!

[janciesko]

@awlauria, thanks for clarifying the time scope for this. I'll look into it and report back.

[janciesko]

@bwbarrett, can you please take a look why ompi fails to compile when enabling the use of Qtheads or Argobots?
The following commit introduced the error: 6767309.
Configure fails to set OPAL_QTHREADS_INCLUDE_PATH in ./opal/mca/threads/qthreads/threads_qthreads.h.in
Same behavior applies to Argobots.

[hppritcha]

I am working on a fix for the first item.