openmpi-5.0.5 won't spawn

[sukanka]

Background information

There may be a regression in openmpi-5.0 series

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v5.0.5, but in fact this regression has been there since 5.0.1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From archlinux repo

Please describe the system on which you are running

• Operating system/version: Arch Linux x86_64 Linux 6.11.6-zen1-1-zen

• Computer hardware: Laptop with AMD Ryzen 7 8845H w/ Radeon 780M Graphics (16) @ 5.10 GHz and NVIDIA GeForce RTX 4070 Max-Q / Mobile

• Network type: Wired

---

Details of the problem

The MWE provided in #11749 (comment) does not work with openmpi 5.0, but it does work with 4.1.6.

[hppritcha]

Would it be possible to test using the open MPI main branch?
Also, you may need to disable some collective components:

export OMPI_MCA_coll=^han,hcoll

They seem to get confused trying to handle the intercomm_merge call in the test case in #11749 .
Are you using the mpi4py testsuite to observe this problem? If so could you post the output from

python3 ./main.py -v -k

?

[rhc54]

FWIW: it works fine for me if you disable those components. Thanks @hppritcha !

[rhc54]

Should have added - it also works fine if you run it using mpirun instead of as a singleton, even with those components active. Seems to be something about singleton operations and those components?

[ggouaillardet]

@sukanka can you confirm you are testing the C program?
also, are you running in singleton mode (e.g. ./a.out) or via mpirun
mpirun works for me too.

[gilles@arch ~]$ pacman -Q openmpi openpmix
openmpi 5.0.5-2
openpmix 5.0.3-1
[gilles@arch ~]$ mpicc -o testspawn testspawn.c 
[gilles@arch ~]$ mpirun -np 1 ./testspawn
Hello, I am rank 1 in the merged comm
Hello, I am rank 3 in the merged comm
Hello, I am rank 2 in the merged comm
Hello, I am rank 0 in the merged comm
[gilles@arch ~]$

[rhc54]

FWIW: quick check indicates that it is hanging in the MPI_Barrier across the intercomm. Removing that line, or disabling han,hcoll, resolves the problem.

[rhc54]

@ggouaillardet You only see the problem in singleton mode.

[ggouaillardet]

@rhc54 I only see the problem in singleton mode (and with coll/han), but wanted to confirm the reporter.

fwiw, in singleton mode, the workaround (credit is yours) is

$ OMPI_MCA_coll=^han,hcoll ./a.out`

[rhc54]

Actually, @hppritcha came up with that workaround.

You might want to check that the singleton is finding the modex info for the child procs - I'm guessing that it doesn't and hangs in the modex_recv call. But that is just a (somewhat educated) guess. Would, however, explain why it all works when run under mpirun. Not sure what the coll/han components have to do with it - maybe something about how they call modex_recv can lead them to block?

[rhc54]

Ohhhh...you know what? That singleton "pushes" its connection info during MPI_Init, which since it was not started by mpirun, means that the connection info goes nowhere. It then kicks off a background mpirun to shepherd the child job - but that mpirun never gets the singleton's connection info! So now when we try to pass the connection info around, it is missing the singleton's connection info.

Thus, it is likely that the singleton "knows" how to connect to the child job - but the child job has no connection info for the singleton parent. Might be something worth checking.

[sukanka]

@sukanka can you confirm you are testing the C program? also, are you running in singleton mode (e.g. ./a.out) or via mpirun mpirun works for me too.

Yeah, I'm testing the C program and running in singleton mode. ( mpirun works for me.)

➜  test ./a.out
Hello, I am rank 0 in the merged comm
Hello, I am rank 2 in the merged comm
Hello, I am rank 3 in the merged comm
Hello, I am rank 1 in the merged comm
# stuck here

openmpi 5.0.5-2
openpmix 5.0.3-1

And the workaround OMPI_MCA_coll=^han,hcoll ./a.out works too.

[rhc54]

Ohhhh...you know what? That singleton "pushes" its connection info during MPI_Init, which since it was not started by mpirun, means that the connection info goes nowhere. It then kicks off a background mpirun to shepherd the child job - but that mpirun never gets the singleton's connection info! So now when we try to pass the connection info around, it is missing the singleton's connection info.

Thus, it is likely that the singleton "knows" how to connect to the child job - but the child job has no connection info for the singleton parent. Might be something worth checking.

Well, I stand corrected - there is a call to PMIx_Commit in the dpm code that pushes the connection info up to the DVM after it is fork/exec'd. So I have nothing further I can contribute - looks like something in han/hcoll.

[bosilca]

That fix makes little sense, not saying it does not work just saying it might looks like addressing the bug but that's not what it does. Put it simply hcoll can only work for special hardware setups, and both of these collective components disable themselves for intercoms (check the comm_query).

[hppritcha]

Well George I recommended that because in gdb traceback i saw a bunch of ranks blocked in some kind of hcoll calls. probably when doing an allreduce for a cid for some part of the merge operation. Its likely the user doesn't even have that installed (which is probably a good thing).

[rhc54]

I know nothing about the coll system any more, but FWIW everything runs fine if I simply remove the MPI_Barrier call. It seems that the barrier "hangs" if one of the intercomm members is a singleton - but works fine if all members are not singletons.

I saw no problems getting thru the intercomm merge operation.

[hppritcha]

i'll take a look in to this.