mip4py: Regression in GitHub Actions

[dalcinl]

mpi4py 's own nightly CI testing on GHA has been failing with ompi@main in the last three days.
However, I understand that OMPI's own CI had not failed, otherwise you would not have merged in the regression.

These are the full logs from the last failed run:
https://github.com/mpi4py/mpi4py-testing/actions/runs/11310080041/job/31454714552
This is the specific error, not the message prte-rmaps-base:all-available-resources-used in the output:

test_async_error_callback (test_util_pool.TestProcessPool.test_async_error_callback) ... 1 more process has sent help message help-prte-rmaps-base.txt / prte-rmaps-base:all-available-resources-used
Exception in thread Thread-7 (_manager_spawn):
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/work/mpi4py-testing/mpi4py-testing/build/lib.linux-x86_64-cpython-312/mpi4py/futures/_core.py", line 350, in _manager_spawn
    comm = serialized(client_spawn)(pyexe, pyargs, nprocs, info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/mpi4py-testing/mpi4py-testing/build/lib.linux-x86_64-cpython-312/mpi4py/futures/_core.py", line 1058, in client_spawn
    comm = MPI.COMM_SELF.Spawn(python_exe, args, max_workers, info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/mpi4py/MPI.src/Comm.pyx", line 2544, in mpi4py.MPI.Intracomm.Spawn
    with nogil: CHKERR( MPI_Comm_spawn(
mpi4py.MPI.Exception: MPI_ERR_UNKNOWN: unknown error

On closer inspection, I noticed a difference between mpi4py and open-mpi configuration:

mpi4py configures oversubscription via $HOME/.openmpi/mca-params.conf
https://github.com/mpi4py/mpi4py-testing/blob/master/.github/workflows/openmpi.yml#L101

Open MPI configures oversubscription in both $HOME/.openmpi/mca-params.conf and $HOME/.prte/mca-params.conf
https://github.com/open-mpi/ompi/blob/main/.github/workflows/ompi_mpi4py.yaml#L80

Looks like something has changed recently, and oversubscription settings in $HOME/.openmpi/mca-params.conf are being ignored. Was this change intentional or is it a regression?

[dalcinl]

@hppritcha Maybe this issue comes from #12853?

[hppritcha]

Highly likely that's the case.

[rhc54]

Unsure exactly what was included in that update, but I was immediately suspicious when that PR was filed because it did not include the corresponding update to the PMIx submodule. As previously noted, the two are tightly coupled - so you should never update one without updating the other.

Might not be the cause here, but it is the first thing I would try.

[dalcinl]

@hppritcha Please let me know once you open a PR. I would like to suggest to update the configuration of of oversubscription in this repo, such that it is in sync with what I do in mpi4py. I assume that OMPI wants all configuration in $HOME/.openmpi/mca-params.conf honored, including that of PRRTE. Am I right?

[hppritcha]

Hmm. well actually this is not a bug. See https://docs.open-mpi.org/en/v5.0.x/mca.html#mca-parameter-changes-between-open-mpi-4-x-and-5-x . You'll need to add rmaps_default_mapping_policy = :oversubscribe to your $HOME/.prte/mca-params.conf, at least for v5.0.x and current main.

The ompi mpi4py github action is kind of misleading here so i'll fix it to not use that deprecated mca param in the $HOME/.openmpi/mca-params.conf file.

[rhc54]

@hppritcha I suppose this opens the question I noted on your PR. We could register synonyms for the OMPI v4 form of the PRRTE params. Do we want to do so? Be a little bit of grunge work, but nothing all that bad. May not be possible for all params as they may not have a corresponding OMPI v5 version - or the values may conflict so that PRRTE doesn't understand them. But we could pick up a number of them, and certainly the "oversubscribe" one should be doable.

I've added a check to ensure we are actually processing the OMPI default param files - pretty sure we are, but I'll check. Just need to finish something else first.

[dalcinl]

So the OMPI documentation @hppritcha pointed to warns use to use THREE different locations to configure Open MPI operation. Are you guys sure you want to put that complexity on users?

I must add a clarification: the purpose of the line setting rmaps_base_oversubscribe = true is just to be able to test against branch v4.0.x, I know that setting is gone in v5.0.x and main, and that's the reason I also set maps_default_mapping_policy = :oversubscribe, but all that in the single config file $HOME/.openmpi/mca-params.conf.

Until a few days ago, oversubscription seemed to work with all branches, and as of today it is still working in v5.0.x. From what @rhc54 said, I understand OMPI from v5.0.x and main simply ignore the old v4.x setting (is that correct?).
Therefore it looks to me that v5.0.x is indeed reading the oversubscription setting from $HOME/.openmpi/mca-params.conf (otherwise my tests would fail), while the main branch lost such ability. Is my diagnosis correct?

[dalcinl]

Guys, I have a somewhat explicit question. Are PRRTE and PMIx (when used via/with Open MPI) supposed to read configuration from OMPI's config file $HOME/.openmpi/mca-params.conf? If that is the case, then this issue looks like a regression. If that is not the case, then I do not understand why mpi4py CI is not failing with ompi@v5.0.x due to oversubscription not being turned on (as mpi4py IS NOT using $HOME/.prte/mca-params.conf`).

[rhc54]

I have said several times that I will investigate it - just not a high priority at the moment when compared with other things.

[hppritcha]

I see @dalcinl . i'll take a further look at what may have caused this change in behavior on our main branch.

[rhc54]

So here is the problem. The change came about because of an OMPI reported issue regarding passing of MCA params. We got that fixed. However, what it exposed was that we were overlaying processing of OMPI, PRRTE, and PMIx params on top of each other, thereby causing confusion.

Long story short: the issue here is that mpirun doesn't know it is working with an OMPI app until after we have already processed the PRRTE and PMIx params, and after we have already initialized much of PRRTE (opened frameworks, etc). So it is far too late in the process to impact PRRTE settings.

This is why we have three param files - to ensure that the settings are applied before anything happens in their respective projects. I personally don't see a simple way around that one - I suppose you could add param file parsing in OMPI's mpirun code before it exec's prterun, though that might get a little klunky as you'd have to track mapping of PRRTE and PMIx param names (e.g., to recognize that "rmaps_base_default_mapping_policy" requires conversion to "PRTE_MCA_rmaps_base_default_mapping_policy).

[bosilca]

From a user perspective this is mind blowingly complicated and cumbersome. And I'm trying to be polite.