Performance issues when using oversubscription

[dalcinl]

I've been testing mpi4py against ompi/main using an automatic schedule that runs on a weekly basis on GitHub Actions.

The GitHub-hosted runners have only two virtual cores, but I run tests with up to 5 MPI processes turning on oversubscription (setting rmaps_default_mapping_policy = :oversubscribe within $HOME/.prte/mca-params.conf).
The current workflow file is here.

Up to January 29, the testsuite used to take around 4m 30s to run to completion [link].
Since February 5, the same testsuite needs around 1h 20m to finish [link].
Unfortunately, the actual logs are long gone, but at least you can see the elapsed time (look for the Test mpi4py (np=5) line).

Is this a regression or some expected change in behavior?
Is there a new configuration option I'm missing that would allow me to go back to the previous behavior?

[jsquyres]

@awlauria Can you answer @dalcinl's question? If there's a config change that is needed, this definitely needs to be documented for v5.0.x.

[devreal]

@dalcinl I don't remember any changes this year that could cause this but I had a patch a while back that added an option to control how MPI processes yield the CPU when idle. The default is still sched_yield but you can set --mca threads_pthreads_yield_strategy nanosleep to use the nanosleep call instead (and --mca threads_pthreads_nanosleep_time X to control the time slept; default is the minimum of a nanosecond, just to trigger the scheduler). This seems to do a better job at forcing processes to be rescheduled. Maybe that will help bring down your run times when oversubscribing. It shouldn't have any negative performance impact if not oversubscribing.

[awlauria]

I am also not aware of anything that went in that would cause such a regression in performance. This seems alarming though, and something that should be tracked down.

[dalcinl]

@devreal My main concern is that I made absolutely no change to the way I configure Open MPI (either compile time or runtime), and yet I get this huge performance regression. Perhaps it is related to some change in GitHub Actions and how the runners are setup or configured. In the mean time, I'll try your suggestion and report back on the outcome.

[dalcinl]

@devreal I tried your suggestion (with the default nanosleep time), but had no effect.

[dalcinl]

@devreal @awlauria After manually bisecting the issue 😓, I can tell for sure the regression is within Open MPI.

The last good commit is 3b4d64f [logs], running mpi4py testsuite on 5 MPI processes takes under 3 minutes.

The regression is introduced after merging #9097 (MPI Sessions PR) in commit 7291361 [logs] (cancelled after 9 minutes of running on 3 MPI processes).

@hppritcha From your recent interactions in other issues I submitted, I believe you may want to get involved in this one.

[devreal]

Thanks for checking. Let me see if I can reproduce that on my machine

[hppritcha]

looking in to this.

[hppritcha]

looks like a piece from 2b335ed disappeared with the sessions support