Support openmpi 5

[ClaudiaComito]

Related
conda-forge/heat-feedstock#15

Feature functionality
Verify that everything runs as expected with openmpi 5. So far, we're testing everything on 4.1.x

[mrfh92]

@ClaudiaComito I have run our GPU-tests on HAICORE on a single node with 4 MPI-processes and they were fine.
Modules:

1) dot   2) compiler/intel/2023.1.0   3) numlib/mkl/2022.0.2   4) devel/cuda/12.2 (E)   5) jupyter/minimal/2024-05-14   6) mpi/openmpi/5.0

More than one node cannot be requested in this configuration; thats why I have only run the single-node test.

[hint: instead of srun python ... you need to do srun --mpi=pmix python ... to make it run, at least in the current configuration of Slurm/OpenMPI on the system]

[mrfh92]

maybe @JuanPedroGHM can run them on two nodes?

[JuanPedroGHM]

Will give it a try, but it might take some time. Will try to squeeze it during next week's business trip.

[JuanPedroGHM]

Test results:

Horeka Partition: dev_accelerated
Python 3.11
OpenMPI 5.0
mpi4py 3.1.6
CUDA 12.2

Nodes	GPUs/Per Node	Ranks	Compiler	Device	Result
2	4	8	LLVM 17	CPU	Stuck at test_solver.py
2	4	8	LLVM 17	GPU	Stuck at test_solver.py
2	4	8	Intel 2023	CPU	Stuck at test_solver.py
2	4	8	Intel 2023	GPU	Stuck at test_solver.py
2	1	2	LLVM 17	CPU	Stuck at test_io.py
2	1	2	LLVM 17	GPU	Stuck at test_io.py
2	1	2	Intel 2023	CPU	Stuck at test_io.py
2	1	2	Intel 2023	GPU	Stuck at test_io.py
1	4	4	LLVM 17	CPU	481 Passed, 4 Skipped, 15 Warnings
1	4	4	LLVM 17	GPU	480 Passed, 5 Skipped, 12 Warnings
1	4	4	Intel 2023	CPU	481 Passed, 4 Skipped, 15 Warnings
1	4	4	Intel 2023	GPU	480 Passed, 5 Skipped, 12 Warnings

Multi-node is not working. Had to interrupt the run after some time, it got stuck unnaturally long when running on multiple nodes.

[ClaudiaComito]

@JuanPedroGHM does mpi4py 4 (#1618 ) change anything?

[JuanPedroGHM]

Have not had the time to test it with OpenMPI 5. I could give it a try tomorrow.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.