FAQ: MPI init failure when running wrapped MPI process

[SteVwonder]

If you attempt to launch an MPI process using a wrapper, it may fail. For example, if you run flux mini run my_script.py mpi_app.exe, where my_script.py Popens mpi_app.exe, the MPI init will fail, even if it is just a single rank MPI application. Same thing for flux mini run totalview mpi_app.exe. The issue is that Python's Popen and Totalview close file descriptors before launching the child process. MPI uses the PMI_FD file descriptor to communicate with Flux in order to bootstrap.

If you are attempting to debug your MPI application with Totalview, follow these instructions: https://flux-framework.readthedocs.io/en/latest/debugging.html#parallel-debugging-using-totalview

If you are attempting to wrap your MPI application with a Python script and Popen, then make sure to pass close_fds=False to Popen: https://docs.python.org/3/library/subprocess.html#subprocess.Popen