OPAL ERROR: Error in file %s/opal/mca/pmix/s1/pmix_s1.c at line 578

[morrisonlevi]

Background information

I am trying to get OpenMPI and Slurm working working with PMIx, but have been hitting issues in various places.

What versions of software are you using?

• OpenMPI 3.1.2
• Slurm 17.11.9
• PMIx 2.1.3

All three of these were built from source.

PMIx is installed to /usr/local/pmix/2.1 and there is an ld.so.conf.d file that includes its libdir in the ld.so.cache.

Slurm is installed to /usr/include/slurm, /usr/lib64/slurm, etc.

OpenMPI is configured with:

'--prefix=/apps/openmpi/3.1.2/gcc-7.3.0_cuda-9.2.88_pmix-2.1.3'
'--enable-wrapper-rpath=no'
'--enable-wrapper-runpath=no' '--with-slurm'
'--with-pmix=/usr/local/pmix/2.1'
'--with-pmi=/usr/local/pmix/2.1'
'--with-pmi-libdir=/usr/local/pmix/2.1/lib'
'--with-cuda=/apps/cuda/9.2.88'
'--with-libevent=/usr'
'CPPFLAGS=-I/usr/local/pmix/2.1/include'
'LDFLAGS=-L/usr/local/pmix/2.1/lib'

Side note: I was unsure what to do for --with-pmi -- I have tried various other settings and this has been the version that has worked the best so far.

Side note 2: I got errors if I did not specific -I and -L in configure flags -- that shouldn't be necessary should it?

Please describe the system on which you are running

• Operating system/version: RHEL 7
• Computer hardware: Probably N/A
• Network type: Probably N/A

---

Details of the problem

If I launch an MPI program with srun --mpi=openmpi ... I get this error:

OPAL ERROR: Error in file %s/opal/mca/pmix/s1/pmix_s1.c at line 578

Which can be seen here. Then, instead of the program using n ranks there are n independent runs of the program. In this example there are 4 tasks all on one node; the issue manifests in other configurations too:

$ srun -n 4 --mpi=openmpi $prog
[m7-1-2:08013] OPAL ERROR: Error in file /apps/src/openmpi/src/openmpi-3.1.2/opal/mca/pmix/s1/pmix_s1.c at line 578
[m7-1-2:08014] OPAL ERROR: Error in file /apps/src/openmpi/src/openmpi-3.1.2/opal/mca/pmix/s1/pmix_s1.c at line 578
[m7-1-2:08012] OPAL ERROR: Error in file /apps/src/openmpi/src/openmpi-3.1.2/opal/mca/pmix/s1/pmix_s1.c at line 578
[m7-1-2:08015] OPAL ERROR: Error in file /apps/src/openmpi/src/openmpi-3.1.2/opal/mca/pmix/s1/pmix_s1.c at line 578
6.52178
8.49445
8.49241
8.49539

The last four numbers are the outputs from each of the four runs.

Doing srun --mpi=pmix does not exhibit this issue:

$ srun -n 4 --mpi=pmix $prog
2.94764

I am unsure how to debug this; what further information should I provide?

[rhc54]

@artpol84 Can you please advise?

You also might want to look at the instructions here: https://pmix.org/support/how-to/slurm-support/

Short summary:

• you need to ensure you configured Slurm with PMIx support.
• You don't want to include any --with-pmi flags for OMPI configure - you should only have the --with-pmix one.
• you need to execute srun with --mpi=pmix_v2

[artpol84]

@rhc54 I think that PMIx case works as expected and PMI1 case is broken (where --mpi=openmpi" is used).

What I would suggest is

• not to use "--mpi=openmpi". I'm not aware of the situation where you want to use it. At least I never use it myself. It was there for a while and might make sense to remove it. @rhc what do you think? Do you know why it was used?
• Instead, try "--mpi=none" (for PMI1) and "--mpi=pmi2".
• PMI1 is always turned on in Slurm. so the fact that app is failing with "--mpi=openmpi" is not expected by me, but trying with "--mpi=none" would help to clarify if we have an issue or not.

[morrisonlevi]

All of these result in the same error I posted:

• --mpi=none
• --mpi=openmpi
• --mpi=pmi2

However, I sometimes get errors like this as well:

m7-1-2:09410] mca_sharedfp_sm_file_open: Error, unable to open file for mmap: /tmp/ompi.m7-1-2.20497/jf.1234/0/4032x4032.grid.done_cid-0.sm

Note that this program does I/O with MPI and the file 4032x4032.grid.done is the name of the output file. The /tmp location is writeable.

---

If I can get --mpi=pmi2 working then I will be happy -- Intel MPI only seems to work with that setting anyway.

[rhc54]

@edgargabriel Any ideas on the OMPIO error?

[morrisonlevi]

I think the MPI-IO error has to do with the fact that it tries to run n independent processes, all of which are using the same file-name, instead of running n processes that are collaborating on the same problem.

[edgargabriel]

What file system is the application executing on, and do you know by any chance whether shared file pointer operations are used in the code (i.e. MPI_File_write/read_shared or MPI_File_write/read_ordered)?

[edgargabriel]

@morrisonlevi I have a bit problem understanding how within the same jobid you can have n independent processes. If that is however the case, you might be right. If the file system supports file locking, you could try to disable the sm component (e.g. --mca sharedfp ^sm), and ompio will select the sharedfp/lockedfile component instead of sharedfp/sm. If the code doesn't use shared file pointer operations at all, you could try to set the io_ompio_sharedfp_lazy_open flag to 1, and the entire sharedfp component stuff is not initialized at all.

[morrisonlevi]

I have a bit problem understanding how within the same jobid you can have n independent processes.

Because PMI isn't working right - it's launching them independently instead of collaboratively. So they all do the same thing with 1 rank each, instead of working on the same thing with all n ranks.

What file system is the application executing on?

It's panfs by panasas.

Do you know by any chance whether shared file pointer operations are used in the code?

There are MPI_File_read_at calls to read the "header", then MPI_File_set_view and MPI_File_read_all to read the actual data. Same process on the write-out.

[edgargabriel]

@morrisonlevi thanks, make sense. I suspect the problems will go away when the PMI issue is resolved. I am not truly familiar with the panfs file system, but you could try the solutions that I pointed out above, since your code does not seem to use shared file pointers.

That being said, I think there could be also a legitimate bug that I need to look at, since an application using MPI_COMM_SELF for reading the same file by many processes could face the same problem. I will try to debug that in the next couple of days.

[rhc54]

If you configure OMPI with --enable-debug and then put OMPI_MCA_pmix_base_verbose=10 in your environment, you should get some debug output when you run srun --mpi=none

[artpol84]

I think I see the issue:
--with-pmi=/usr/local/pmix/2.1

This is not correct. If you want to use Slurm's PMI2/PMI1 you need to point "--with-pmi" on the Slurm's libraries (usually in [/usr]/lib/ or [/usr]/lib64/)

[morrisonlevi]

Unfortunately I have already tried it the other way. I still get this same error. Tomorrow I will rebuild Slurm just to be completely sure and report all current configure flags.

[artpol84]

I would double-check. It was clearly an error and if you change it and nothing is changing - it's a sign that might be something was stale.
Can you double-check configure.log that the right library was selected.
And also run and post here result of

$ ldd <ompi-install-path>/lib/openmpi/mca_pmix_s1.so
$ ldd <ompi-install-path>/lib/openmpi/mca_pmix_s2.so

[artpol84]

In any case for future investigation, you should stick to Slurm libraries.

[morrisonlevi]

I have tried to thoroughly clean and then rebuild Slurm and OpenMPI. It seems that it was picking up artifacts from a previous build as suspected. Now both srun --mpi=none ... and srun --mpi=pmi ... with OMPI_MCA_pmix_base_verbose=10 set show that libpmi.so.0 could not be found:

[m7-1-2:07433] mca_base_component_repository_open: unable to open mca_pmix_s1: libpmi.so.0: cannot open shared object file: No such file or directory (ignored)

So I rebuilt with LDFLAGS="-Wl,-rpath,/usr/local/lib (which is where the Slurm's PMI libraries are at) and it all seems to be working.

I guess my only remaining question is: should OpenMPI add the --with-pmi-libdir path to the RPATH/RUNPATH for the things that need PMI? My guess is that if I had to specify the libdir then I probably need the rpath set.