MPI I/O Crashes and Corruption in HDF5

[ax3l]

Background information

What version of Open MPI are you using?

3.1.3

Update: affects 2.X, 3.X and 4.0.0

Describe how Open MPI was installed

From source via spack or as modules build from source on various different HPC systems.

Please describe the system on which you are running

• Operating system/version: Debian 9.6 and derivates such as Ubuntu
• Computer hardware: Laptops and HPC
• Network type: local and remote (Ethernet and IB)

Details of the problem

We are currently reporting two parallel HDF5 issues that either crash on write or corrupt data. The issues only occur with OpenMPI, not with MPICH 3.3 which we used for comparison.

We (@ax3l, @psychocoderHPC) are not the upstream authors of HDF5, but wanted to inform and connect you, so you are aware of these issues as they might be rooted in the OpenMPI I/O layer somewhere.

Link to HDF5 parallel I/O write issues with reproducers:

• crash with some special-size struct types: https://forum.hdfgroup.org/t/cannot-write-more-than-512-mb-in-1d/5118
  -> common/ompio: fix a floating point division problem #6286 common/ompio: possible rounding issue #6287 (a specific OpenMPI I/O issue)
• data corruption: https://forum.hdfgroup.org/t/hdf5bug-h5fd-mpio-collective-chunking/5279
  -> opal/datatype: fix opal_convertor_raw() #6295 Provide a better fix for #6285. #6326 Improve LOOP datatype exchanges performances for sizes above the eage… #6172 (a general OpenMPI Datatype issue)

[edgargabriel]

Looking into the issues.

[edgargabriel]

The first issue (cannot write more than 512MB) is an easy fix. First, the test passes with all collective I/O components except for 'individual', which is however the selected one for 1-process scenarios.

The culprit is a floating point division of two very close, large values.

no _of_cyles = ceil(536870916/536870912)

should be 2, but is 1 if you use floats (which is what the individual component used). If you use double precision floating point values, the result is as expected 2 (which is what all other collective I/O components use). A fix is coming soon.

If you want to run the code before the next release is available, simply exclude the individual fcoll component for this scenario, e.g.

mpirun --mca fcoll ^individual -np 1 ./....

[ax3l]

Thank you for so quickly looking into this!
So the first issue is located at:

• https://github.com/open-mpi/ompi/blob/v4.0.0/ompi/mca/common/ompio/common_ompio_file_write.c#L112
• https://github.com/open-mpi/ompi/blob/v4.0.0/ompi/mca/common/ompio/common_ompio_file_read.c#L134

Jippey, well spotted! Do you think we can get this in 4.0.1?

[edgargabriel]

I will file the pr's this afternoon (after my class), but the decision on whether it still makes it into the 4.0.1 release is out of my hand unfortunately.

[ax3l]

No problem, probably needs backporting to the other stable release lines as well.

Do you have an idea about the second issue?

[edgargabriel]

not yet, I will look at it later today.

[ax3l]

Thanks a lot, much appreciated!