How to use HighFive in a threadsafe manner?

[DominicWindisch]

I'm programming a data processing framework which needs to read different HDF5 files from different threads (only one thread per file). Everything worked just fine using a single thread, however, moving to multiple threads lead to a variety of error messages.

I have then recompiled HDF5 with the threadsafe option. Since this cannot be used in conjunction with the C++ library, I tried using HighFive instead. However, using HighFive lead to the same error messages.

I have now reimplemented the exact same logic using the C API which works just fine using multiple threads.

This leads me to my question: How exactly does HighFive "support" threadsafe HDF5? Is there something I need to enable first or is the support limited to specific functions?

[1uc]

Could you please upload the reproducer?

[1uc]

I can expand slightly. To my knowledge we don't use the C++ library provided by HDF5. Hence, thread-safety should be inherited, unless you have a race condition related to modifying state of HighFive objects.

In our internal projects we typically use a mutex to serialize all access to HDF5. (To my knowledge the HDF5 thread-safe build is only thread-safe, but not concurrent. Essentially it too acquires an API lock early on and serializes all calls to HDF5.)

[DominicWindisch]

Thanks for the reply. I don't have the time to create a MWE at the moment. A few comments:
When compiling HDF5, one can only enable the C++ library OR the threadsafe OR the parallel option. I've compiled with threadsafe. So yes, HighFive does not rely on the C++ library provided by HDF5 because I did not even have this installed at the time.

Consequently, I can only guess that HighFive somehow manages/abstracts calls to the C API. Which - when compiled with threadsafe option - should be safe to use in multiple threads. It essentially serialized accesses, but thats fine in my case.

When I'm only accessing the C API from my program, everything works fine. There's a couple of threads doing essentially these steps (plus checking the return value of all HDF5 API calls to ensure the results are valid):

/* sino is a std::vector whose address is passed on to further processing stages after reading the data */

hid_t h5_file = H5Fopen("the/file/path.hdf5");
hid_t ds_rawData = H5Dopen(h5_file, "/rawData", H5P_DEFAULT);
hid_t sp_rawData = H5Dget_space(ds_rawData_);
auto ndims_rawData = H5Sget_simple_extent_ndims(sp_rawData_);
hsize_t dims_rawData[3], maxdims_rawData[3];
H5Sget_simple_extent_dims(sp_rawData_, dims_rawData, maxdims_rawData);
auto cparms_rawData = H5Dget_create_plist(ds_rawData_);
auto rank_chunk_rawData = H5Pget_chunk(cparms_rawData, 3, chunk_dims_rawData);
hid_t msp_rawData_ = H5Screate_simple(rank_chunk_rawData, chunk_dims_rawData, NULL);

for (auto i = 0; i < numFrames; i++) {
   auto sino = memorManager.getNewBuffer(); // takes a new pre-allocated buffer managed by a memory manager class
   offset_rawData_[2] = i;
   H5Sselect_hyperslab(sp_rawData_, H5S_SELECT_SET, offset_rawData_, NULL, count_rawData_, NULL);
   H5Dread(ds_rawData_, H5T_NATIVE_UINT16, msp_rawData_, sp_rawData_, H5P_DEFAULT, sino.data());
   /* tell other threads to process data in sino */
}

This works fine for multiple concurrent thread accessing different HDF5 files (one file per thread). But thats obviously quite a lot of boilerplate to read a part of the dataset, hence I initially tried to use HighFive for the same functionality.

Oddly enough, when running my logic within HighFive and a threadsafe HDF5 C lib, I get a variety of error messages (anything from segfaults to HDF5 itself warning about all kinds of access errors). From my understanding, HighFive should only access the C API which should be safe, so I'm somewhat puzzled as to how HighFive is able to create these race conditions.

[1uc]

We've recently reviewed the issue and agree: HighFive is likely not thread safe. Some HighFive objects modify themselves without protecting those writes. We've removed the claims from the front page.

Again within BBP we define a mutex and just liberally lock as soon as the code is heading towards HDF5 or HighFive.

We've removed the claim about thread-safety from the README and hope to implement this feature at some point in the future.