How to use HighFive in a threadsafe manner? #675
-
I'm programming a data processing framework which needs to read different HDF5 files from different threads (only one thread per file). Everything worked just fine using a single thread, however, moving to multiple threads lead to a variety of error messages. I have then recompiled HDF5 with the I have now reimplemented the exact same logic using the C API which works just fine using multiple threads. This leads me to my question: How exactly does HighFive "support" threadsafe HDF5? Is there something I need to enable first or is the support limited to specific functions? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
Could you please upload the reproducer? |
Beta Was this translation helpful? Give feedback.
-
We've recently reviewed the issue and agree: HighFive is likely not thread safe. Some HighFive objects modify themselves without protecting those writes. We've removed the claims from the front page. Again within BBP we define a mutex and just liberally lock as soon as the code is heading towards HDF5 or HighFive. We've removed the claim about thread-safety from the README and hope to implement this feature at some point in the future. |
Beta Was this translation helpful? Give feedback.
We've recently reviewed the issue and agree: HighFive is likely not thread safe. Some HighFive objects modify themselves without protecting those writes. We've removed the claims from the front page.
Again within BBP we define a mutex and just liberally lock as soon as the code is heading towards HDF5 or HighFive.
We've removed the claim about thread-safety from the README and hope to implement this feature at some point in the future.