Support for HSDS/REST VOL in HighFive – interest in upstreaming?

[vincentmr]

Hi folks,
We're a company building large-scale physics simulators, and we've been happily using HighFive for C++ HDF5 bindings. As we port parts of our stack to an online compute platform, we've added support for HSDS via the REST VOL.
To enable this, we’ve forked HighFive and made some patches here to get things working. Before we go further, we wanted to ask: would you be interested in incorporating support for REST VOL in the official HighFive releases?
If so I could polish the PR (e.g. setting up CI with REST VOL), improve some features (like containers of containers of ... std::string's), and ensure everything aligns with your coding conventions and contribution guidelines.
There aren't that many options out there for HDF5 C++ bindings + REST VOL support so I thought this could be a useful addition for the community. Let us know your thoughts!
Cheers.

[1uc]

It sounds like a nice feature and I'm glad you got it working.

Three comments (to start):

1. There's a lot of #ifdef happening. Does this mean that the choice of using REST VOL must be made at compile time? If yes, would this prevent applications from reading both regular HighFive files and HSDS files? This will then lead to questions like are the limitations imposed on us by HDF5 or are they a result of the implementation? The former means we might need to use inheritence or something pimpl like to be able to create HSDS File, DataSet, etc. The latter would likely require to keep refining until the #ifdefs vanish.

2. Why are strings so problematic?

3. As a preview of the types of things that I would like to have differently: The files h5?_wrapper contain an extremely simplistic, mechanical and dumb wrapper of the C-API. They have one task: call the HDF5 function with a similar name and check the error code. Nothing else. I don't want to pollute this layer with complex logic. This makes it easy to see when we're not checking an error code, and makes it easier to review if we're checking the correct condition.

I seems like a worthwhile feature, but I'm skeptical if it's close to mergeable.

[vincentmr]

Thanks for the feedback @1uc ; just saw this now as it appears the notification got lost in the spam.

I seems like a worthwhile feature, but I'm skeptical if it's close to mergeable.

100% agree on both counts. That’s why I’m reaching out, I’d like to understand what direction would make this contribution mergeable from your point of view.

1. HDF5 itself requires the VOL connector to be configured either via environment variables or through FAPL property lists before opening a file. Once a file is open, the VOL is effectively opaque to HighFive's abstraction layer. In our current setup, we went with the simpler route: assume the environment is correctly configured (e.g., via HDF5_PLUGIN_PATH, HDF5_VOL_CONNECTOR, etc.), and that users avoid relative file paths. In this case, things “just work” with REST VOL, albeit with some limitations. You can see how we configure this in .devcontainer/Dockerfile. That said, we could certainly move toward a cleaner abstraction - for instance, using FAPLs explicitly or isolating the backend differences using polymorphism or traits. I fully agree that the current #ifdef sprawl is not sustainable long-term and I’m open to refactor it if you think a particular design direction makes more sense.
2. I ran into trouble with variable-length strings via the C interface - HSDS doesn’t seem to handle them cleanly. So, as a workaround, I converted std::string and std::vector<std::string> to fixed-length strings internally. This is hacky so will have to be revisited for sure, but it got me past the std::string and std::vector<std::string> tests.
3. A few C-API primitives simply aren’t supported by HSDS (e.g., H5Dget_offset). In such cases, I threw exceptions directly from the h5*_wrapper layer. Likewise, for functions like H5Ldelete, which behave differently under HSDS (e.g., returning 0 instead of -1 if a link doesn’t exist), I added a call to h5l_exists to preserve expected behavior. In this case, it returns 0 instead of -1 if the link does not exists (which wouldn't throw when it should). I can move all such logic higher up the stack, or refactor this however you see fit.

The current patch is rough, it was built to explore feasibility and unblock our development. But I think REST VOL support in HighFive could be valuable and I’d be glad to work toward something that’s clean and upstream-worthy. Would it make sense to start by formalizing the compile-time option + environment setup approach, and build from there?

[1uc]

I think the most important aspect would be:

1. to make sure that HighFive with HSDS can still read regular files,
2. unless impossible, the ifdefs need to be eliminated.

The reason I think we need both is that HighFive is almost popular enough that a very unlucky user might be dependent on two libraries that themselves use HighFive. Having two versions of HighFive that use the same symbols in incompatible ways, seems like a way to create and unsolvable case of dependency hell.

I've too little experience (i.e. heard one short presentation) with HSDS to be sure, but the rest of HDF5, works through setting property lists, and calling well-defined APIs. For example, one configures the file to have properties, X, Y and Z. But then still calls H5Dread (independent of what X, Y and Z were). It's unfortunate (and surprising) that HSDS support is this invasive. One example is the H5Dget_offset, while I can vaguely understand why there can't be an offset, it's surprising that they don't simply return HADDR_UNDEF (the error code). If they did HighFive would convert the error code to an exception. Then if someone calls H5Dget_offset in a context it's not permitted they get a runtime error (no special code or ifdefs required).

Some of the arguments sound like HSDS is buggy. Which given how new it is, seems not entirely impossible.

@vincentmr on the one hand I'm very happy to see interest in HighFive and happy to have other developers involved. At the same time, I really don't want to get into a situation where you've worked for weeks of months on polishing the PR; and I feel I have to reject it because I'm concerned it would break for other users or because it would add to much "tech debt".

I think the best way forward that protects you from wasting time, is if you continue using and refining the fork; but you do so as and when you need it for your company or otherwise suits you. It'll either get cleaner as you go; messier (because HSDS is even more invasive than thought); or just stay the way it is. If it naturally reaches a cleaner state, submit it again :) If you see that there's 10 or 20 thumbs ups or hearts on your issue we know there's real interest and we can revisit it more urgently. We might get lucky and by the time there's real interest in HSDS with HighFive the quirky behaviour you saw in HSDS will have sorted itself out.