Skip to content

Basic RESTVOL support #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Basic RESTVOL support #2

wants to merge 14 commits into from

Conversation

vincentmr
Copy link
Collaborator

@vincentmr vincentmr commented May 30, 2025

Context:

The Highly Scalable Data Service (HSDS) is an open source REST-based solution for reading, writing, and sharing data within object-based storage environments. REST VOL is the HDF5 library plug-in that clients can use to connect with HSDS. As the documentation says, REST VOL allows "organizations to easily move their data to the cloud without a large expenditure in re-writing code." That said, several HDF5 features are missing and must be accounted for.

There aren't a lot (or any?) modern and publicly available C++ bindings libraries out there. With this PR, we'd like to bring a minimal level of REST VOL support into HighFive.

Description of the Change:

  • Introduce CMake option HIGHFIVE_ENABLE_RESTVOL to control REST VOL support.
  • At first, REST VOL requires C++ 17. This may not be a strict requirement, but we're using it in some work arounds.
  • Introduce CMake code block to detect and link rest-vol library.
  • Introduce macro HIGHFIVE_USE_RESTVOL to branch behaviour in the source code.
  • Add SliceException class.

This allows handling some limitations:

  • Variable-length strings. We turn std::strings into null-terminated fixed-length strings. This introduces a spurious null character when they are read back. In the tests it is truncated with trim_if_rest_vol.
  • H5O_info1_t/H5Oget_info1 (and no addr)
  • H5Dget_offset
  • Relative path for files. This is an issue for the tests, we currently wrap the file paths in to_abs_if_rest_vol which appends / if using REST VOL.
  • H5Dcreate2 appears to sometimes ignore H5Pset_create_intermediate_group, so we manually create the parent groups recursively.
  • H5Iget_file_id. For now we simply do not keep track of _file_obj, but we might try to pass it around in the future. This breaks getFile.
  • Slabs and selections.
  • The hid of scalar datasets appears corrupted, so we pass H5S_ALL to h5d_write.
  • H5Ldelete does not fail if the file doesn't exists.
  • H5Tget_nmembers

We adapted the basic tests and those of Eigen.

  • RESTVOL_DISABLED hides tests that might work, but do not in the PR.
  • RESTVOL_UNSUPPORTED hides tests that test features that are not supported by REST VOL.
  • delete_file_if_exists wraps around std::remove which doesn't work with REST VOL.
  • to_abs_if_rest_vol makes the file paths absolute if REST VOL is active.
  • trim_if_rest_vol removes null char from strings if REST VOL is active.

This tries to keep the original tests as intact as possible. In many places, we cannot call createDataSet/write for std::string since we need to turn them into fixed-length string first. We thus branch with HIGHFIVE_USE_RESTVOL and use the auto interface.

Benefits:

  • Enough REST VOL support to be useful.

Possible Drawbacks:

  • Nested containers of std::string not handled.
  • getFile broken.
  • Selections and slabs not supported.

@vincentmr vincentmr changed the title Fix file paths Basic RESTVOL support May 30, 2025
@vincentmr vincentmr marked this pull request as ready for review June 2, 2025 15:37
@1uc
Copy link

1uc commented Jun 3, 2025

A fix for the issue with XTensor you're seeing in "Unit Tests / OSX (macOS-14, 20)" has just been merged:

highfive-devs#36

Rebasing should fix the issue. (Please let me know if it didn't.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants