Add a function to produce a ManifestStore from HDF5 files

[maxrjones]

First draft of first steps for addressing #498, based on changes originally proposed in #490

[maxrjones]

FYI I'll return to this PR later to fix the failing test - the issue is that we now have support for multiple chunk_key_encodings but no way yet to specify which is used in the Zarr V3 array metadata.

[TomNicholas]

This seems good - you could potentially go the whole way to using open_zarr to implement loadable_variables inside the HDF reader in this PR. Then we do that for every other reader, then we factor that outside of the readers (i.e. reduce them to each just being a function).

[maxrjones]

This seems good - you could potentially go the whole way to using open_zarr to implement loadable_variables inside the HDF reader in this PR. Then we do that for every other reader, then we factor that outside of the readers (i.e. reduce them to each just being a function).

The abstraction described here is a necessary prerequisite. Basically we need a utility function like get_default_object_store(filepath) -> ObjectStore that goes from a valid URL to an ObjectStore instance in order to make the store argument in _create_manifest_store() optional.

While there's a lot of code in https://github.com/developmentseed/obstore/blob/main/obstore/python/obstore/fsspec.py that would help the development of that function, I think it's a big enough task to warrant its own PR.

Somewhat relatedly, I'm planning to take tomorrow off and don't want to slow you down. So, my vote would be for you or @sharkinsspatial to take over this PR if you'd want to make progress before Monday.

[maxrjones]

The typing error is because we don't have a way to handle "empty" chunk manifests in a ManifestArray. We need an analogous solution to the current approach of an empty numpy array in a xarray Variable, but I don't understand the circumstances requiring an empty chunk manifest well enough to propose a solution.

[TomNicholas]

Every staticmethod here aside from open_virtual_dataset may as well just be an actual function. You will need to do that anyway later to refactor to make the reader just one function.

[TomNicholas]

But you could leave that to do in #498.

[sharkinsspatial]

👍 That is the plan. I think @maxrjones intent here was to make a very small change set to demonstrate the feasibility of using ManifestStore and that we would refactor the structure in a PR for #498.

[TomNicholas]

Generally it's better to write tests that use higher-level public API instead of directly calling internals if you can help it, but that's just a nit.

[sharkinsspatial]

👍 I think the tests will be easier to structure when we move away from VirtualBacked base to something more functional and the returned ManifestStore will be simpler to introspect and reason about than a complete xr.Dataset.