[Feature]: Native NetCDF-4/HDF5 reading, without GDAL and in the browser: how would you like it architected? #396

Before you start

• I searched the existing issues and the roadmap and did not find a duplicate request.

What problem does this solve?

#77 lists "native NetCDF-4/HDF5" as still missing under multidimensional rasters. Today these files open through GDAL (#117), so only on the desktop, only with GDAL installed, and only as gridded rasters.

Some context on why we're asking: we're building this for RAS Commander, our open-source Python library for HEC-RAS automation. HEC-RAS stores its geometry and results (2D mesh water surfaces, velocities, time series) in HDF5. GDAL can't present those as rasters, because the values are per mesh cell rather than on a grid. In the meantime we plan to ship an LLM-agent plugin that uses the existing Python library from RustGIS, so HEC-RAS users can work in RustGIS before any core change lands.

Native HDF5 and NetCDF reading would reach much further than HEC-RAS: the PACE, EMIT, ECOSTRESS, NEON and ERA5 files your HyperCoast and leafmap notebooks use, and NOAA's GOES, VIIRS and National Water Model output. It would work without GDAL, in the browser too, and fill in the multidimensional row of #77 next to Zarr.

We'd like to do the work. The only question is how you'd prefer it architected in RustGIS, so we'd rather ask before writing anything.

Proposed solution

We tested the most promising pure-Rust option on the sample files from opengeos/datasets and a few public NOAA files:

• hdf5-pure 0.47.0 for HDF5 and NetCDF-4.
• netcdf3 0.6.1 for classic NetCDF. Two of the samples (air_temperature.nc, era5_monthly_temperature_2023.nc) are NetCDF-3, which isn't HDF5 at all.

Every dataset was compared byte for byte with h5py, or with scipy for NetCDF-3.

Totals: 315 HDF5 datasets, about 7.8 GiB decoded, all identical. Attribute counts match on all 370 objects.

HEC-RAS files: 16,889 of 16,889 datasets identical across 80 plan and geometry files from HEC-RAS 5.0.7 to 7.0.1, including 555 MB plans.

Browser builds: hdf5-pure builds for wasm32-unknown-unknown. A minimal reader adds about 516 KB (192 KB gzipped).

Range reads: it reads files piece by piece, so it fits the remote.rs read-and-retry model. Showing one time step of a 555 MB HEC-RAS plan by URL took about 25 round trips and under 5 MB of transfer.

Gaps we found:

• Dimension links: attrs() skips NetCDF-4's DIMENSION_LIST attribute, a variable-length list of object references. The dimensions can still be recovered from _Netcdf4Coordinates and _Netcdf4Dimid.
• Filters: szip, zstd and blosc aren't implemented. None of these files used them.
• Dependency size: hdf5-pure is large (about 129k lines including its writer), young, and releases often, so we'd pin an exact version.

Three ways it could fit. We're happy with whichever you prefer:

1. Built into rustgis-data, next to Zarr. NetCDF and HDF5 variables become rasters, with the same variable and dimension pickers, CF dates and range reads as Zarr. This works on the desktop and the web, and GDAL stays the fallback for other formats.
   • 1a. Depend on hdf5-pure and netcdf3, pinned. Fastest; the results above are what we'd ship.
   • 1b. Write a focused reader in-house, like zarr.rs and sqlite.rs, with no new dependencies. It would be read-only, cap every size it reads, and use no unsafe. That's more work for us, because NetCDF-4 uses newer HDF5 structures than HEC-RAS does. The tests above would become its conformance suite.
2. The same, behind an optional cargo feature, the way geolibre is, so builds without it are unchanged.
3. An optional plugin. A WebAssembly plugin can't read files or byte ranges today, so this would need a small addition to the plugin interface. Either way, any HEC-RAS-specific logic (mesh results, ghost cells, plan metadata) would live outside the core, in our own crate or plugin, unless you'd rather have it in tree.

Which would you prefer? Or would you rather we approach this differently altogether? We'll follow your lead on structure, naming and scope, and can start with a small first PR (one NetCDF-4 variable as a raster) so you can judge it before anything larger.

Estimated scope

Medium: a broader feature that is feasible in a reasonable timeframe

Alternatives considered

• GDAL (GDAL rasters: JPEG 2000, NetCDF, HDF, ERDAS Imagine, HGT and more #117). It works today on the desktop. But it needs GDAL installed, it can't run in the browser, and it converts each variable to a GeoTIFF, which doesn't suit mesh data such as HEC-RAS results.
• libhdf5 bindings (hdf5-metno). Reliable, but C and desktop only, so the browser would still need a pure-Rust reader. That means two readers to keep in agreement.
• Other pure-Rust HDF5 crates (hdf5-reader, hdf5-rust, hdf5-io). Each failed on at least part of our HEC-RAS test set (unsupported datatypes or superblock versions).
• Converting first (Python to GeoParquet or Zarr). This already works with RustGIS today, but users then can't simply open the original file.

Our test scripts and the full results are available if they'd help.

Thank you for considering this, and for RustGIS!