Scale/Offset (and maybe other?) Filters

[bnlawrence]

(This is on the list for V1)

HDF5 has the ability to read and write data using scale-offset filters to remove the precision of the data stored and use this in a filter/compression pipeline. From the h5py documentation it is not obvious that a reading library like pyfive has to do anything to read this data - if all that has happened is that the precision has been cut, and then the data compressed, a decompression pipeline doesn't need to do anything new.

However that doesn't sound like a scale-offset filter at all, and if we look a bit further under the hood, e.g. hdf5 docs , we find that the algorithm is a bit more complicated, and that the minimum-bits and the minimum value need to be stored with the compressed data for decompression and post-decompression.

Next steps for doing this would be to get some tests, with some test integer and floating point data (definitely both) to compress. We'd need to see what was in the filter pipeline message, and understand what the H5Z_FILTER_SCALEOFFSET filter looks like. Registered filter details can be found here.

Note: Should we consider the n-bit filter(H5Z_FILTER_NBIT) at the same time? Will we get it for free during the implementation?

Note: In implementation, also consider to what extent we want to support the hdf5plugin (or something similar) and any blosc extras so we can deal with any blosc filtering operation?

Note: We don't think NetCDF4 supports any of this.

[kmuehlbauer]

Note: We don't think NetCDF4 supports any of this.

@bnlawrence Thanks for adding this. I'll just link the netcdf-c docs here, they can build in support for several plugins (including hdf5plugin), IIUC: https://docs.unidata.ucar.edu/netcdf-c/current/filters.html

[bmaranville]

For reading, the filters are handled at the HDF5 library level and netcdf doesn't even have to be aware of them. For writing, it does, of course, have to pass through the correct arguments to the underlying library to enable filters.

[bmaranville]

I think the algorithm might be a bit complicated... reimplementing in python might be tricky. This filter is included in the core libhdf5 C library, and the implementation is here: https://github.com/HDFGroup/hdf5/blob/hdf5_1_14_6/src/H5Zscaleoffset.c

It is not included in the external plugins at e.g. hdf5plugin for this reason.

[bnlawrence]

If we only have to decompress we might have a fighting chance :-) ... but yes, I think this entire piece of functionality will need a bit of time to think through and implement. It wont be a priority for us in the next few months, but it's definitely on the radar.