Better compression algorithms for NetCDF

[crusaderky]

As of today, Dataset.to_netcdf() exclusively allows writing uncompressed or compressed with zlib.
zlib was absolutely revolutionary when it was released... in 1995. Time has passed, and much better compression algorithms have appeared over time.
Good news is, h5py supports LZF out of the box, and is extensible with plugins to support theoretically any other algorithm. h5netcdf exposes such interface through its new (non-legacy) API; however Dataset.to_netcdf(engine='h5netcdf') supports the legacy API exclusively.

I already tested that, once you manage to write to disk with LZF (using h5netcdf directly), open_dataset(engine='h5netcdf') transparently opens the compressed store.

Options:

• write a new engine for Dataset.to_netcdf() to support the new h5netcdf API.
• switch the whole engine='h5netcdf' to the new API. Drop support for the old parameters in to_netcdf(). This is less bad than it sounds, as people can switch to another engine in case of trouble. This is the cleanest solution, but also the most disruptive one.
• switch the whole engine='h5netcdf' to the new API; have to_netcdf() accept both new and legacy parameters, and implement a translation layer of parameters from the legacy API to the new API. The benefit here is that, as long as the user sticks to the legacy API, he can hop between engines transparently. On the other hand I have a hard time believing anybody would care.
• ?

[jhamman]

This is an interesting idea. I think something similar was discussed in #66.

The main problem I see is that current netCDF libraries don't support LZF so I think we're really talking about adding an HDF5 backend. In practice, this could use the new API of h5netcdf or h5py directly. Either way, my initial thought is that we want a to_hdf5() method on the dataset.

[petacube]

how about zstandard?

…

On Aug 28, 2017, at 7:22 PM, Joe Hamman ***@***.***> wrote: This is an interesting idea. I think something similar was discussed in #66 <#66>. The main problem I see is that current netCDF libraries don't support LZF so I think we're really talking about adding an HDF5 backend. In practice, this could use the new API of h5netcdf or h5py directly. Either way, my initial thought is that we want a to_hdf5() method on the dataset. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1536 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ac5fKqKijwFO916RY0ARUZq9O25QBm9Jks5sc0ungaJpZM4PFJ5x>.

[shoyer]

h5netcdf already produces (slightly) incompatible netCDF files for some edge cases (e.g., complex numbers). This should probably be fixed, either by disabling these features or requiring an explicit opt-in, but nobody has gotten around to writing a fix yet (see h5netcdf/h5netcdf#28).

In practice, many of our users seem to be pretty happy making use of these new features. LZF compression would just be another one.

I like @jhamman's idea of adding a dedicated to_hdf5() method that handles encoding with h5netcdf's new API. This would basically be a clone of to_netcdf(). In practice, I guess we would implement this with another engine for h5netcdf.

@petacube zstandard is great, but it's not in h5py yet! I think we'll need zarr for that (see #1528)

[dopplershift]

@shoyer I just want to clarify: is the netCDF C library able to read files using these features?

[shoyer]

@dopplershift No, I don't think so. NetCDF-C only supports zlib compression (and doesn't support h5py's handling of complex variables, either, which use an HDF5 enumerated type).

[dopplershift]

But these are still considered netCDF files, not HDF5 files? As in, they declare attributes that say "this is a netCDF file"?

[shoyer]

But these are still considered netCDF files, not HDF5 files? As in, they declare attributes that say "this is a netCDF file"?

Yes, I suppose so (and this should be fixed). h5netcdf currently writes the _NCProperties attribute to all files, though it uses a custom format that could be detected.

I hadn't really thought about this because the convention for marking HDF5 files as netCDF files is very recent and not actually enforced by any software (to my knowledge).

[dopplershift]

(cc @WardF)

Please, please, please don't write out "netCDF" files that don't conform to the spec. Either work with us to try to add the needed features to the spec (and the C-library) or call them something else. The spec exists for a reason. When such non-conformant files are distributed (and they will be), this creates a needless support load for the netcdf-c and netcdf-java developers (not to mention netCDF4-python).

Yes, I suppose so (and this should be fixed). h5netcdf currently writes the _NCProperties attribute to all files, though it uses a custom format that could be detected.

You're not actually saying the netCDF-c library should check for this custom format, are you?

[shoyer]

Please, please, please don't write out "netCDF" files that don't conform to the spec.

Of course not. I understand the issue here.

I'll issue a fix for h5netcdf to disable this unless explicitly opted into, but we'll also need a fix for xarray to support the users who are currently using it to save data with complex values -- probably by adding a to_hdf5() method.

Here is the NetCDF-C issue I opened on reading these sorts of HDF5 enums: Unidata/netcdf-c#267.

You're not actually saying the netCDF-c library should check for this custom format, are you?

No.

[shoyer]

I'm adding a loud warning about this (will eventually be an error) to h5netcdf.

[dopplershift]

Cool. Thx @shoyer .

[WardF]

For what it is worth, we have a branch at netCDF-C that allows for different compression plugins using the hdf5 plugin architecture. It will not be in the 4.5.0 release, but once 4.5.0 is finished we will be looking at it for either the subsequent release. There is still testing and documentation to be completed, but @DennisHeimbigner (who implemented it) can speak more about it.

[DennisHeimbigner]

The github branch filters.dmh for the netcdf-c library
now exposes the HDF5 dynamic filter capability.
This is documented here:
https://github.com/Unidata/netcdf-c/blob/filters.dmh/docs/filters.md
I welcome suggestions for improvements.

I also note that I am extending this branch to now handle szip compression.
It turns out there is now a patent-free implementation called libaec (HT Rich Signell)
so there is no reason not to make it available.