NCZarr - Netcdf Support for Zarr

[DennisHeimbigner]

I am moving the conversation about NCZarr to its own issue. See Issue https://github.com/zarr-developers/zarr/issues/317 for initial part of this discussion.

[DennisHeimbigner]

Naming issue:
I have about convinced myself that rather than creating KVP level objects
like .zdimensions, I should just use the existing Zarr attribute mechanism.
In order to do this, it is necessary to setup some naming conventions for such
attributes. Basically, we need to identify that an attribute is special (and probably
hidden) and for which extension(s) it applies.
For NCZarr, let me propose this:

1. All such attributes start with two underscores
2. Next is a 2-4 character tag specific to the extension: "NCZ" for NCZarr.
3. another underscore
4. the rest of the attribute name.

So, we might have "__NCZ_dimensions" instead of .zdimensions.

[jakirkham]

Thanks for opening this @DennisHeimbigner.

Encountered issue ( zarr-developers/zarr-python#280 ) again recently. So figured that might interest you given some of this discussion about how to manage specs. Though each issue has its own place I think.

If we do go down the attribute road, agree that having some non-conflicting name convention is important. The other option might be to widen the spec of things like .zarray to allow specs subclassing Zarr's spec to add additional relevant content here as others have mentioned. A third option similar to what you have done would be to add something like .zsubspec, which users can fill as needed. We might need certain keys in there like subspec name, subspec version, etc., but otherwise leave it to users to fill these out as needed.

[alimanfoo]

Thanks @DennisHeimbigner.

Just to add that, on the question of whether to pack everything into attributes (.zattrs) or whether to store metadata separately under other store-level keys (.zdims, .ztypdefs, etc.), I think both are reasonable and personally I have no objection to either.

I lean slightly towards using attributes (.zattrs) because it plays nicely with some existing API features. E.g., the NCZ metadata can be accessed directly via the attributes API. And, e.g., the NCZ metadata would get included if using consolidated metadata, which is an experimental approach to optimising cloud access, available in the next release of Zarr Python. But neither of these are blockers to the alternative approach, because it is straightforward to read and decode JSON objects directly from a store, and it would also be straightforward to modify the consolidated metadata code to include other objects.

[DennisHeimbigner]

We have learned from the existing netcdf-4 that datasets exist with
very large (~14mb) metadata.
I was looking at the Amazon S3 query capabilities and they are extremely limited.
So the idea of consolidated metadata seems like a very good idea.
This reference:
https://zarr.readthedocs.io/en/latest/tutorial.html#consolidating-metadata
does not provide any details of the form of the proposed consolidated metadata.
Note that there may not be any point in storing all of the metadata, especially
if lazy reading of metadata is being used (as it is in the netcdf-4 over hdf5
implementation).
Rather I think that what is needed is just a skeleton so that query is never needed:
we would consolidate the names and kinds (group, variable, dimension, etc)
and leave out e.g. attributes and variable types and shapes.

[DennisHeimbigner]

here is a proposed consolicated metadata structure for NCZarr.
It would be overkill for standard Zarr, which is simpler.
Sorry if it is a bit opaque since it is a partial Antlr grammar.
nczmetadata.txt

[alimanfoo]

We have learned from the existing netcdf-4 that datasets exist with very large (~14gb) metadata.

Wow, that's big. I think anything near that size will be very sub-optimal in zarr, because of metadata being stored as uncompressed JSON documents. I wonder if in cases like that, it might be necessary to examine what is being stored as metadata, and if any largish arrays are included then consider storing them as arrays rather than as attributes.

I was looking at the Amazon S3 query capabilities and they are extremely limited. So the idea of consolidated metadata seems like a very good idea. This reference: https://zarr.readthedocs.io/en/latest/tutorial.html#consolidating-metadata does not provide any details of the form of the proposed consolidated metadata.

Apologies the format is not documented as yet. There's an example here: zarr-developers/zarr-python#268 (comment)

[DennisHeimbigner]

That was a typo. The correct size is 14 mb.

[alimanfoo]

That was a typo. The correct size is 14 mb.

Ah, OK! Although 14MB is still pretty big, it's probably not unmanageable.

[DennisHeimbigner]

Depends on what manageable means, I suppose. We have situations where
projects are trying to load a small part of the metadata from thousands of files
each of which has the amount of metadata. Needless to say, this is currently
very slow. We are trying various kinds of optimizations around lazy loading
of metadata but the limiting factor will be HDF5. A similar situation
is eventually going to occur here, so thinking about various optimizations
is important.

[alimanfoo]

Depends on what manageable means, I suppose. We have situations where projects are trying to load a small part of the metadata from thousands of files each of which has the amount of metadata. Needless to say, this is currently very slow. We are trying various kinds of optimizations around lazy loading of metadata but the limiting factor will be HDF5. A similar situation is eventually going to occur here, so thinking about various optimizations is important.

That's helpful to know. FWIW the consolidated metadata feature currently in zarr python was developed for the xarray use case, where the need (as I understand it) is to load *all* metadata up front. So that feature combines the content from all .zarray, .zgroup and .zattrs objects from the entire group and dataset hierarchy into a single object, which can then be read from object storage in a single HTTP request. If you have use cases where you have a large amount of metadata but only need to read parts of it at a time, that obviously might not be optimal. However, 14MB is not an unreasonable amount to load from object storage, would probably be fine to do interactively (IIRC bandwidth to object storage from compute nodes within the same cloud is usually ~100MB/s). I'm sure there would be other approaches that could be taken too that could support partial/lazy loading of metadata. Happy to discuss at any point.

[jakirkham]

Are you able to provide data on where most of the time is being spent, @DennisHeimbigner?

[DennisHeimbigner]

Issue: Attribute Typing
I forgot to address one important difference between the netcdf-4 model and Zarr:
attribute typing. In netcdf-4, attributes have a defined type. In Zarr, attributes are
technically untyped, although in some case it is possible to infer a type from the value
of the attribute.

This is most important with respect to the _FillValue attribute for a variable.
There is an implied constraint (in netcdf-4 anyway) that the type of the attribute
must be the same as the type of the corresponding variable. There is no way to
guarantee this for Zarr except by doing inferencing.,

Additionally, if the variable is of a structured type, there is currently no standardized
way to define the fill value for such a type nor is there a way to use structured types
with other, non-fillvalue, attributes.

Sadly, this means that NCZarr must add yet another attribute that specifies
the types of other attributes associated with a group or variable.

[alimanfoo]

Hi @DennisHeimbigner,

Regarding the fill value specifically, the standard metadata for a zarr array includes a fill_value key. There are also rules about how to encode fill values to deal with values that do not have a natural representation in JSON. This includes fill values for arrays with a structured dtype. If possible, I would suggest to use this feature of standard array metadata, rather than adding a separate _FillValue attribute. If not, please do let us know what's missing, that would be an important piece of information to carry forward when considering spec changes.

Regarding attributes in general, we haven't tried to standardise any method to encode values that do not have a natural JSON representation. Currently it is left to the application developer to decide their own method for encoding and decoding values as JSON, e.g., I believe xarray has some logic for encoding values in zarr attributes. There has also been some discussion of this at #354 and #156.

Ultimately it would be good to standardise some conventions (or at least define some best practices) for representing various common value types in JSON, such as typed arrays. I'm more than happy for the community to lead on that.

[DennisHeimbigner]

This reference -- https://zarr.readthedocs.io/en/stable/spec/v2.html#fill-value-encoding --
does not appear to address fill values for structured types. Did you get the reference wrong?

[alimanfoo]

If an array has a fixed length byte string data type (e.g., "|S12"), or a structured data type, and if the fill value is not null, then the fill value MUST be encoded as an ASCII string using the standard Base64 alphabet.

I.e., use base 64 encoding.