[draft] zarr object models

[d-v-b]

this ZEP defines a representation of a zarr hierarchy, called a Zarr Object Model (ZOM). The purpose of this ZEP is to standardize an abstract representation of zarr hierarchies to support declarative Zarr APIs, and to give type systems access to the structure of zarr hierarchies. A side effect of this ZEP is a standardization of consolidated metadata, which can be defined as a flattening transformation applied to a ZOM representation of a zarr hierarchy.

I didn't use the template structure for this ZEP because it felt limiting, but if that's a big problem I can bring more of that structure back in.

In terms of what needs to be done:

• The prose needs work. I want to carefully distinguish the "base ZOM", which is kind of a meta-model, from the ZOM for particular zarr versions (2 and 3), from ZOM representations of actual zarr hierarchies. A delicate touch is also required to distinguish zarr arrays (which contain data) from the ZOM representation of a zarr array, which contains now data.
• I need to complete the JSON schemas for zarr V3. help is appreciated!

cc @jhamman

[jbms]

For zarr v3, the user-defined attributes are stored within the main metadata file under the attributes member name. Renaming that to attrs in the representation proposed here may be confusing.

[d-v-b]

@jbms That's a great point; I'm happy to expand attrs to attributes. On the topic of naming, are there any objections to members?

[normanrz]

I am not sure I got the motivation for this. I understand the motivation of consolidated metadata. Maybe this ZEP should morph into that? Then, it should be an extension to the zarr.json for persistence imo. Also, I would focus the ZEP on v3.

[ivirshup]

Agree with @normanrz, a section for motivation and usage would be helpful.

[d-v-b]

I totally agree that the motivation needs to be expanded. At the moment, all we say is

Such a data structure or interface would facilitate operations like evaluating whether two Zarr hierarchies are identically structured, evaluating whether a given Zarr hierarchy has a specific structure, or creating a Zarr hierarchy with a desired structure.

@normanrz do you agree that these are valid motivations, and worthy of expanding on, or should we provide additional motivations?

Also, I would focus the ZEP on v3.

The basic ZOM applies to v3 and v2 equally, and I think it's important to emphasize this, because the ZOM representation would be useful for converting from v2 hierarchies to v3 (and from v3 to v4, if v4 ever exists). Would it help if I made this point clearly in the ZEP?

[keller-mark]

To help with the motivation, I think this point

give type systems access to the structure of zarr hierarchies

could be emphasized further, perhaps with an example of how it could eventually work.

It seems like this is a kind of "extended consolidated metadata" and could be framed more in that way. Beyond the "base consolidated metadata", my understanding is that this would also include the contents of the .zattrs/.zarray/.zgroup files (and the v3 equivalent) which would be used to implement the support for typing / validation / comparison.

the ZOM representation would be useful for converting from v2 hierarchies to v3 (and from v3 to v4, if v4 ever exists)

Perhaps it would be simpler to use the metadata file names directly in the flattened representation rather than abstracting / trying to unify across versions. Using $refs in JSON schemas could enable supporting both v2 and v3 despite possibly using different/split metadata file names.

It could be useful to define the name of an optional property that could be used to specify the URL of a JSON schema to use for validation of ZOM-structured stores such as OME-NGFF or AnnData. For example, Vega-Lite uses a $schema property for this.

Perhaps outside the scope of this proposal, but related, it might be useful for Zarr to make a distinction between ZOM-structured stores vs. non-ZOM-structured stores in the name of the root file/directory (as a convention, not a requirement). For example, as a human, if i look in my file explorer and see something.zarr, it would be nice at first glance to know whether it contains a particular ZOM-structured store or not (without context or looking at the contents). For example, a convention like something.anndata.zarr could be used for this. This would be analogous to using .h5ad rather than .h5 to store AnnData-structured HDF5. I have started doing this in personal work with Zarr stores, but it does not seem like it is a convention for Zarr stores in the wild.

[jhamman]

Thanks for getting this started @d-v-b! I think this will really aid in the maintenance and development of Zarr implementations, old and new.

[jhamman]

• dataclass zarr

(I have an unpublished version that I can share soon)

[jhamman]

see zarr-developers/zarr-python#1526

[normanrz]

zarrita also has attrs classes that define the metadata (minus the new members properties) https://github.com/scalableminds/zarrita/blob/async/zarrita/metadata.py#L259

[jhamman]

it may also be worth summarizing some of the intended benefits to existing/internal applications. For example, the utilization of a standard data object internally within zarr-python may help improve workflow for creating large hierarchies by allowing users to create the ZOM metadata before passing it to a zarr.creation method.

[jhamman]

let me know if you could use some help generating this.

[d-v-b]

some help here would be great, thank you!

[d-v-b]

just a clarification about the timeline: I'm working on a transatlantic move, so I can't promise a huge investment in this until mid-october. Thanks for your patience!

[d-v-b]

as per feedback, the field used for user metadata has been renamed from attrs to attributes. I also added a motivating example (comparing if two zarr hierarchies have the same structure), and I moved the zarr v2 stuff behind the zarr v3 examples (although those need fleshing out)

[d-v-b]

see also: https://github.com/google/tensorstore/blob/master/tensorstore/driver/zarr/schema.yml

[normanrz]

zarrita also has attrs classes that define the metadata (minus the new members properties) https://github.com/scalableminds/zarrita/blob/async/zarrita/metadata.py#L259

[normanrz]

I want to reiterate the idea of broadening this ZEP to include persisted consolidated metadata. Basically, why not allow to store the members property in a zarr.json?
We would need to define the semantics of consolidated metadata (e.g. do member nodes still needs json files, does the members hierarchy need to be exhaustive). I would be happy to contribute that if there is interest to move this ZEP in that direction.

[d-v-b]

there's definitely interest in that, my apologies for not making this more clear earlier. I'm not a user of consolidated metadata so I don't have a lot of experience with it, but I think for this ZEP to encompass consolidated metadata functionality as it exists today (i.e, a flat list of string keys pointing to JSON objects) we would need to define a tree flattening operation, and possible make members nullable (because in a flattened representation a ZOM group shouldn't hold a reference to its contents). Alternatively, if the flattened representation of the hierarchy used in consolidated metadata isn't essential to its function, we could simply just put a ZOM in JSON and leave it to clients to do the flattening. I don't have strong feelings either way! You should absolutely feel free contribute something here.

[rabernat]

I think we should keep consolidated metadata separate from this ZOM concept. Consolidated metadata is a serialization choice, while the object model describes the relationships between entities in a more abstract, serialization independent way.

We intend to propose a ZEP which uses a STAC style link-relation element to allow a client to traverse an entire hierarchy without being able to list a store. This is similar to consolidated metadata but more scalable because it does not require all the metadata to be in a single json file. For context, we have hierarchies with 100_000 nodes. Would be happy to collaborate and iterate with you on that.

[d-v-b]

Consolidated metadata is a serialization choice, while the object model describes the relationships between entities in a more abstract, serialization independent way.

In this case, maybe it would be good to get a statement like this in this ZEP to clarify the relationship between the abstract ZOM and consolidated metadata.

[rabernat]

I am now wondering if this members property needs to be tweaked to accommodate the case of very large hierarchies. As is, it seems like the entire hierarchy may have to be explicitly populated all at once.

In python terms, I'd like to allow members to be either a set of child objects or a generator that yields such objects lazily. Is this making it too complicated?

[normanrz]

(i.e, a flat list of string keys pointing to JSON objects)

It wouldn't have to have this structure. It could be a nested json structure like this:

{
  "zarr_format": 3,
  "node_type": "group",
  "attributes": {},
  "members": {
    "some_group": {
      "zarr_format": 3,
      "node_type": "group",
      "attributes": {},
      "members": {
        "some_array": {
          "zarr_format": 3,
          "node_type": "array",
          ...
        }
      }
    }
  }
}

I think we should keep consolidated metadata separate from this ZOM concept. Consolidated metadata is a serialization choice, while the object model describes the relationships between entities in a more abstract, serialization independent way.

I think there is strong overlap between the ZOM and consolidated metadata. This ZEP introduces a JSON schema that describes the existing metadata of groups and arrays with a new addition of the members property. I think it would be very confusing, if consolidated metadata would end up with different terminology than the ZOM.

We intend to propose a ZEP which uses a STAC style link-relation element to allow a client to traverse an entire hierarchy without being able to list a store. This is similar to consolidated metadata but more scalable because it does not require all the metadata to be in a single json file. For context, we have hierarchies with 100_000 nodes. Would be happy to collaborate and iterate with you on that.

That sounds great. As I said, we can discuss the semantics and features of the consolidated metadata. That could include linking. I don't think we should limit ourselves by what the implementation in zarr-python currently has.

[d-v-b]

I am now wondering if this members property needs to be tweaked to accommodate the case of very large hierarchies. As is, it seems like the entire hierarchy may have to be explicitly populated all at once.

This concern is real! See also janelia-cellmap/pydantic-zarr#2 . The proposal there was to make members nullable, where None would encode "The members have not been parsed", and to give a tree parser the option to limit the depth of traversal, which would result in "truncated" GroupSpec instances being valid. But maybe the python generator approach obviates the need to express this with nullability? I'm open to suggestions here

[rabernat]

It would be nice to be able to distinguish between "there are definitely no members" vs. "there may be members, but they have to be discovered explicitly"

[d-v-b]

in pydantic-zarr members is now nullable, and it's been extremely useful. That being said, this can be viewed as a well-defined transformation of the base type, so it's not clear if the ZEP actually needs to address it.

[MSanKeys963]

Hi @d-v-b. I've fixed the RTD build issue in #51.

The current PR can be viewed here: https://zeps--46.org.readthedocs.build/en/46/draft/ZEP0006.html

[d-v-b]

I threw in a JSON schema for ZOM[v3], a ZOM[v3] example hierarchy serialized to JSON, and some python / typescript static typing examples.

I am wondering if it would make more sense to push these hierarchy definitions into the v2 and v3 specs as addenda? This ZEP could exist for posterity, but this would be an easier way to formally associate a particular ZOM with a specific Zarr version. Since the change would be purely additive, it seems safe to do retrospectively (in the case of Zarr v2).

Thoughts?

[bogovicj]

What should the value of attributes be when there are no user-specified attributes? Say, for a zarr v2 group with no .zattrs.

Since the attributes field is required, null and {} both seem reasonable to me. The thing about {} is that it won't be possible to distinguish between non-existent attributes, and attributes containing an empty object. So perhaps null is preferable.

If we're willing not to require the attributes field, then a node without the field could work too.

I havn't formed a preference.

[d-v-b]

@bogovicj so far i've been thinking about exclusively using {} for "empty attributes", but that's just because I have never been in a situation where "no attributes at all" vs "attributes are there, but empty" was an important distinction. I think this experience stems from attributes historically being "easy" to access, I guess because they are small in size and always have the same name.

By contrast, I think there's an argument for making the .members property of a group nullable to distinguish "there are no members" from "have not checked for members". This accommodates two scenarios: first, some storage backends (http) don't allow discovering subgroups / arrays, so members: null would signify that it wasn't possible to check for members. Second, some hierarchy parsers might want to limit how deep into a zarr hierarchy they go, so members: null would signify that no check for members was done for a given group. See janelia-cellmap/pydantic-zarr#2 for a discussion of this latter point.

That being said, if the attributes: null vs attributes: {} distinction can do some work for someone, then I'd be fine considering making attributes nullable