Certain extension points of zarr are not implemented

[RFLeijenaar]

Zarr is built with extensibility in mind with several extension points defined in the spec. However, zarr-python does not yet implement all these extension points. Notably, chunk grid and chunk key encoding are not extendable yet, and I believe storage transformers aren't even implemented altogether.

For some background: I was interested in extending the chunk key encoder to address issues with filesystems that struggle with a large number of files in a single directory. The default V3 implementation uses / as the dimension separator, but this is not helpful when you have a single large dim. Sharding with large shards provides a solution, but the current implementation requires full shard rewrites upon writing, even when appending.

I couldn't find any concrete plans to add the missing extension points. Are there any such plans?

I could help to try and implement this for chunk key encodings, but also realize that the current setup with the registry and global mutable config is not ideal. #3360

I am curious what your thoughts are.

[d-v-b]

Thanks for writing this issue, It's definitely in scope for the project! The main reason we haven't implemented extensibility for chunk key encodings or chunks grids yet is basically just a lack of time.

Architecturally, I think chunk key encodings should be extensible on the writing side by implementing a new ChunkKeyEncoding, but this will fail on the reading side because we don't have a registry of chunk key encodings. We absolutely should, and it can be really simple -- just a key : value pairing where the keys are names, and the values are chunk key encoding classes. Happy to provide guidance here.

Chunk grids are more complicated. I suspect our indexing machinery currently assumes regular chunking to some extent. The good news is that this absolutely will be changed when we support irregular chunking (and that's a "when", not an "if").

[RFLeijenaar]

Thank you for the quick response, and for thinking along!

I can indeed confirm that ChunkKeyEncoding is extensible on the write side. When you read, it will parse the named config in ChunkKeyEncoding.from_dict(), which will obviously fail.

I have had a look at the repo and think the registry can be implemented in registry.py similar to zarr.buffer and zarr.ndbuffer.

The parsing in _parse_chunk_key_encoding() and ArrayV3Metadata.__init__() needs to be updated to read and construct classes from the registry.

Is this the correct direction? Since you mention a lack of time, I am definitely willing to give it a shot if you are in support. But must admit that I am not that experienced with contributing to open-source projects.

[d-v-b]

Is this the correct direction? Since you mention a lack of time, I am definitely willing to give it a shot if you are in support. But must admit that I am not that experienced with contributing to open-source projects.

Yes that's the right direction. Add a new registry for chunk key encodings in registry.py, ensure that the built-in chunk key encodings get added to the registry by adding required code to chunk_key_encodings.py, then add the required logic to the _parse_chunk_key_encoding() function that looks inside the registry. And feel free to ask questions in our developer chat#Zarr if you're stuck on something.