Backwards compatibility for reading Array.filters and Array.codecs

[TomAugspurger]

Zarr version

v3

Numcodecs version

n/a

Python Version

n/a

Operating System

n/a

Installation

n/a

Description

As part of getting xarray ready for zarr v3, I'm looking at how to handle the codec and filter API.

The primary / first place this is accessed is https://github.com/pydata/xarray/blob/1c6300c415efebac15f5ee668a3ef6419dbeab63/xarray/backends/zarr.py#L555-L556, which just reads the values of .filters and .compressor to place them in the DataArray.encoding. A few questions:

1. I'd like to add a .codecs property to the CodecPipeline ABC. This is fine for the BatchedCodecPipeline which AFAICT is the only actual codec pipeline. Does anyone foresee an issue with that? I'm not sure why that class is abstract and loadable through the config.
2. Is it fair to say that filters is the same as array_array_codecs?
3. Is it fair to say that compressor is the same as array_bytes_codecs?

There's also https://github.com/pydata/xarray/blob/1c6300c415efebac15f5ee668a3ef6419dbeab63/xarray/backends/zarr.py#L79, which accesses Codec.codec_id. I'm not sure yet about how to handle that, but right now the best is maybe .to_dict()["name"] (or we could have .to_dict() access a code_id)?

Steps to reproduce

n/a

Additional output

No response

[d-v-b]

Is it fair to say that filters is the same as array_array_codecs?
Is it fair to say that compressor is the same as array_bytes_codecs?

Unfortunately, no. This was discussed in an earlier discussion here: #1944.

There's also https://github.com/pydata/xarray/blob/1c6300c415efebac15f5ee668a3ef6419dbeab63/xarray/backends/zarr.py#L79, which accesses Codec.codec_id. I'm not sure yet about how to handle that, but right now the best is maybe .to_dict()["name"] (or we could have .to_dict() access a code_id)?

Probably the place to look for this functionality would be the classes that adapt the v2 compressor / filters to the v3 codecs api:

zarr-python/src/zarr/codecs/_v2.py

Line 69 in fbd1658

class V2Filters(ArrayArrayCodec):

In v3 filters / codecs are stored as dicts, but in #2179 I switch to storing instances of numcodecs.abc.Codec, which i think would permit re-using the old object inspection code?

[d-v-b]

(xref to the issue tracking the top-level codecs / filters / compressor api): #1943

[TomAugspurger]

Thanks for those links. I'll try to digest them and will propose a plan that'll either be compatibility code, or (more likely?) a PR to the migration guide.

[jhamman]

I think we can close this now. @TomAugspurger - feel free to reopen if you feel that #2652 and #2463 didn't quite hit the mark.

[TomAugspurger]

Looks good!