Zstd codec checksum configuration not recognized by TensorStore for Zarr v2

[mkitti]

TensorStore does not recognize the checksum property in Zarr v2. It only recognizes it for Zarr v3:

• Zarr v2: https://google.github.io/tensorstore/driver/zarr/index.html#json-driver/zarr2/Compressor/zstd
• Zarr v3: https://google.github.io/tensorstore/driver/zarr3/index.html#json-driver/zarr3/Codec/zstd

However, as of the implementation in #180, the checksum property is always written:

Zarr.jl/src/Compressors/zstd.jl

Line 43 in 8a5ea8c

JSON.lower(z::ZstdCompressor) = Dict("id"=>"zstd", "level" => z.config.compressionLevel, "checksum" => z.config.checksum)

cc: @nhz2 , @bjarthur

[nhz2]

This matches what zarr-python outputs.

julia> using PythonCall
julia> using CondaPkg
julia> CondaPkg.add("zarr"; version="2.*")
julia> zarr = pyimport("zarr")
julia> numcodecs = pyimport("numcodecs")
julia> zarr.__version__
Python: '2.18.7'

julia> numcodecs.__version__
Python: '0.15.1'

julia> g = zarr.group("testzarr2");

julia> z1 = g.create_dataset("a", shape=(2,6,10),chunks=(1,2,3), dtype="i4", compressor=numcodecs.Zstd());

julia> read("testzarr2/a/.zarray", String) |> print
{
    "chunks": [
        1,
        2,
        3
    ],
    "compressor": {
        "checksum": false,
        "id": "zstd",
        "level": 0
    },
    "dtype": "<i4",
    "fill_value": 0,
    "filters": null,
    "order": "C",
    "shape": [
        2,
        6,
        10
    ],
    "zarr_format": 2
}

julia> numcodecs.Zstd().get_config()
Python: {'id': 'zstd', 'level': 0, 'checksum': False}

xref: zarr-developers/zarr-python#2647

This looks like a bug in TensorStore, it should be reading the compressor metadata more flexibly, like we have here:

Zarr.jl/src/Compressors/zstd.jl

Lines 22 to 27 in 8a5ea8c

	function getCompressor(::Type{ZstdCompressor}, d::Dict)
	ZstdCompressor(;
	level=get(Returns(0), d, "level"),
	checksum=Bool(get(Returns(false), d, "checksum")),
	)
	end

[mkitti]

How about the resolution in zarr-developers/zarr-python#2655 ?

[nhz2]

I like that. In general, we could not include compressor config options that are optional and at the default value to save some space in the JSON file.

But in this case, shouldn't this change be made in numcodecs? Since there isn't a spec for zstd in zarr v2 aside from the numcodecs source code, it seems strange not to match what numcodecs.Zstd().get_config() returns.

[mkitti]

Here's a test I just ran, writing a Zarr v2 array with zarr-python 3.0.8:

import zarr
import numcodecs

print(f"zarr version: {zarr.__version__}")
print(f"numcodecs version: {numcodecs.__version__}")

arr = zarr.create_array(
    "zstd_array",
    shape=(512,512),
    dtype="uint8",
    chunks=(64,64),
    fill_value=0,
    compressors=numcodecs.zstd.Zstd(level=3, checksum=False),
    zarr_format=2,
    overwrite=True
)

with open("zstd_array/.zarray", "r") as f:
    print("With checkum = False")
    print(f.read())
    print()

arr = zarr.create_array(
    "zstd_array",
    shape=(512,512),
    dtype="uint8",
    chunks=(64,64),
    fill_value=0,
    compressors=numcodecs.zstd.Zstd(level=3, checksum=True),
    zarr_format=2,
    overwrite=True
)

with open("zstd_array/.zarray", "r") as f:
    print("With checkum = True")
    print(f.read())

Output:

zarr version: 3.0.8
numcodecs version: 0.16.1
With checkum = False
{
  "shape": [
    512,
    512
  ],
  "chunks": [
    64,
    64
  ],
  "fill_value": 0,
  "order": "C",
  "filters": null,
  "dimension_separator": ".",
  "compressor": {
    "id": "zstd",
    "level": 3
  },
  "zarr_format": 2,
  "dtype": "|u1"
}

With checkum = True
{
  "shape": [
    512,
    512
  ],
  "chunks": [
    64,
    64
  ],
  "fill_value": 0,
  "order": "C",
  "filters": null,
  "dimension_separator": ".",
  "compressor": {
    "id": "zstd",
    "level": 3,
    "checksum": true
  },
  "zarr_format": 2,
  "dtype": "|u1"
}

[mkitti]

I'll also note that zarr-python version 3 defaults to using Zstandard when compressors is not specified, even when saving with zarr_format=2.

import zarr
import numcodecs

print(f"zarr version: {zarr.__version__}")
print(f"numcodecs version: {numcodecs.__version__}")

arr = zarr.create_array(
    "zstd_array",
    shape=(512,512),
    dtype="uint8",
    chunks=(64,64),
    fill_value=0,
    zarr_format=2,
    overwrite=True
)

with open("zstd_array/.zarray", "r") as f:
    print("No compressors")
    print(f.read())
    print()

Output:

zarr version: 3.0.8
numcodecs version: 0.16.1
No compressors
{
  "shape": [
    512,
    512
  ],
  "chunks": [
    64,
    64
  ],
  "fill_value": 0,
  "order": "C",
  "filters": null,
  "dimension_separator": ".",
  "compressor": {
    "id": "zstd",
    "level": 0
  },
  "zarr_format": 2,
  "dtype": "|u1"
}

[mkitti]

I like that. In general, we could not include compressor config options that are optional and at the default value to save some space in the JSON file.

But in this case, shouldn't this change be made in numcodecs? Since there isn't a spec for zstd in zarr v2 aside from the numcodecs source code, it seems strange not to match what numcodecs.Zstd().get_config() returns.

Apparently, the numcodecs source code is only authoritative at a certain time...