`np.dtypes.StringDType(na_object=None)` doesn’t get preserved

[flying-sheep]

Zarr version

3.1.5

Numcodecs version

0.16.5

Python Version

3.13

Operating System

Mac

Installation

uvx --with=zarr --with=numpy ipython

Description

Trying to store a numpy string array with na_object=np.nan fails, but storing one with

Steps to reproduce

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///

import zarr

arr_str = np.array(["x", None], dtype=np.dtypes.StringDType(na_object=None))
print(arr_str)  # -> array(['x', None], dtype=StringDType(na_object=None))

g = zarr.open_group(store={})
g["arr_str"] = arr_str
print(g["arr_str"][:])  # -> array(['x', ''], dtype=StringDType())

Additional output

No response

[d-v-b]

the zarr string data type does not express the na_object semantics, so I wouldn't expect na_object=NaN to work. We would need to add this to the spec, and even then I think composing string with the optional dtype is a better solution.

given that, what would you expect zarr python to do here?

[d-v-b]

IMO we should error if na_object is set:

>>> np.dtypes.StringDType(na_object=None).na_object
>>> np.dtypes.StringDType().na_object
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    np.dtypes.StringDType().na_object
AttributeError: 'numpy.dtypes.StringDType' object has no attribute 'na_object'. Did you mean: 'hasobject'?

dynamic attributes, great stuff

[flying-sheep]

given that, what would you expect zarr python to do here?

IMO we should error if na_object is set

Yeah, for example! Better than losing information and therefore corrupting data.

If there happens to be an unambiguous way to map between StringDType(na_object=…) and the new optional codec, transparently doing that would also be nice. We could define a special encoding for some common NA objects zarr-python recognizes (e.g. None, np.nan, pd.NA), but I get when that’s not something you want to do in zarr-python.

dynamic attributes, great stuff

I think that’s intentional in this case, as the fill value can be None. So instead of a sentinel object (which would be truthy or falsy and/or coerce to itself when compared with == like pandas.NA) they chose that, which needs to be be unambiguously checked with hasattr.