cftime autochunking does not work with kerchunk reference datasets (& presumably other virtualised data)

[charles-turner-1]

What happened?

The cftime autochunking we implemented in #10527 doesn't appear to work with kerchunk reference datasets: see: https://github.com/intake/intake-esm/actions/runs/19956553123/job/57226785198?pr=737:

        if dtype.hasobject:
>           raise NotImplementedError(
                "Can not use auto rechunking with object dtype. "
                "We are unable to estimate the size in bytes of object data"
            )
E           NotImplementedError: Can not use auto rechunking with object dtype. We are unable to estimate the size in bytes of object data

I've only just started prodding into this, but it appears to stem from this:

>>> data
CopyOnWriteArray(array=_ElementwiseFunctionArray(LazilyIndexedArray(array=<xarray.backends.zarr.ZarrArrayWrapper object at 0x1663f2a80>, key=BasicIndexer((slice(None, None, None),))), func=functools.partial(<function _apply_mask at 0x154e240e0>, encoded_fill_values={np.bytes_(b' ')}, decoded_fill_value=nan, dtype=dtype('O')), dtype=dtype('O')))
>>> _contains_cftime_datetimes(data)
False

which then bypasses the dtype faking logic we implemented:

limit: int | None
if _contains_cftime_datetimes(data):
    limit, dtype = fake_target_chunksize(data, chunkmanager.get_auto_chunk_size())
else:
    limit = None
    dtype = data.dtype

I haven't tested with other virtualisation formats/methods, but I assume it will similarly fail for eg. a virtual icechunk store.

Gonna dig into this myself - just flagging it so others are aware

---

Will fill this out as I progress investigating the issue.

What did you expect to happen?

No response

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
#   "s3fs=0.4.2",
# ]
# ///
#
# This script automatically imports the development branch of xarray to check for issues.
# Please delete this header if you have _not_ tested this script with `uv run`!

import xarray as xr
xr.show_versions()
import s3fs

xarray_open_kwargs = {'engine': 'zarr', 'chunks': 'auto', 
                      'backend_kwargs': {
                          'storage_options': {
                              'remote_protocol': 's3', 
                              'remote_options': {
                                'anon': True, 'asynchronous': True
                                },
                              'fo': 'https://storage.googleapis.com/xr-10989/noaa-nwm-test-reference.json'},
                              'consolidated': False},
                              'use_cftime': True,
                               'consolidated': False}

xr.open_dataset("reference://", **xarray_open_kwargs)

Steps to reproduce

No response

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Anything else we need to know?

No response

Environment

Details

[dcherian]

_contains_cftime_datetimes(data) : we should load the first value and check.

I'm surprised this isn't picked by existing tests that read data from disk. these wrappers are only applied when reading from disk...

[charles-turner-1]

  def _contains_cftime_datetimes(array: Any) -> bool:
2075         """Check if an array inside a Variable contains cftime.datetime objects"""
2076         # return True
2077  ->     breakpoint()                                                                  
2078         if cftime is None:
2079             return False
2080     
2081         if array.dtype == np.dtype("O") and array.size > 0:
2082             first_idx = (0,) * array.ndim
2083             if isinstance(array, ExplicitlyIndexed):
2084                 first_idx = BasicIndexer(first_idx)
2085             sample = array[first_idx]
2086             return isinstance(np.asarray(sample).item(), cftime.datetime)
2087     
2088         return False
(Pdb++) array.dtype
dtype('<i4')

I started digging into this yesterday & it looks like _contains_cftime_datetimes is reading (I assume) byte range specifiers used internally by kerchunk, and mistakenly assuming those are the data themselves..

I assume we'll just need to shim a dtype through somehow? I was digging through the codebase for inspiration, I think something like the ZarrArrayWrapper?

class ZarrArrayWrapper(BackendArray):
    __slots__ = ("_array", "dtype", "shape")

    def __init__(self, zarr_array):
        # some callers attempt to evaluate an array if an `array` property exists on the object.
        # we prefix with _ to avoid this inference.

        # TODO type hint this?
        self._array = zarr_array
        self.shape = self._array.shape

        # preserve vlen string object dtype (GH 7328)
        if (
            not _zarr_v3()
            and self._array.filters is not None
            and any(filt.codec_id == "vlen-utf8" for filt in self._array.filters)
        ):
            dtype = coding.strings.create_vlen_dtype(str)
        else:
            dtype = self._array.dtype

        self.dtype = dtype

[dcherian]

Can you make a minimal example please?

[charles-turner-1]

import xarray as xr
import s3fs

xarray_open_kwargs = {'engine': 'zarr', 'chunks': 'auto', 
                      'backend_kwargs': {
                          'storage_options': {
                              'remote_protocol': 's3', 
                              'remote_options': {
                                'anon': True, 'asynchronous': True
                                },
                              'fo': 'https://storage.googleapis.com/xr-10989/noaa-nwm-test-reference.json'},
                              'consolidated': False},
                              'use_cftime': True,
                              'consolidated': False}

xr.open_dataset("reference://", **xarray_open_kwargs)

Versions: s3fs=0.4.2, 2025.12.1.dev1+g58...

Happy to (try to?) tackle this myself, might need a nudge in the right direction though!

[charles-turner-1]

Wait... looks like a CRS got dragged along for the ride in the sample dataset I was playing with and was causing the issues. Adding it to drop variables resolves the issue.

I'll close this, might reopen if more careful investigation shows anything up.