Cannot open zarr store stored in Azure blob file system

[lsim-aegeri]

What happened?

I'm trying to save data to a zarr store hosted on my Azure blob filesystem. I'm able to write the data just fine, but then am unable to open the dataset afterwards--when I try to open it, I get an empty dataset with no variables. I believe this is related to the storage backend because everything works when I write the data to my local filesystem instead. I also don't think this is a zarr version issue because the same thing happens with zarr v2 and v3, and when I use consolidated metadata with zarr v2.

What did you expect to happen?

I should be able to read data from the zarr stores I write to my Azure blob filesystem.

Minimal Complete Verifiable Example

import xarray as xr
import numpy as np
import pandas as pd
import adlfs

ds = xr.Dataset(
    {"foo": (("x", "y"), np.random.rand(4, 5))},
    coords={
        "x": [10, 20, 30, 40],
        "y": pd.date_range("2000-01-01", periods=5),
        "z": ("x", list("abcd")),
    },
)

store = 'abfs://weatherblob/xr-test/test.zarr-v3'
fs = adlfs.AzureBlobFileSystem(
    account_name="abcdefg",
    sas_token="ABCDEFG",
)
ds.to_zarr(
    store, 
    storage_options=fs.storage_options, 
    mode='w', 
    consolidated=False, 
    zarr_format=3, 
)

ds_zarr = xr.open_zarr(
    store, 
    storage_options=fs.storage_options, 
    consolidated=False, 
    zarr_format=3, 
)

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

ds.info returns: 
Size: 248B
Dimensions:  (x: 4, y: 5)
Coordinates:
  * x        (x) int64 32B 10 20 30 40
  * y        (y) datetime64[ns] 40B 2000-01-01 2000-01-02 ... 2000-01-05
    z        (x) <U1 16B 'a' 'b' 'c' 'd'
Data variables:
    foo      (x, y) float64 160B 0.768 0.1867 0.1145 ... 0.3455 0.7483 0.8156>

ds_zarr.info returns: 
Size: 0B
Dimensions:  ()
Data variables:
    *empty*>

Also, the blob storage bucket has data at the zarr's path. 
* All the .json and chunk files that I'd expect are there. 
* Active blobs: 9 blobs, 3.04 KiB (3,110 bytes).

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.11.9 (main, Apr 6 2024, 17:59:24) [GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.15.0-1071-azure
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: None
libnetcdf: None

xarray: 2025.3.1
pandas: 2.2.3
numpy: 2.1.3
scipy: 1.15.2
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
zarr: 3.0.6
cftime: None
nc_time_axis: None
iris: None
bottleneck: 1.4.2
dask: 2025.3.0
distributed: 2025.3.0
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2025.3.2
cupy: None
pint: None
sparse: 0.16.0
flox: 0.10.2
numpy_groupies: 0.11.2
setuptools: None
pip: None
conda: None
pytest: None
mypy: None
IPython: 9.1.0
sphinx: None
adlfs: 2024.12.0

[welcome]

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

[max-sixty]

potentially have look at the files in the bucket — how do they differ from the files that are written to a local filesystem? if we write to one and then copy to the other, does that work?

fwiw it seems unlikely this in an xarray issue (though not impossible)

[lsim-aegeri]

Yeah, I can't say for sure where the issue is coming from but I am getting the results I'd expect after downgrading to an older version of xarray.

INSTALLED VERSIONS

commit: None
python: 3.11.9 (main, Apr 6 2024, 17:59:24) [GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.15.0-1071-azure
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: None
libnetcdf: None

xarray: 2024.7.0
pandas: 2.2.3
numpy: 2.1.3
scipy: 1.15.2
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
zarr: 2.18.3
cftime: None
nc_time_axis: None
iris: None
bottleneck: 1.4.2
dask: 2025.3.0
distributed: 2025.3.0
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2025.3.2
cupy: None
pint: None
sparse: 0.16.0
flox: 0.10.2
numpy_groupies: 0.11.2
setuptools: None
pip: None
conda: None
pytest: None
mypy: None
IPython: 9.1.0
sphinx: None

And write to a zarr v2 store:

ds.to_zarr(
    store, 
    storage_options=fs.storage_options, 
    mode='w', 
    consolidated=False, 
    zarr_version=2, 
)
ds_zarr = xr.open_zarr(
    store, 
    storage_options=fs.storage_options, 
    consolidated=False, 
    zarr_version=2, 
)
ds_zarr.info

I get what I'd expect.

Size: 248B
Dimensions:  (x: 4, y: 5)
Coordinates:
  * x        (x) int64 32B 10 20 30 40
  * y        (y) datetime64[ns] 40B 2000-01-01 2000-01-02 ... 2000-01-05
    z        (x) <U1 16B dask.array<chunksize=(4,), meta=np.ndarray>
Data variables:
    foo      (x, y) float64 160B dask.array<chunksize=(4, 5), meta=np.ndarray>>

I'd really prefer to use the latest versions though, because I'm very excited for zarr v3 and that requires more recent versions of xarray.

[max-sixty]

potentially have look at the files in the bucket — how do they differ from the files that are written to a local filesystem? if we write to one and then copy to the other, does that work?

could you look at this?

[lsim-aegeri]

Yes, working on it!

[lsim-aegeri]

Very weird.

• When I write to blob they open incorrectly from blob.
• When I write locally and then copy the files to blob they open correctly from blob.
• When I write to blob, copy locally, then copy back to blob they open incorrectly from blob.
• When I write to blob, copy locally, rename, then copy back to blob with the new name, they open correctly from blob.

I compared all the json files when writing locally and to blob and could not see any differences. I also checked file sizes and they were the same.

Do you know what to make of this?

[lsim-aegeri]

But I think you're right, this probably isn't purely an xarray issue. I can also read the files if I write to blob, then clone within blob, and open from blob. It feels like something related to permissions? I'll investigate further.

I do wonder if a helpful change could be made to xarray to prevent this, given reading and writing works fine using xarray v2024.7.0. Will follow up when I've identified the problem.

[oj-tooth]

Hi,

Just wanted to follow up on this issue, I've also encountered this bug after writing a Zarr v3 store to the JASMIN s3-compatible object store in the UK. So far, I've tracked the error down to ZarrStore.open_group() in zarr.py (see here), which is called if you pass a string rather than a store to open_zarr(). From what I can tell, store = ZarrStore.open_group() is failing to behave as expected because zarr.open_group() returns a group with no members or arrays.

I can open the unconsolidated Zarr Store as expected by manually passing the store as follows:

store = zarr.storage.FsspecStore(fs=obj_store, path=dest)

xr.open_zarr(store, zarr_format=3, consolidated=False)

where obj_store is an async s3fs.S3FileSystem instance. Planning to look into this more tomorrow, so will post any updates then!

[oj-tooth]

In case of interest, the following storage_options seem to have solved this issue for me by ensuring successful store creation in the open_zarr() call:

store_path = "s3://bucket/prefix"

ds = xr.open_zarr(fpath, zarr_format=3, consolidated=False, storage_options={"anon":True, "asynchronous":True, "client_kwargs":{"endpoint_url":"https://my_end_point_url"}})

Find it interesting though that reading the same store created with consolidated metadata does not require any storage_options as behind the scenes zarr.open_consolidated() behaves as expected when providing the complete (including endpoint) url.

[TomNicholas]

Thank you all for diving into this. It's highly likely that this is a bug in the zarr-python library rather than in xarray. Has anyone tried to replicate the bug simply by opening the zarr store using zarr-python directly and listing the contents of the group using the zarr-python API? That would be my next move.

[oj-tooth]

Hi Tom,
Yes, I looked into this further yesterday and traced the issue to the use_consolidated=False case in the zarr.open_group() zarr-python function. I created a zarr v3 store with and without consolidated metadata and the consolidated metadata store arrays can be accessed with use_consolidated=True below (or using open_consolidated() as an alias), whereas use_consolidated=False with the unconsolidated store returns an empty group -> empty dataset when consumed by xarray.

zarr.open_group(url, mode='r', zarr_format=3, use_consolidated=False)

I'll drop an issue in zarr-python as currently the way around this in xarray is to manually specify storeage_options even though, in our case, the data should be read-only accessible for users at the url.