`memory://` URL scheme no longer routes to fsspec, breaking interop with `MemoryFileSystem`

[maxrjones]

Zarr version

v3.2.0

Numcodecs version

v0.16.5

Python Version

3.13

Operating System

macOS (also reproduced on Linux in xarray's CI)

Installation

via pixi (conda-forge zarr=3.2.0)

Description

In zarr-python 3.2.0, zarr.storage._common.make_store special-cases the memory:// URL scheme to construct a ManagedMemoryStore instead of falling through to FsspecStore.from_url(...). This silently changes the meaning of memory://... URLs.

Previously, a memory://... string was treated as an fsspec URI and resolved against fsspec's process-wide MemoryFileSystem. Now it resolves to zarr's own in-process ManagedMemoryStore, which is a separate, isolated key/value store with no connection to fsspec.

As a result, the pattern of writing to fsspec's MemoryFileSystem via a mapper and then opening the resulting store via its memory://... URL is broken.

This is what make_store does in 3.2.0:

elif isinstance(store_like, str) and parsed is not None:
    if parsed.scheme == "memory":
        # Create or get a ManagedMemoryStore
        return ManagedMemoryStore(name=parsed.name, path=parsed.path, read_only=_read_only)

The previous behaviour was that, when fsspec is installed, a memory://... URL passed as a string continues to be routed through fsspec (so writers and readers using the same URL share state). If ManagedMemoryStore is the new default, then either:

• it should share state with fsspec's MemoryFileSystem, or
• the breaking change should be documented in release notes with a deprecation/warning path, and ideally an opt-in scheme (e.g. managed-memory://) so existing fsspec-backed users aren't silently broken.

Concrete impact: xarray's test_open_fsspec fails against zarr 3.2.0 with GroupNotFoundError: No group found in store 'memory://out2.zarr' at path '', even though the data is present in fsspec's MemoryFileSystem. Any downstream library doing the same write-via-mapper / read-via-URL pattern will silently read from an empty store.

Steps to reproduce

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
#   "fsspec",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

import fsspec
import zarr

# Write a zarr group through fsspec's MemoryFileSystem
m = fsspec.filesystem("memory")
mm = m.get_mapper("out.zarr")
g = zarr.open_group(mm, mode="w", zarr_format=3)
g.attrs["hello"] = "world"

# The data is in fsspec's MemoryFileSystem
print("fsspec contents:", m.find("/"))
# -> ['/out.zarr/zarr.json']

# Reading the same data via the matching URL fails in 3.2.0
zarr.open_group("memory://out.zarr", mode="r")
# zarr.errors.GroupNotFoundError:
#   No group found in store 'memory://out.zarr' at path ''

Additional output

No response

[d-v-b]

thanks for this report! For as long as we rely on this type of behavior from fsspec, we need tests for it. Long term, I'm not sure that users should assume the behavior you demonstrate in the example. We don't really want users to assume that all memory-based storage uses fsspec.

IMO the way forward is for zarr to own a registry of store classes. Users should be able to associate the scheme "memory://" to a store class (e.g., zarr-python's managed memory store, or fsspec's memory storage).

For backwards compatibility, we could dynamically modify the default state of this registry based on installed packages. So if fsspec is installed, fsspec's memory storage would be used by default.

[maxrjones]

yeah, I agree with both the short and long-term solutions

[d-v-b]

is it important for the xarray tests that the fsspec-backed memory store is used, or would any url-addressible memory storage (e.g., zarr's) work for this test?

[d-v-b]

e.g., can the xarray tests do this instead:

g = zarr.open_group("memory://out.zarr", mode="w", zarr_format=3)
g.attrs["hello"] = "world"

# Reading the same data via the matching URL fails in 3.2.0
zarr.open_group("memory://out.zarr", mode="r")

and not use fsspec at all