fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata - #4334
Draft
d-v-b wants to merge 6 commits into
Draft
fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata#4334d-v-b wants to merge 6 commits into
d-v-b wants to merge 6 commits into
Conversation
d-v-b
force-pushed
the
fix/zero-length-single-invariant
branch
from
September 9, 2026 18:28
7c5688b to
302b638
Compare
…, clamps and metadata Invariant: a chunk edge length is always >= 1; a dimension's extent may be 0, in which case the dimension has zero chunks (ceildiv(0, size) == 0). Zero-length-axis bugs have recurred since 2017 (#150, #241, #303, zarr-developers#972, zarr-developers#1977, zarr-developers#2434, zarr-developers#3711, zarr-developers#4305, zarr-developers#4307, zarr-developers#4328) because the layers disagreed on this invariant and every span-derived chunk spelling clamped on its own: - The metadata layer (common.py, metadata/v3.py) required chunk edges >= 1, but the in-memory FixedDimension allowed size == 0 with four special-case branches left over from zarr-developers#2434, so normalization could build a grid the metadata constructor then rejected. FixedDimension now rejects size < 1 and the four `if self.size == 0` branches are gone. VaryingDimension already required edges > 0 and is unchanged. - `chunks=-1`, `chunks=False`, `chunks="auto"` (_guess_regular_chunks, both the typesize == 0 early return and the np.maximum line) and `shards="auto"` each derived "one chunk covering the axis" independently. They now all go through one helper, `_full_span_chunk_size(span) = max(span, 1)`, which is the single definition of that phrase for a possibly zero-length axis. - Zarr format 2 metadata had no chunk >= 1 check, so a legacy `chunks: [0]` document opened fine and read uninitialised memory after a resize. It now raises a clear ValueError at parse time, matching the format 3 grid. - Rectilinear grids had no creation-time spelling for a zero-length axis: normalize_chunks_1d required sum(edges) == span, which no list of positive edges can satisfy for span 0, even though the same state is reachable via resize((0,)) and round-trips through reopen. For span == 0 any non-empty list of positive edges is now accepted verbatim, producing the same VaryingDimension(edges, extent=0) that resize produces; the strict sum check is kept for span > 0. Tests: the per-spelling regression test from zarr-developers#4328 is replaced by one matrix over {-1, False, "auto", 1, (1,...), [[2, 2]]} x {(0,), (0, 4), (4, 0), (0, 0), ()} x {v2, v3} x {no shards, shards="auto" with and without a byte budget, explicit shards}, with separate small tests for each error case. Tests that constructed FixedDimension(size=0) now assert it raises, and a zero-extent test covers the behaviour the old special cases were guarding. Assisted-by: ClaudeCode:claude-fable-5-1
zarr-python 2.18.7 writes `chunks: [0]` for `zarr.zeros((0,), chunks=False)` and for `chunks=(0,)`, so stores with that document exist. Rejecting them at open would turn a previously-readable array into an error; leaving the 0 in place read uninitialised memory after a resize. Normalize the edge to 1 with a ZarrUserWarning instead — the same grid every other "one chunk spans the axis" spelling produces — and keep rejecting a zero edge on an axis that has data. Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
Measured against zarr 2.18.7: `zeros((0,), chunks=False)`, `chunks=-1` and `chunks=(0,)` all write `chunks: [0]`, after which nchunks, read, write, append, resize and reopen-then-read every raise ZeroDivisionError. There was never a working behaviour to preserve; normalizing the edge to 1 makes such arrays usable for the first time. Say so in the comment and fragment instead of claiming the stores were previously readable. Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b
force-pushed
the
fix/zero-length-single-invariant
branch
from
September 9, 2026 18:28
302b638 to
047a92e
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4334 +/- ##
=======================================
Coverage 94.34% 94.34%
=======================================
Files 92 92
Lines 12935 12938 +3
=======================================
+ Hits 12203 12206 +3
Misses 732 732
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Makes chunk sizes positive while allowing zero-length array axes. FixedDimension now rejects size=0, and full-span chunk inference uses a shared helper returning max(span, 1). Rectilinear creation accepts a non-empty list of positive chunk sizes on an empty axis and retains those sizes for later growth.
For legacy V2 metadata, a stored chunk edge of 0 on an axis with extent 0 is interpreted as chunk size 1 with a ZarrUserWarning. A zero chunk edge on a positive-length axis is rejected. This is a compatibility policy for the current grid model; the metadata does not prove whether chunk payloads exist in the store.
The historical evidence is narrower than the original description claimed. In an isolated Python 3.12 environment with zarr 2.18.7, NumPy 2.2.6 and numcodecs 0.15.1, creating an empty int32 array with chunks=False, -1, or (0,) stores chunks:[0]. For each spelling, nchunks, full reads, empty full writes, append, resize, and reopen-then-read raise ZeroDivisionError. Attribute updates and attribute reads succeed, so saying these arrays could not perform any operation was incorrect. This test does not establish behavior for every 2.x release or every release from 3.0 through 3.3, and those universal claims have been removed.
Tests cover six chunk spellings over empty/scalar shapes and applicable format/sharding combinations: 124 cases execute and 26 unsupported combinations are skipped. Separate cases check invalid chunk edges and positive-extent V2 rejection. Audit validation:
hatch run test.py3.12-minimal:pytest tests/test_metadata/test_v2.py tests/test_chunk_grids.py tests/test_unified_chunk_grid.py -q— 681 passed, 26 skipped. The former 7409-test total was a historical run, not reproduced in this audit.The legacy-reader choice remains reviewable: normalize empty-axis zero edges with a warning, as implemented here, or reject them. Neither choice requires claiming that no historical reader ever handled such metadata. This PR does not reconstruct a universal history of zero-length-axis bugs or attribute that history to a counted set of authors.