fix: preserve shard shape when writing to icechunk - #952
Merged
TomNicholas merged 13 commits intoApr 9, 2026
Merged
Conversation
Sharded arrays lose their shard shape when written to icechunk via virtualizarr - the shard shape gets replaced with the inner chunk shape. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes zarr-developers#951. The icechunk writer was passing `metadata.chunks` (the inner chunk shape) to `group.require_array(chunks=...)`. For sharded arrays this should be `metadata.chunk_grid.chunk_shape` (the shard shape). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
for more information, see https://pre-commit.ci
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #952 +/- ##
==========================================
- Coverage 89.31% 89.30% -0.02%
==========================================
Files 33 33
Lines 2041 2039 -2
==========================================
- Hits 1823 1821 -2
Misses 218 218
🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TomNicholas
temporarily deployed
to
test-release
March 31, 2026 18:22 — with
GitHub Actions
Inactive
TomNicholas
temporarily deployed
to
test-release
March 31, 2026 20:32 — with
GitHub Actions
Inactive
maxrjones
requested changes
Apr 8, 2026
| name=name, | ||
| shape=metadata.shape, | ||
| chunks=metadata.chunks, | ||
| chunks=metadata.chunk_grid.chunk_shape, |
Member
There was a problem hiding this comment.
@TomNicholas the current change will lose the inner chunk structure. I cannot drop in-line suggestion to L517, but I think this code block should look like:
filters, serializer, compressors = extract_codecs(metadata.inner_codecs)
arr = group.require_array(
name=name,
shape=metadata.shape,
chunks=metadata.chunks,
shards=metadata.shards,
dtype=metadata.data_type.to_native_dtype(),
filters=filters,
compressors=compressors,
serializer=serializer,
dimension_names=var.dims,
fill_value=metadata.fill_value,
)- metadata.chunks / metadata.shards tells require_array both the inner chunk shape and shard shape, so it constructs the ShardingCodec correctly
- metadata.inner_codecs — extracts the codecs inside the sharding codec (e.g. BytesCodec, ZstdCodec)
Member
Author
There was a problem hiding this comment.
oh interesting - thanks. Apparently this seemed to work even without this change, due to some subtlety in zarr-python? I/Claude couldn't find a way to make a test that failed before adding your suggestion but passed after.
But I'm made this change now anyway as it's much more explicit.
8 tasks
…reation Use metadata.inner_codecs instead of metadata.codecs so that the codecs inside the ShardingCodec (e.g. BytesCodec, ZstdCodec) are extracted directly, and pass metadata.chunks/metadata.shards explicitly to require_array so it constructs the ShardingCodec correctly rather than relying on zarr to interpret a passed-through ShardingCodec serializer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
for more information, see https://pre-commit.ci
maxrjones
approved these changes
Apr 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
metadata.chunks(inner chunk shape) instead ofmetadata.chunk_grid.chunk_shape(shard shape) when creating arrays, causing sharded arrays to lose their shard configurationCloses #951
Test plan
test_sharded_array_roundtrip_icechunkpassestest_write_loadable_variablefailure is unrelated)🤖 Generated with Claude Code