Skip to content

fix: preserve shard shape when writing to icechunk - #952

Merged
TomNicholas merged 13 commits into
zarr-developers:mainfrom
TomNicholas:fix/sharded-array-icechunk-roundtrip
Apr 9, 2026
Merged

fix: preserve shard shape when writing to icechunk#952
TomNicholas merged 13 commits into
zarr-developers:mainfrom
TomNicholas:fix/sharded-array-icechunk-roundtrip

Conversation

@TomNicholas

Copy link
Copy Markdown
Member

Summary

  • Fixes the icechunk writer using metadata.chunks (inner chunk shape) instead of metadata.chunk_grid.chunk_shape (shard shape) when creating arrays, causing sharded arrays to lose their shard configuration
  • Adds a regression test that creates a sharded zarr store, virtualizes it, writes to icechunk, and verifies both shard and chunk shapes are preserved

Closes #951

Test plan

  • New test test_sharded_array_roundtrip_icechunk passes
  • Existing icechunk writer tests unaffected (pre-existing test_write_loadable_variable failure is unrelated)

🤖 Generated with Claude Code

TomNicholas and others added 6 commits March 30, 2026 14:50
Sharded arrays lose their shard shape when written to icechunk via
virtualizarr - the shard shape gets replaced with the inner chunk shape.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes zarr-developers#951. The icechunk writer was passing `metadata.chunks` (the inner
chunk shape) to `group.require_array(chunks=...)`. For sharded arrays
this should be `metadata.chunk_grid.chunk_shape` (the shard shape).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Mar 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.30%. Comparing base (fd2665c) to head (758c1ef).
⚠️ Report is 52 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #952      +/-   ##
==========================================
- Coverage   89.31%   89.30%   -0.02%     
==========================================
  Files          33       33              
  Lines        2041     2039       -2     
==========================================
- Hits         1823     1821       -2     
  Misses        218      218              
Files with missing lines Coverage Δ
virtualizarr/writers/icechunk.py 91.35% <100.00%> (-0.11%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comment thread virtualizarr/writers/icechunk.py Outdated
name=name,
shape=metadata.shape,
chunks=metadata.chunks,
chunks=metadata.chunk_grid.chunk_shape,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomNicholas the current change will lose the inner chunk structure. I cannot drop in-line suggestion to L517, but I think this code block should look like:

        filters, serializer, compressors = extract_codecs(metadata.inner_codecs)
        arr = group.require_array(
            name=name,
            shape=metadata.shape,
            chunks=metadata.chunks,
            shards=metadata.shards,
            dtype=metadata.data_type.to_native_dtype(),
            filters=filters,
            compressors=compressors,
            serializer=serializer,
            dimension_names=var.dims,
            fill_value=metadata.fill_value,
        )
  • metadata.chunks / metadata.shards tells require_array both the inner chunk shape and shard shape, so it constructs the ShardingCodec correctly
  • metadata.inner_codecs — extracts the codecs inside the sharding codec (e.g. BytesCodec, ZstdCodec)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh interesting - thanks. Apparently this seemed to work even without this change, due to some subtlety in zarr-python? I/Claude couldn't find a way to make a test that failed before adding your suggestion but passed after.

But I'm made this change now anyway as it's much more explicit.

TomNicholas and others added 2 commits April 9, 2026 12:55
…reation

Use metadata.inner_codecs instead of metadata.codecs so that the
codecs inside the ShardingCodec (e.g. BytesCodec, ZstdCodec) are
extracted directly, and pass metadata.chunks/metadata.shards
explicitly to require_array so it constructs the ShardingCodec
correctly rather than relying on zarr to interpret a passed-through
ShardingCodec serializer.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@TomNicholas
TomNicholas merged commit 9185715 into zarr-developers:main Apr 9, 2026
18 checks passed
@TomNicholas
TomNicholas deleted the fix/sharded-array-icechunk-roundtrip branch April 9, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sharded arrays lose shard shape when written to icechunk

2 participants