array.nchunks is ambiguous #3296

Activity

1. dstansby commented on Jul 25, 2025

   

   dstansby

   on Jul 25, 2025

   Contributor

   More actions

   At the moment it reports the number of chunks in the shards:

   >>> import zarr
   >>> arr = zarr.create_array(store={}, shape=(8,), shards=(4,), chunks=(2,), dtype='uint8')
   >>> arr.nchunks
   4

   which seems to make sense to me? What's the situation/reasoning for it to report the number of shards instead?
2. d-v-b commented on Jul 25, 2025

   

   d-v-b

   on Jul 25, 2025

   ContributorAuthor

   More actions

   the docstring says "The number of chunks in the stored representation of this array." For v3 data with sharding, the number of stored objects is not the same as the number of chunks.
3. d-v-b commented on Jul 25, 2025

   

   d-v-b

   on Jul 25, 2025

   ContributorAuthor

   More actions

   nchunks_initialized has the same problem -- these properties come from a time before there was a difference between the number of chunks and the number of files stored on disk. that means we kind of need to pick one of the two meanings for these properties. for nchunks_initialized I think it's clear that the intent is to report the number of objects on disk supporting the array. I'm extending that logic to nchunks.
4. LDeakin commented on Jul 25, 2025

   

   LDeakin

   on Jul 25, 2025

   Member

   More actions

   A bit of a curveball, but what if .chunks / .nchunks instead consistently mapped to the on-disk chunk shape, irrespective of the presence of the sharding codec? This is how it was in the zarr-python 3.0.0 betas.

   At the moment, reading/writing "chunk-by-chunk" with a sharded array is a bit of a footgun. And workarounds like this are necessary:

   • https://github.com/zarrs/zarr_benchmarks/blob/9c07a5ccf8994c76a9a02fcf9cb60a0a42b4860c/src/zarr_benchmarks/zarrs_python_benchmark_read.py#L54
   • https://github.com/zarrs/zarr_benchmarks/blob/9c07a5ccf8994c76a9a02fcf9cb60a0a42b4860c/src/zarr_benchmarks/zarrs_dask_python_benchmark_roundtrip.py#L32

   Introduce a new .subchunks concept instead? Perhaps a good time to change this is when you start supporting non-regular chunk grids, because .chunks will break anyway.
5. d-v-b commented on Jul 26, 2025

   

   d-v-b

   on Jul 26, 2025

   ContributorAuthor

   More actions

   @LDeakin have a look at #3299, I go all-in on distinguishing shards (on-disk chunk shape) from chunks (unit of concurrent reading). If I understand you correctly, this would remove these footguns
6. d-v-b commented on Jul 27, 2025

   

   d-v-b

   on Jul 27, 2025

   ContributorAuthor

   More actions

   A bit of a curveball, but what if .chunks / .nchunks instead consistently mapped to the on-disk chunk shape, irrespective of the presence of the sharding codec?

   can you explain what you mean by "on-disk chunk shape" here?
7. LDeakin commented on Jul 27, 2025

   

   LDeakin

   on Jul 27, 2025

   Member

   More actions

   The chunk shape defined by the chunk grid.
8. d-v-b commented on Jul 27, 2025

   

   d-v-b

   on Jul 27, 2025

   Last edited by d-v-b

   ContributorAuthor

   More actions

   there are two complications with the idea of making the top-level Array attributes which refer to "chunks" to express the in-metadata chunks / chunk_grid.chunk_shape attributes:

   • When reading data, people expect that the chunks attribute conveys the unit of parallelism for that operation. With sharding, this is no longer true.
   • We are already distinguishing between shards and chunks in our array creation API. chunks specifies the unit of read parallelism, and shards, if specified, turns on sharding and allows separate control over the unit of write parallelism. So if you specify chunks alone, it controls the chunks / chunk_grid.chunk_shape attributes, but if you specify chunks and shards together, the chunks controls the chunk_shape attribute of the sharding codec config, and shards controls the chunks / chunk_grid.chunk_shape attributes. IMO we should try to be consistent with this terminology, which argues for normalizing "shards" as the in-memory name for the chunks / chunk_grid.chunk_shape attributes.
9. maxrjones

   added

   design discussionDiscussion about the design of the libraryDiscussion about the design of the library

   on Jun 26, 2026