You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The following functions are currently implemented using map_direct, which assumes that the input arrays have been materialized as Zarr arrays so they can be accessed directly, typically in a way that crosses block boundaries.
concat
flip
groupby_blockwise
index (i.e. __getitem__)
map_overlap
It would however be possible to re-implement them using general_blockwise which maps an output block to an arbitrary set of input blocks. The advantage is that the optimizer would then be able to fuse operations to avoid materializing intermediate Zarr arrays, which would potentially be much more efficient.
These are all quite tricky to re-implement (particularly indexing, although the ndindex library would be helpful), so this is more of a long-term aspiration.
The text was updated successfully, but these errors were encountered:
As noted in cubed-dev/cubed-xarray#26, in the case of index, the special case of slices that don't cross chunk boundaries it would be easy to implement, and useful since it's so common.
The following functions are currently implemented using
map_direct
, which assumes that the input arrays have been materialized as Zarr arrays so they can be accessed directly, typically in a way that crosses block boundaries.concat
flip
groupby_blockwise
index
(i.e.__getitem__
)map_overlap
It would however be possible to re-implement them using
general_blockwise
which maps an output block to an arbitrary set of input blocks. The advantage is that the optimizer would then be able to fuse operations to avoid materializing intermediate Zarr arrays, which would potentially be much more efficient.These are all quite tricky to re-implement (particularly indexing, although the
ndindex
library would be helpful), so this is more of a long-term aspiration.The text was updated successfully, but these errors were encountered: