Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-implement map_direct functions using general_blockwise #464

Closed
5 tasks done
tomwhite opened this issue May 18, 2024 · 2 comments
Closed
5 tasks done

Re-implement map_direct functions using general_blockwise #464

tomwhite opened this issue May 18, 2024 · 2 comments

Comments

@tomwhite
Copy link
Member

tomwhite commented May 18, 2024

The following functions are currently implemented using map_direct, which assumes that the input arrays have been materialized as Zarr arrays so they can be accessed directly, typically in a way that crosses block boundaries.

  • concat
  • flip
  • groupby_blockwise
  • index (i.e. __getitem__)
  • map_overlap

It would however be possible to re-implement them using general_blockwise which maps an output block to an arbitrary set of input blocks. The advantage is that the optimizer would then be able to fuse operations to avoid materializing intermediate Zarr arrays, which would potentially be much more efficient.

These are all quite tricky to re-implement (particularly indexing, although the ndindex library would be helpful), so this is more of a long-term aspiration.

@tomwhite
Copy link
Member Author

As noted in cubed-dev/cubed-xarray#26, in the case of index, the special case of slices that don't cross chunk boundaries it would be easy to implement, and useful since it's so common.

@tomwhite
Copy link
Member Author

tomwhite commented Oct 2, 2024

the special case of slices that don't cross chunk boundaries it would be easy to implement, and useful since it's so common

Fixed in #586

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant