Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generalize dask.delayed calls to go through ChunkManager #7810

Open
TomNicholas opened this issue May 2, 2023 · 0 comments · May be fixed by #7811
Open

Generalize dask.delayed calls to go through ChunkManager #7810

TomNicholas opened this issue May 2, 2023 · 0 comments · May be fixed by #7811
Labels
topic-arrays related to flexible array support topic-chunked-arrays Managing different chunked backends, e.g. dask

Comments

@TomNicholas
Copy link
Member

TomNicholas commented May 2, 2023

[Deepak: Should we add chunked_array_type and from_array_kwargs to open_mfdataset?

I actually don't think we need to - from_array_kwargs is only going to get directly passed down to open_dataset, and hence could be considered part of **kwargs.

This should actually just work, except in the case of parallel=True. For that we could add delayed to the ChunkManager ABC, so that if cubed does implement cubed.delayed it could be added, else a NotImplementedError would be raised. I think all of this wouldn't be necessary if we had lazy concatenation in xarray though (xref #4628). That suggestion would mean we should also replace other instances of dask.delayed in other parts of the codebase though... I think I will split this into a separate issue in the interests of getting this one merged.

Originally posted by @TomNicholas in #7019 (comment)

@github-actions github-actions bot added the needs triage Issue that has not been reviewed by xarray team member label May 2, 2023
@TomNicholas TomNicholas added topic-arrays related to flexible array support and removed needs triage Issue that has not been reviewed by xarray team member labels May 2, 2023
@TomNicholas TomNicholas linked a pull request May 2, 2023 that will close this issue
4 tasks
@TomNicholas TomNicholas added the topic-chunked-arrays Managing different chunked backends, e.g. dask label Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic-arrays related to flexible array support topic-chunked-arrays Managing different chunked backends, e.g. dask
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant