Compatibility with dask 2021.02.0#4884
Merged
crusaderky merged 3 commits intopydata:masterfrom Feb 11, 2021
Merged
Conversation
keewis
approved these changes
Feb 10, 2021
Collaborator
keewis
left a comment
keewis
left a comment
There was a problem hiding this comment.
looks good to me. I don't know a lot about dask, though, so someone else might need to look at this.
dcherian
reviewed
Feb 10, 2021
xarray/core/dataset.py
Outdated
| # TODO We're wasting a lot of key-level work. We should write a fast | ||
| # variant of HighLevelGraph.cull() that works at layer level | ||
| # only. | ||
| dsk2 = dsk.cull(keys) |
Contributor
There was a problem hiding this comment.
should we add tests for this code path?
Contributor
Author
There was a problem hiding this comment.
I will once dask/dask#7203 is out. I can remove the code path for now if you prefer.
Contributor
Author
There was a problem hiding this comment.
I removed the HLG and reworked the whole thing
dcherian
approved these changes
Feb 11, 2021
dcherian
added a commit
to dcherian/xarray
that referenced
this pull request
Feb 12, 2021
* upstream/master: (24 commits) Compatibility with dask 2021.02.0 (pydata#4884) Ensure maximum accuracy when encoding and decoding cftime.datetime values (pydata#4758) Fix `bounds_error=True` ignored with 1D interpolation (pydata#4855) add a drop_conflicts strategy for merging attrs (pydata#4827) update pre-commit hooks (mypy) (pydata#4883) ensure warnings cannot become errors in assert_ (pydata#4864) update pre-commit hooks (pydata#4874) small fixes for the docstrings of swap_dims and integrate (pydata#4867) Modify _encode_datetime_with_cftime for compatibility with cftime > 1.4.0 (pydata#4871) vélin (pydata#4872) don't skip the doctests CI (pydata#4869) fix da.pad example for numpy 1.20 (pydata#4865) temporarily pin dask (pydata#4873) Add units if "unit" is in the attrs. (pydata#4850) speed up the repr for big MultiIndex objects (pydata#4846) dim -> coord in DataArray.integrate (pydata#3993) WIP: backend interface, now it uses subclassing (pydata#4836) weighted: small improvements (pydata#4818) Update related-projects.rst (pydata#4844) iris update doc url (pydata#4845) ...
dcherian
added a commit
to dcherian/xarray
that referenced
this pull request
Feb 17, 2021
* upstream/master: FIX: h5py>=3 string decoding (pydata#4893) Update matplotlib's canonical (pydata#4919) Adding vectorized indexing docs (pydata#4711) Allow fsspec URLs in open_(mf)dataset (pydata#4823) Fix typos in example notebooks (pydata#4908) pre-commit autoupdate CI (pydata#4906) replace the ci-trigger action with a external one (pydata#4905) Update area_weighted_temperature.ipynb (pydata#4903) hide the decorator from the test traceback (pydata#4900) Sort backends (pydata#4886) Compatibility with dask 2021.02.0 (pydata#4884)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4860
Reverts #4873
Restore compatibility with dask 2021.02.0 by avoiding improper assumptions on the implementation details of
da.Array.__dask_postpersist__().This PR does not align xarray to the new dask collection spec (dask/dask#7093), as I just realized that Datasets violate the rule of having all dask keys with the same name if they contain more than one dask variable - and cannot do otherwise. So I have to change the dask collection spec again to accommodate them.