map_blocks #3276

dcherian · 2019-09-02T15:32:42Z

xref Implementing map_overlap #3147, closes arithmetic resulting in inconsistent chunks #3323
Tests added
Passes black . && mypy . && flake8
Fully documented, including whats-new.rst for all changes and api.rst for new API

This is a second attempt. It works with Datasets also in that it can pass Dataset chunks to the user's function. I haven't figured out how to reconstruct a Dataset from things returned from the user's function. For DataArrays, I currently assume that the user function returns chunks of the same shape as the provided chunk.

A few questions:

I wonder if we should introduce a meta xarray object that describes the final result that user expects back i.e. a template DataArray or Dataset with chunks, dtype etc. specified. We could have a utility function that makes it easy to create these templates from existing objects.
If the user function doesn't return anything, (e.g. it writes to a file), how do I "compute" the graph? In this case, we could rename this function to apply_blocks and release that till we figure out point 1?
Should we also add Dataset.to_delayed and DataArray.to_delayed? This would make use-cases like the one in point 2 easier.

ping @mrocklin

xarray/core/parallel.py

dcherian · 2019-09-08T04:05:21Z

I can reconstruct Datasets now. ~~This version requires a template kwarg to know what the returned object looks like. I'll add some code to infer that next.~~

Now infers properties of the returned object.

dcherian · 2019-09-14T03:12:39Z

Ready for review.

@crusaderky I'd like to add DataArray.map_blocks and Dataset.map_blocks but don't know how to work around the circular import issue.

xarray/core/parallel.py

xarray/tests/test_dask.py

dcherian · 2019-09-14T13:56:19Z

Thanks for the extremely thorough review @crusaderky. I'll work on this slowly over the next few days

1. skip index graph nodes. 2. var → name 3. quicker dataarray creation. 4. Add restrictions to docstring. 5. rename chunk construction task. 6. error when non-xarray object is returned. 7. restore non-coord dims. review

* master: Fix whats-new date :/ Revert to dev version Release v0.13.0 auto_combine deprecation to 0.14 (pydata#3314) Deprecation: groupby, resample default dim. (pydata#3313) Raise error if cmap is list of colors (pydata#3310) Refactor concat to use merge for non-concatenated variables (pydata#3239) Honor `keep_attrs` in DataArray.quantile (pydata#3305) Fix DataArray api doc (pydata#3309) Accept int value in head, thin and tail (pydata#3298) ignore h5py 2.10.0 warnings and fix invalid_netcdf warning test. (pydata#3301) Update why-xarray.rst with clearer expression (pydata#3307) Compat and encoding deprecation to 0.14 (pydata#3294) Remove deprecated concat kwargs. (pydata#3288) allow np-array levels and colors in 2D plots (pydata#3295) Remove some deprecations (pydata#3292) Make argmin/max work lazy with dask (pydata#3244) Add head, tail and thin methods (pydata#3278) Updater to testing environment name (pydata#3253)

* upstream/master: Fix concat bug when concatenating unlabeled dimensions. (pydata#3362) Rolling minimum dependency versions policy (pydata#3358) added geocube and rioxarray to related projects (pydata#3383) Lint (pydata#3373) Docs/more fixes (pydata#2934) Explicitly keep track of indexes with merging (pydata#3234)

dcherian · 2019-10-09T01:38:12Z

hahaha the conflict was that you deleted dask_array_compat! I added it back in for meta_from_array

crusaderky · 2019-10-10T09:24:34Z

@dcherian make_meta is failing on dask 1.2

dcherian · 2019-10-10T14:57:43Z

@crusaderky should be fixed.

jhamman · 2019-10-10T18:02:19Z

@crusaderky - are we happy here?

crusaderky · 2019-10-10T22:46:47Z

@dcherian I gave it a polish pass, please review my changes.

dcherian · 2019-10-10T23:19:37Z

Added one line to docstring. Looks good otherwise. Thanks @crusaderky

crusaderky · 2019-10-10T23:44:15Z

Thank you

mrocklin · 2019-10-10T23:49:12Z

Woo!

…

On Thu, Oct 10, 2019 at 4:44 PM crusaderky ***@***.***> wrote: Merged #3276 <#3276> into master. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3276?email_source=notifications&email_token=AACKZTDCOM7M7XVOVQ6CCA3QN645VA5CNFSM4IS6SKCKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOUEZ5DQI#event-2704527809>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTHDHVURZWN5R5TPSXDQN645VANCNFSM4IS6SKCA> .

jhamman · 2019-10-10T23:59:37Z

Fantastic work all! This is going to be a killer addition to the project. 🎉

jakirkham · 2019-10-11T00:03:25Z

Congratulations! 🎉

shoyer · 2019-10-11T00:34:05Z

yes, nice work @dcherian and @crusaderky !

dcherian · 2019-10-11T01:37:42Z

!

@shoyer

commit b0c336f Author: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> Date: Mon Oct 21 04:52:36 2019 -0400 Whatsnew for #3419 (#3422) * pyupgrade --py36-plus * Update xarray/core/nputils.py Co-Authored-By: crusaderky <crusaderky@gmail.com> * Update xarray/core/parallel.py Co-Authored-By: crusaderky <crusaderky@gmail.com> * Update xarray/tests/test_cftime_offsets.py Co-Authored-By: crusaderky <crusaderky@gmail.com> * Update xarray/tests/test_cftime_offsets.py Co-Authored-By: crusaderky <crusaderky@gmail.com> * whatsnew commit 2984415 Author: Rhys Doyle <48983334+rdoyle45@users.noreply.github.com> Date: Mon Oct 21 01:17:47 2019 +0100 Revert changes made in #3358 (#3411) * Revert #3358 * Revision * Code review * Merge from master * Obsolescence note commit 3c462b9 Author: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> Date: Sun Oct 20 20:16:58 2019 -0400 Python3.6 idioms (#3419) * pyupgrade --py36-plus * Update xarray/core/nputils.py Co-Authored-By: crusaderky <crusaderky@gmail.com> * Update xarray/core/parallel.py Co-Authored-By: crusaderky <crusaderky@gmail.com> * Update xarray/tests/test_cftime_offsets.py Co-Authored-By: crusaderky <crusaderky@gmail.com> * Update xarray/tests/test_cftime_offsets.py Co-Authored-By: crusaderky <crusaderky@gmail.com> commit 9886e3c Author: crusaderky <crusaderky@gmail.com> Date: Sun Oct 20 23:42:36 2019 +0100 Temporarily mark pseudonetcdf-3.1 as incompatible (#3420) commit 0f7ab0e Author: Dan Nowacki <dnowacki@usgs.gov> Date: Thu Oct 17 14:13:44 2019 -0700 Fix and add test for groupby_bins() isnan TypeError. (#3405) * Fix and add test for groupby_bins() isnan TypeError. * Better testing * black commit 6cd50cc Author: pmallas <6051395+pmallas@users.noreply.github.com> Date: Thu Oct 17 10:15:47 2019 -0400 Update where docstring to make return value type more clear (#3408) * Update where docstring to make return value type more clear The where docstring was a little confusing to me. I misunderstood "Same type as caller' to mean the values in the xarray not the xarray itself. I think this small change will clean it up for most users. Thanks. * Update xarray/core/common.py Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> commit 55b1ac0 Author: keewis <keewis@users.noreply.github.com> Date: Thu Oct 17 05:13:39 2019 +0200 tests for arrays with units (#3238) * create the empty test file * add tests for data array aggregation functions * include pint in the ci * ignore missing type annotations for pint * really skip the tests if pint is not available * remove the reason from the importorskip call * test that the dataarray constructor does not strip the unit * convert every unit stripped warning to an error * work around pint not implementing np.allclose yet * remove the now unnecessary filterwarnings decorator * xfail all tests that depend on pint having a __array_function__ * treat nans as equal * implement tests for simple arithmetic operations * use param's id argument to assign readable names * add tests for sel() and isel() * add more readable names for the unary arithmetics * xfail every test that is not yet xfailing These don't pass because the constructor raises a unit stripped warning - fixed in pint#764. * only xfail if pint is not the current development version This is test is not really reliable, but sufficient for now. * always use lists instead of tuples for indexing * add tests for loc and squeeze * black * add names and xfail marks to the parameters * add tests for interp and interp_like * implement tests for reindex * remove the xfail marks where it is not clear yet that pint is to blame * add tests for reindex_like * don't pass the new DataArray to a kwarg * xfail if not on pint dev * refactor the tests * add tests for univariate and bivariate ufuncs * black * xfail aggregation only if pint does not implement __array_function__ yet * remove the global filterwarnings mark apparently, this caused the tests to change behavior, resulting in different errors, or causing tests to pass that should actually fail. * add a test case for the repr * create a pytest mark that explicitly requires pint's __array_function__ * also check the string representation in addition to the repr * add helpers for creating method tests * check that simple aggregation methods work * use format() instead of format strings * make sure the repr of method calls is different from functions * merge the two aggregation tests * explicitly check whether pint supports __array_function__ relying on versions is somewhat fragile. * provide a fallback for the new base quantity * check that no warning is raised for both with and without coords * also check that the repr works both with and without coords * wrap all aggregation function calls * xfail every call that fails because of something outside xarray * xfail tests related to dimension coordinates and indexes * use the dimensions from the original array * allow passing arguments to the method on call * add tests for comparisons * add tests for detecting, filling and dropping missing values * mark the missing value tests as requiring pint to support duck arrays * add tests for isin, where and interpolate_na * reformat unit ids and add a test parameter for compatible units * remove an unnecessary xfail * add tests for the top-level replication functions (*_like) * check for whatever pint does with *_like functions * add tests for combine_first * xfail the bivariate ufunc tests * xfail the call to np.median * move the top-level function tests out of the DataArray namespace class * add cumsum and cumprod to the list of aggregation functions * add tests for the numpy methods * check for equal units directly after checking the magnitude * add tests for content manipulation methods * add tests for comparing DataArrays (equals, indentical) * add a test for broadcast_equals * refactor the comparison operation tests * rewrite the strip, attach and assert_equal functions and add extract * preserve multiindex in strip and attach * attach the unit from element "data" as fallback * fix some small typos * compare QuantityScalar and QuantitySequence based on their values * make the isel test more robust * add tests for reshaping and reordering * unify the structure of the tests * mark the remaining tests as requiring a recent pint version, too * explicitly handle quantities as parameters * change the repr of the function / method wrappers * check whether __init__ and repr / str handle units in data and coords * generalize array_attach_units * move the redefinition of DimensionalityError * identify quantities using isinstance * typo * skip tests with a pint version without __array_function__ * compare DataArrays where possible * mark only the compatible unit as xfailing * preserve the name of data arrays * also attach units to x_mm * Test in more CI environments; documentation * What's New * remove a stale function * use Quantity directly for instance tests * explicitly set roll_coords to silence a deprecation warning * skip the whole module if pint does not implement __array_function__ the advantage is that now forgetting to decorate a test case is not possible. * allow to attach units using the mapping from extract_units * add tests for computation methods resampling fails until I figure out how to use it with non-datetime coords. * add tests for grouped operations * add a test for rolling_exp * add a todo note for the module level skip on __array_function__ * add a test for dot * use attach_units instead of manually attaching * modify the resample test to actually work * add a test for to_unstacked_dataset * update whats-new.rst and installing.rst * reformat the whats-new.rst entry * What's New commit 1f81338 Author: keewis <keewis@users.noreply.github.com> Date: Wed Oct 16 20:54:27 2019 +0200 Fixes to the resample docs (#3400) * add a missing newline to make sphinx detect the code block * update the link to the pandas documentation * explicitly state that this only works with datetime dimensions * also put the datetime dim requirement into the function description * add Series.resample and DataFrame.resample as reference * add the changes to whats-new.rst * move references to the bottom of the docstring commit 3f9069b Author: Joseph Hamman <jhamman@ucar.edu> Date: Mon Oct 14 14:22:08 2019 -0700 Revert to dev version commit 62943e2 Author: Joseph Hamman <jhamman@ucar.edu> Date: Mon Oct 14 12:51:46 2019 -0700 Release v0.14.0 commit 30472ec Author: Joseph Hamman <jhamman@ucar.edu> Date: Mon Oct 14 12:24:05 2019 -0700 updates for 0.14 release [black only] commit 4519843 Author: Joseph Hamman <jhamman@ucar.edu> Date: Mon Oct 14 12:18:54 2019 -0700 updates for 0.14 release commit 4f5ca73 Author: Deepak Cherian <dcherian@users.noreply.github.com> Date: Mon Oct 14 18:06:53 2019 +0000 Make concat more forgiving with variables that are being merged. (#3364) * Make concat more forgiving with variables that are being merged. * rename test. * simplify test. * make diff smaller. commit ae1d8c7 Author: Crypto Jerônimo <36364045+crypto-jeronimo@users.noreply.github.com> Date: Sun Oct 13 15:38:36 2019 +0100 Fix documentation typos (#3396) commit 863e490 Author: Joe Hamman <jhamman@ucar.edu> Date: Sat Oct 12 17:33:33 2019 -0400 OrderedDict --> dict, some python3.5 cleanup too (#3389) * OrderedDict --> dict, some python3.5 cleanup too * respond to part of @shoyer's review * fix set attr syntax on netcdf4 vars * fix typing errors * update whats new and todo comments * Typing annotations * Typing annotations * Fix regression * More substantial changes * More polish * Typing annotations * Rerun notebooks commit 6851e3e Author: crusaderky <crusaderky@gmail.com> Date: Sat Oct 12 21:05:32 2019 +0100 Annotate LRUCache (#3395) commit 4c05d38 Author: Stephan Hoyer <shoyer@gmail.com> Date: Fri Oct 11 08:47:57 2019 -0700 BUG: overrides to a dimension coordinate do not get aligned (#3393) Fixes GH3377 commit 3f29551 Author: Deepak Cherian <dcherian@users.noreply.github.com> Date: Thu Oct 10 23:44:19 2019 +0000 map_blocks (#3276) * map_block attempt 2 * Address reviews: errors, args + kwargs support. * Works with datasets! * remove wrong comment. * Support chunks. * infer template. * cleanup * cleanup2 * api.rst * simple shape change error check. * Make test more complicated. * Fix for when user function doesn't set DataArray.name * Now _to_temp_dataset works. * Add whats-new * chunks kwarg makes no sense right now. * review feedback: 1. skip index graph nodes. 2. var → name 3. quicker dataarray creation. 4. Add restrictions to docstring. 5. rename chunk construction task. 6. error when non-xarray object is returned. 7. restore non-coord dims. review * Support nondim coords in make_meta. * Add Dataset.unify_chunks * doc updates. * minor. * update comment. * More complicated test dataset. Tests fail :X * Don't know why compute is needed. * work with DataArray nondim coords. * fastpath unify_chunks * comment. * much improved tests. * Change args, kwargs syntax. * Add dataset, dataarray methods. * api.rst * docstrings. * Fix unify_chunks. * Move assert_chunks_equal to xarray.testing. * minor changes. * Better error handling when inferring returned object * wip * wip * better to_array * Docstrings + nicer error message. * remove unify_chunks in map_blocks + better tests. * typing for unify_chunks * address more review comments. * more unify_chunks tests. * Just use dask.core.utils.meta_from_array * get tests working. assert_equal needs a lot of fixing. * more unify_chunks test. * assert_chunks_equal fixes. * copy over meta_from_array. * minor fixes. * raise chunks error earlier and test for map_blocks raising chunk error * fix. * Type annotations * py35 compat * make sure unify_chunks does not compute. * Make tests functional by call compute before assert_equal * Update whats-new * Work with attributes. * Support attrs and name changes. * more assert_equal * test changing coord attribute * fix whats new * rework tests to use fixtures (kind of) * more review changes. * cleanup * more review feedback. * fix unify_chunks. * read dask_array_compat :) * Dask 1.2.0 compat. * documentation polish * make_meta reflow * cosmetic * polish * Fix tests * isort * isort * Add func call to docstrings. commit 291cb80 Author: Deepak Cherian <dcherian@users.noreply.github.com> Date: Thu Oct 10 18:23:20 2019 +0000 Add groupby.dims & Fix groupby reduce for DataArray (#3338) * Fix groupby reduce for DataArray * bugfix. * another bugfix. * bugfix unique_and_monotonic for object indexes (uniqueness is enough) * Add groupby.dims property. * update reduce docstring to point to xarray.ALL_DIMS * test for object index dims. * test reduce dimensions error. * Add whats-new * fix docs build * sq whats-new * one more test. * fix test. * undo monotonic change. * Add dimensions to repr. * Raise error if no bins. * Raise nice error if no groups were formed. * Some more error raising and testing. * Add dataset tests. * update whats-new. * fix tests. * make dims a cached lazy property. * fix whats-new. * whitespace * fix whats-new commit 3f0049f Author: crusaderky <crusaderky@gmail.com> Date: Wed Oct 9 19:01:29 2019 +0100 Speed up isel and __getitem__ (#3375) * Variable.isel cleanup/speedup * Dataset.isel code cleanup * Speed up isel * What's New * Better error checks * Speedup * type annotations * Update doc/whats-new.rst Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> * What's New * What's New * Always shallow-copy variables commit 132733a Author: Deepak Cherian <dcherian@users.noreply.github.com> Date: Tue Oct 8 22:13:47 2019 +0000 Fix concat bug when concatenating unlabeled dimensions. (#3362) * Fix concat bug when concatenating unlabeled dimensions. * Add whats-new * Add back older test. * fix test * Revert "fix test" This reverts commit c33ca34. * better fix commit 6fb272c Author: crusaderky <crusaderky@gmail.com> Date: Tue Oct 8 22:23:46 2019 +0100 Rolling minimum dependency versions policy (#3358) * - Downgrade numpy to 1.14, pandas to 0.20, scipy to 0.19 (24 months old) - Downgrade dask to 1.1 (6 months old) - Don't pin patch versions * Apply rolling policy (see #3222) * Automated tool to verify the minimum versions * Drop Python 3.5 * lint * Trivial cosmetic * Cosmetic * (temp) debug CI failure * Parallelize versions check script * Remove hacks for legacy dask * Documentation * Assorted cleanup * Assorted cleanup * Fix regression * Cleanup * type annotations upgraded to Python 3.6 * count_not_none backport * pd.Index.equals on legacy pandas returned False when comparing vs. a ndarray * Documentation * pathlib cleanup * Slide deprecations from 0.14 to 0.15 * More cleanups * More cleanups * Fix min_deps_check * Fix min_deps_check * Set policy of 12 months for pandas and scipy * Cleanup * Cleanup * Sphinx fix * Overhaul readthedocs environment * Fix test crash * Fix test crash * Prune readthedocs environment * Cleanup * Hack around versioneer bug on readthedocs CI * Code review * Prevent random timeouts in the readthedocs CI * What's New polish * Merge from Master * Trivial cosmetic * Reimplement pandas.core.common.count_not_none commit 3e2a754 Author: Alan D. Snow <alansnow21@gmail.com> Date: Tue Oct 8 09:36:52 2019 -0500 added geocube and rioxarray to related projects (#3383) commit 4254b4a Author: crusaderky <crusaderky@gmail.com> Date: Fri Oct 4 23:17:56 2019 +0100 Lint (#3373) * raise exception instance, not class * isort * isort * Bump mypy version commit 283b4fe Author: Deepak Cherian <dcherian@users.noreply.github.com> Date: Fri Oct 4 17:04:36 2019 +0000 Docs/more fixes (#2934) * Move netcdf to beginning of io.rst * Better indexing example. * Start de-emphasizing pandas * misc. * compute, load, persist docstrings + text. * split-apply-combine. * np.newaxis. * misc. * some dask stuff. * Little more dask. * undo index.rst changes. * link to dask docs on chunks * Fix io.rst. * small changes. * rollingupdate. * joe's review commit dfdeef7 Author: Stephan Hoyer <shoyer@gmail.com> Date: Thu Oct 3 21:42:50 2019 -0700 Explicitly keep track of indexes with merging (#3234) * Explicitly keep track of indexes in merge.py * Typing fixes * More tying fixes * more typing fixes * fixup commit 86fb71d Author: Deepak Cherian <dcherian@users.noreply.github.com> Date: Thu Oct 3 15:41:50 2019 +0000 groupby repr (#3344) * groupby repr * add test. * test datetime and nondim coord * rename test_da → repr_da * Add whats-new * Update doc/whats-new.rst commit dd2b803 Author: Ryan May <rmay@ucar.edu> Date: Wed Oct 2 15:43:44 2019 -0600 Remove setting of universal wheels (#3367) Universal wheels indicate that one wheel supports Python 2 and 3. This is no longer the case for xarray. This causes builds to generate files with names like xarray-0.13.0-py2.py3-none-any.whl, which can cause pip to incorrectly install the wheel when installing from a list of wheel files. commit 21705e6 Author: crusaderky <crusaderky@gmail.com> Date: Tue Oct 1 19:13:55 2019 +0100 Revisit # noqa annotations (#3359) commit fb575eb Author: crusaderky <crusaderky@gmail.com> Date: Tue Oct 1 15:11:21 2019 +0100 Fix codecov.io upload on Windows (#3360) commit 1ab2279 Author: Deepak Cherian <dcherian@users.noreply.github.com> Date: Mon Sep 30 21:12:22 2019 +0000 Add how do I ... section (#3357) * Add how do I ... section * Bbugfix. * Update doc/howdoi.rst Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> * Update doc/howdoi.rst Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> * small updates. * Add more. commit bd1069b Author: Gregory Gundersen <ggundersen@gmail.com> Date: Sun Sep 29 19:39:53 2019 -0400 Add glossary to documentation (#3352) * First draft at terminology glossary. * Made name matching rules more explicit and hopefully clearer. * Amended what's new. * Changes based on feedback. * More changed based on feedback. commit b51683f Author: Anderson Banihirwe <axbanihirwe@ualr.edu> Date: Sun Sep 29 07:50:21 2019 -0600 Documentation improvements (#3328) * Add examples for full_like, zeros_like, ones_like * Add examples for xr.align * Add examples for xr.merge * Update xr.where docstring * Update xr.dot docstring * Update xarray/core/common.py Co-Authored-By: Deepak Cherian <dcherian@users.noreply.github.com> * Update xarray/core/common.py Co-Authored-By: Deepak Cherian <dcherian@users.noreply.github.com> * Update xr.combine_by_coords docstring * Apply black formatting only * More black formatting * Remove unnecessary pandas bits * Fix indentation issues * Update assign and pipe * Update `Dataset.reindex` with examples * Update `Dataset.fillna` with examples * Address styling issues * Update docstring Co-Authored-By: Deepak Cherian <dcherian@users.noreply.github.com> commit f3c7da6 Author: Gregory Gundersen <ggundersen@gmail.com> Date: Sat Sep 28 15:57:36 2019 -0400 Remove `complex.nc` from built docs (#3353) * Rolling back to prevent a different issue from leaking into this one. * Amended what's new. commit 6ece6a1 Author: Tony Tung <ttung@chanzuckerberg.com> Date: Thu Sep 26 22:45:26 2019 -0700 Fix DataArray.to_netcdf type annotation (#3325) It calls DataSet.to_netcdf, which returns Union[bytes, "Delayed", None]. So this should as well. commit 16fdac9 Author: crusaderky <crusaderky@gmail.com> Date: Thu Sep 26 10:38:46 2019 +0100 CI test suites with pinned minimum dependencies (#3346) * CI test suites with pinned minimum dependencies * code review * Clarity re lxml commit ea101f5 Author: Tom Nicholas <35968931+TomNicholas@users.noreply.github.com> Date: Thu Sep 26 10:51:58 2019 +0200 Bugfix/plot accept coord dim (#3345) Bug in plot.line fixed by ensuring 1D coords are cast down to their associated dims. There was previously two particular cases where this would not happen. commit 85c9a40 Author: crusaderky <crusaderky@gmail.com> Date: Wed Sep 25 02:40:54 2019 +0100 CI environments overhaul (#3340) * Rationalize and align CI environments. Add many optional dependencies to individual CI suites. * pynio and cdms2 are not available on Windows * cfgrib causes Python interpreter crash on Windows * dtype of np.arange defaults to int64 on Linux and int32 on Windows * Suppress failure to delete file on Windows * Mark hypotesis tests as @slow

map_block attempt 2

b090a9e

dcherian mentioned this pull request Sep 2, 2019

[WIP] Add map_blocks. #3258

Closed

3 tasks

shoyer reviewed Sep 2, 2019

View reviewed changes

xarray/core/parallel.py Outdated Show resolved Hide resolved

xarray/core/parallel.py Show resolved Hide resolved

dcherian added 2 commits September 2, 2019 17:12

Address reviews: errors, args + kwargs support.

3948798

Works with datasets!

4f159c8

dcherian added 11 commits September 7, 2019 22:06

remove wrong comment.

9179f0b

Support chunks.

20c5d5b

infer template.

b16b237

cleanup

43ef2b7

cleanup2

5ebf738

api.rst

8a460bb

simple shape change error check.

505f3f0

Make test more complicated.

fe1982f

Fix for when user function doesn't set DataArray.name

066eb59

Now _to_temp_dataset works.

83eb310

Add whats-new

008ce29

dcherian changed the title ~~[WIP] map_blocks attempt 2~~ map_blocks Sep 9, 2019

dcherian changed the title ~~map_blocks~~ [WIP] map_blocks Sep 9, 2019

chunks kwarg makes no sense right now.

adbe48e

dcherian changed the title ~~[WIP] map_blocks~~ map_blocks Sep 9, 2019

crusaderky requested changes Sep 14, 2019

View reviewed changes

review feedback:

924bf69

1. skip index graph nodes. 2. var → name 3. quicker dataarray creation. 4. Add restrictions to docstring. 5. rename chunk construction task. 6. error when non-xarray object is returned. 7. restore non-coord dims. review

dcherian force-pushed the map_blocks_2 branch from 0660b80 to 924bf69 Compare September 19, 2019 16:02

dcherian added 2 commits September 19, 2019 16:05

Support nondim coords in make_meta.

8aed8e7

Add Dataset.unify_chunks

d0797f6

dcherian mentioned this pull request Sep 19, 2019

arithmetic resulting in inconsistent chunks #3323

Closed

dcherian added 2 commits October 8, 2019 19:32

read dask_array_compat :)

2c7938a

Dask 1.2.0 compat.

08ed873

crusaderky added 8 commits October 10, 2019 22:46

Merge remote-tracking branch 'upstream/master' into map_blocks_2

67663aa

documentation polish

99d61fc

make_meta reflow

687689e

cosmetic

f588cb6

polish

d476e2f

Fix tests

26a6a0d

isort

6491753

isort

b227bea

crusaderky approved these changes Oct 10, 2019

View reviewed changes

crusaderky mentioned this pull request Oct 10, 2019

map_blocks doesn't work when dask isn't installed #3391

Closed

Add func call to docstrings.

2a41906

crusaderky merged commit 3f29551 into pydata:master Oct 10, 2019

jhamman mentioned this pull request Oct 11, 2019

Implementing map_overlap #3147

Open

dcherian deleted the map_blocks_2 branch October 17, 2019 01:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

map_blocks #3276

map_blocks #3276

dcherian commented Sep 2, 2019 •

edited

Loading

dcherian commented Sep 8, 2019 •

edited

Loading

dcherian commented Sep 14, 2019

dcherian commented Sep 14, 2019

dcherian commented Oct 9, 2019

crusaderky commented Oct 10, 2019

dcherian commented Oct 10, 2019

jhamman commented Oct 10, 2019

crusaderky commented Oct 10, 2019

dcherian commented Oct 10, 2019

crusaderky commented Oct 10, 2019

mrocklin commented Oct 10, 2019 via email

jhamman commented Oct 10, 2019

jakirkham commented Oct 11, 2019

shoyer commented Oct 11, 2019

dcherian commented Oct 11, 2019

map_blocks #3276

map_blocks #3276

Conversation

dcherian commented Sep 2, 2019 • edited Loading

dcherian commented Sep 8, 2019 • edited Loading

dcherian commented Sep 14, 2019

dcherian commented Sep 14, 2019

dcherian commented Oct 9, 2019

crusaderky commented Oct 10, 2019

dcherian commented Oct 10, 2019

jhamman commented Oct 10, 2019

crusaderky commented Oct 10, 2019

dcherian commented Oct 10, 2019

crusaderky commented Oct 10, 2019

mrocklin commented Oct 10, 2019 via email

jhamman commented Oct 10, 2019

jakirkham commented Oct 11, 2019

shoyer commented Oct 11, 2019

dcherian commented Oct 11, 2019

dcherian commented Sep 2, 2019 •

edited

Loading

dcherian commented Sep 8, 2019 •

edited

Loading