Skip to content

Add asynchronous load method #10327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 142 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
142 commits
Select commit Hold shift + click to select a range
01e7518
new blank whatsnew
TomNicholas Oct 24, 2024
83e553b
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Oct 24, 2024
e44326d
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 8, 2024
4e4eeb0
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 20, 2024
d858059
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 21, 2024
d377780
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 21, 2024
3132f6a
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 23, 2024
900eef5
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 29, 2024
4c4462f
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Dec 4, 2024
5b9b749
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jan 6, 2025
fadb953
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jan 8, 2025
57d9d23
Merge branch 'main' of https://github.com/TomNicholas/xarray
TomNicholas Mar 13, 2025
11170fc
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 19, 2025
0b8fa41
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 20, 2025
f769f85
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 20, 2025
4eef318
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Apr 7, 2025
29242a4
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas May 15, 2025
e6b3b3b
test async load using special zarr LatencyStore
TomNicholas May 15, 2025
3ceab60
don't use dask
TomNicholas May 16, 2025
071c35a
async all the way down
TomNicholas May 16, 2025
29374f9
remove assert False
TomNicholas May 16, 2025
ab12bb8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 16, 2025
62aa39d
add pytest-asyncio to CI envs
TomNicholas May 16, 2025
dfe8bf7
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas May 16, 2025
a906dec
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 16, 2025
629ab31
assert results are identical
TomNicholas May 16, 2025
7e9ae0f
implement async load for dataarray and dataset
TomNicholas May 17, 2025
d288351
factor out common logic
TomNicholas May 17, 2025
e0731a0
consolidate tests via a parametrized fixture
TomNicholas May 17, 2025
9b41e78
async_load -> load_async
TomNicholas May 17, 2025
67ba26a
make BackendArray an ABC
TomNicholas May 18, 2025
9344e2e
explain how to add async support for any backend in the docs
TomNicholas May 18, 2025
f8f8563
add new methods to api docs
TomNicholas May 19, 2025
30ce9be
whatsnew
TomNicholas May 19, 2025
5d15bbd
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas May 19, 2025
1f02de1
Merge branch 'main' into async.load
TomNicholas May 19, 2025
2342b50
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 19, 2025
b6d4a82
Fix ci/minimum_versions.py
TomNicholas May 19, 2025
2079d7e
fix formatting
TomNicholas May 21, 2025
48e4534
concurrently load different variables in ds.load_async using asyncio.…
TomNicholas May 21, 2025
cca7589
test concurrent loading of multiple variables in one dataset
TomNicholas May 21, 2025
dfe9b87
fix non-awaited load_async
TomNicholas May 21, 2025
84099f3
rearrange test order
TomNicholas May 21, 2025
ab000c8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 21, 2025
a8b7b46
add test for orthogonal indexing
TomNicholas May 23, 2025
82c7654
explicitly forbid orthogonal indexing
TomNicholas May 23, 2025
5eacdb0
support async orthogonal indexing via https://github.com/zarr-develop…
TomNicholas May 23, 2025
9f33c09
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas May 23, 2025
093bf50
add test for vectorized indexing (even if it doesn't work)
TomNicholas May 23, 2025
4073a24
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 23, 2025
842a06c
add test for basic indexing
TomNicholas May 23, 2025
e19ab55
correct test to actually use vectorized indexing
TomNicholas May 23, 2025
b9e8e06
refactor to parametrize indexing test
TomNicholas May 23, 2025
8bc7bea
implement async vectorized indexing
TomNicholas May 29, 2025
6c47e3f
revert breaking change to BackendArray
TomNicholas May 29, 2025
a86f646
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 29, 2025
884ce13
remove indirection in _ensure_cached method
TomNicholas May 29, 2025
a43af86
IndexingAdapters don't need async get
dcherian May 29, 2025
17d7a0e
Add tests
dcherian May 29, 2025
d824a2d
Add decoding test
dcherian May 29, 2025
6a13611
Add IndexingAdapter mixin
dcherian May 29, 2025
d79ed54
[cherry] Making decoding arrays lazy too
dcherian May 30, 2025
1da3359
parametrized over isel and sel
TomNicholas May 30, 2025
dded9e0
mock zarr.AsyncArray.getitem in test
TomNicholas May 30, 2025
4c347ad
tidy up the mocking
TomNicholas May 30, 2025
b4ed8ee
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jun 20, 2025
0b55247
Merge branch 'main' into async.load
TomNicholas Jun 20, 2025
4018e28
ensure the correct zarr class's method is patched for each test
TomNicholas Jun 20, 2025
6da81ce
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 20, 2025
7972164
add degenerate test case of no indexing
TomNicholas Jun 20, 2025
618424a
factor out the Latency part of LatencyStore
TomNicholas Jun 20, 2025
cd97481
use mocks in multiple objects test
TomNicholas Jun 20, 2025
75abdec
use mocks in multiple variables test
TomNicholas Jun 20, 2025
7409372
trim latencystore down to just what's needed to dodge https://github.…
TomNicholas Jun 20, 2025
1f79034
parametrizing indexing test over xarray classes
TomNicholas Jun 20, 2025
9881e8d
ensure we actually test vectorized indexing for Variable
TomNicholas Jun 20, 2025
1197798
use create_test_data
TomNicholas Jun 20, 2025
642fd48
add @pytest.mark.asyncio
dcherian Jun 20, 2025
a5a44f5
Merge branch 'main' into async.load
TomNicholas Jul 22, 2025
f22b56b
remove outdated readonly_store
TomNicholas Jul 22, 2025
ebfede5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 22, 2025
e882914
Merge branch 'main' into async.load
TomNicholas Jul 30, 2025
87c7fcb
enable tests to run when recent version of zarr-python is not available
TomNicholas Jul 31, 2025
67c77cc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 31, 2025
0d4bb0f
separate tests to only run on correct versions of zarr
TomNicholas Aug 1, 2025
ed2c808
clear error message if async oindexing not available
TomNicholas Aug 1, 2025
df32020
parametrize over zarr_format
TomNicholas Aug 1, 2025
aba2917
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas Aug 1, 2025
d3e6a64
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 1, 2025
4570aed
add pytest-asyncio to other test CI env
TomNicholas Aug 1, 2025
ddde08e
Merge branch 'main' into async.load
TomNicholas Aug 1, 2025
0ec670e
fix some mypy errors
TomNicholas Aug 1, 2025
a3a3b62
use method directly when possible
TomNicholas Aug 4, 2025
b747f1c
Merge branch 'main' into async.load
TomNicholas Aug 4, 2025
a28a6a9
remove repeated API docs from bad merge
TomNicholas Aug 4, 2025
577cc72
fix bad merge in release note
TomNicholas Aug 4, 2025
a68579f
fix other bad merge in whatsnew
TomNicholas Aug 4, 2025
acc5c94
remove prints
TomNicholas Aug 4, 2025
7776d41
remove last print statement
TomNicholas Aug 4, 2025
0b1ebb5
test async basic indexing raises informative error before zarr-python…
TomNicholas Aug 4, 2025
df09780
test correct error message is raised for each indexing case
TomNicholas Aug 4, 2025
84f8e30
ensure each test runs on the earliest version of xaarr it can
TomNicholas Aug 4, 2025
19090b0
remove pointless repeated getitem
TomNicholas Aug 4, 2025
49416db
set N_LAZY_VARS correctly in test
TomNicholas Aug 4, 2025
2ed8455
remove unused import
TomNicholas Aug 4, 2025
a8a2860
rename flag to make it more clear its only for orthogonal and vectori…
TomNicholas Aug 4, 2025
ef6afdf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 4, 2025
de98308
remove IndexingAdapter special case
TomNicholas Aug 6, 2025
e32ea13
type fixes
dcherian Aug 6, 2025
da2d43c
return a deepcopy
TomNicholas Aug 6, 2025
ac3127f
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas Aug 6, 2025
d46fc3f
try again
dcherian Aug 6, 2025
cc253c7
one more
dcherian Aug 6, 2025
78c9116
Try again
dcherian Aug 6, 2025
a727ecb
try fixing _in_memory error by not returning the adapter class
TomNicholas Aug 11, 2025
959edc2
Merge branch 'main' into async.load
TomNicholas Aug 11, 2025
9b7afc2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 11, 2025
b4ef26f
remove scope=module from fixture for robustness
TomNicholas Aug 11, 2025
a7918e4
modify test to be happy with either error message
TomNicholas Aug 11, 2025
199d50a
use Variable instead of Dataset to avoid race condition of indexing b…
TomNicholas Aug 11, 2025
1cbe913
fix bad merge in API docs
TomNicholas Aug 12, 2025
46d9414
add test to test_backends.py
TomNicholas Aug 12, 2025
b4a5a90
fix bug found by new test, causing pandas indexes to be converted to …
TomNicholas Aug 12, 2025
c5bed0d
Merge branch 'main' into async.load
TomNicholas Aug 12, 2025
bc1fe4e
add test to test_variable.py for lazy async indexing
TomNicholas Aug 12, 2025
10f7e61
move async load tests from test_async.py to test_backends.py
TomNicholas Aug 12, 2025
8416f00
parametrize all tests over zarr_format
TomNicholas Aug 12, 2025
4edd503
remove test_async.py file entirely
TomNicholas Aug 12, 2025
fcb2c11
lint
TomNicholas Aug 12, 2025
4760574
Stricter assertions
TomNicholas Aug 13, 2025
8bce3bb
Update doc/whats-new.rst
TomNicholas Aug 13, 2025
37be891
add docstring for Variable.load_async
TomNicholas Aug 13, 2025
3bd7b8a
make all load-related docstrings consistent
TomNicholas Aug 13, 2025
dfaac7e
note about users being responsible for limiting concurrency
TomNicholas Aug 13, 2025
d0a129a
remove parametrization over zarr_format
TomNicholas Aug 13, 2025
f30a3a0
account for Dataset having multiple lazy vars
TomNicholas Aug 13, 2025
432bbd5
refactor test parametrization to use pytest.param(..., id=...) syntax
TomNicholas Aug 13, 2025
4f40792
refactor TestBackendIndexing to combine sync and async checks in one …
TomNicholas Aug 13, 2025
cf1d127
move test_load_async onto test base class
TomNicholas Aug 13, 2025
54ab88c
Merge branch 'main' into async.load
TomNicholas Aug 13, 2025
dda58bf
should fix mypy error
TomNicholas Aug 13, 2025
02d661d
add back in the parametrize_zarr_format to avoid trying to write v3 d…
TomNicholas Aug 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
  • Loading branch information
pre-commit-ci[bot] committed May 23, 2025
commit 4073a24f563c1010d6fb50b6757b38196909d02c
2 changes: 1 addition & 1 deletion xarray/tests/test_async.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,11 +186,11 @@

# test orthogonal indexing
indexer = {"x": [2, 3]}
result = await ds.sel(indexer).load_async()

Check failure on line 189 in xarray/tests/test_async.py

View workflow job for this annotation

GitHub Actions / ubuntu-latest py3.12 all-but-dask

TestAsyncLoad.test_indexing AttributeError: 'AsyncArray' object has no attribute 'oindex'

Check failure on line 189 in xarray/tests/test_async.py

View workflow job for this annotation

GitHub Actions / macos-latest py3.13

TestAsyncLoad.test_indexing AttributeError: 'AsyncArray' object has no attribute 'oindex'

Check failure on line 189 in xarray/tests/test_async.py

View workflow job for this annotation

GitHub Actions / ubuntu-latest py3.13 all-but-numba

TestAsyncLoad.test_indexing AttributeError: 'AsyncArray' object has no attribute 'oindex'

Check failure on line 189 in xarray/tests/test_async.py

View workflow job for this annotation

GitHub Actions / ubuntu-latest py3.13

TestAsyncLoad.test_indexing AttributeError: 'AsyncArray' object has no attribute 'oindex'
xrt.assert_identical(result, ds.sel(indexer).load())

# test vectorized indexing
# TODO this shouldn't pass! I haven't implemented async vectorized indexing yet...
indexer = xr.DataArray([2, 3], dims=['x'])
indexer = xr.DataArray([2, 3], dims=["x"])
result = await ds.foo[indexer].load_async()
xrt.assert_identical(result, ds.foo[indexer].load())
Loading