-
Notifications
You must be signed in to change notification settings - Fork 115
update backends tutorials intro #266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
5e2045b
add draft chart to markdown file
JessicaS11 3c65328
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 5c803c2
add mermaid sphinx extension
JessicaS11 67aa3f6
revise backend content
JessicaS11 42aa2ad
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 3bae4fc
add some color styling
JessicaS11 06da29a
add sphinx mermaid package to env
JessicaS11 21ed978
save conflict changes
JessicaS11 bbcb478
Merge branch 'main' into engine-flow
JessicaS11 f28966f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 53659c9
update conda lock files
JessicaS11 bbe08f9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 7d74f63
add myst parser to use mermaid lexer with sphinx
JessicaS11 c7e1adf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] bac54de
fix dash to underscore
JessicaS11 45de572
try changing order of extensions
JessicaS11 8a6a28f
add source_suffix sphinx config
JessicaS11 44c8d64
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 746f9f0
Merge branch 'main' into engine-flow
JessicaS11 382a532
add curly braces
JessicaS11 57acd92
remove diagram (moved to docs) and update text
JessicaS11 da9e806
revert environment changes to main
JessicaS11 5870819
Merge branch 'main' into engine-flow
JessicaS11 0857457
add h5netcdf to list of backends
JessicaS11 daa22e6
Merge branch 'main' into engine-flow
JessicaS11 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,44 +1,49 @@ | ||
# Creating new backends | ||
# Reading data using backends | ||
|
||
## Introduction | ||
|
||
You can [read different type of files](https://docs.xarray.dev/en/stable/user-guide/io.html)) in `xr.open_dataset` by specifying the engine to be used: | ||
You can [read different types of files](https://docs.xarray.dev/en/stable/user-guide/io.html) in `xr.open_dataset` by specifying the engine to be used: | ||
|
||
```python | ||
import xarray as xr | ||
xr.open_dataset("my_file.grib" , engine="cfgrib") | ||
``` | ||
|
||
For each available engine there is an underlying backend, that reads the data and pack them in a dataset. | ||
Navigating Xarray backends can be confusing, | ||
so we recommend checking out [this flow chart](https://docs.xarray.dev/en/stable/user-guide/io.html) | ||
to help you figure out which engine you need and how to use it. | ||
|
||
Xarray bundles several backends internally for the following formats: | ||
You can see what backends are currently available in your working environment | ||
with `xarray.backends.list_engines()`. | ||
|
||
- netcdf4 - netCDF4 | ||
- scipy - netCDF3 | ||
- zarr - Zarr | ||
- pydap - DAP | ||
- ... | ||
## Why use the Xarray backend API to write your own backend? | ||
|
||
External Backends that use the new backend API (xarray >= v0.18.0) that allows to add support for backend without any change to Xarray | ||
- Your users don't need to learn a new interface; they can use `xr.open_dataset` with the `engine` kwarg. | ||
- With little extra effort you can have lazy loading with Dask. Simply implement a function for reading blocks and Xarray will manage lazy loading with Dask for you. | ||
- It's easy to implement: using the backend API (introduced in v0.18.0), you don't need to integrate any code in Xarray. | ||
|
||
- [cfgrib](https://github.com/ecmwf/cfgrib) - GRIB | ||
- [tiledb](https://github.com/TileDB-Inc/TileDB-CF-Py) - TileDB | ||
- [rioxarray](https://corteva.github.io/rioxarray/stable/) - GeoTIFF, JPEG-2000, ESRI-hdr, etc (via GDAL) | ||
- [xarray-sentinel](https://github.com/bopen/xarray-sentinel) - Sentinel-1 SAFE | ||
- ... | ||
## More Information | ||
|
||
## Why using the Xarray backend API | ||
See the [documentation](https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html) for more details on adding and registering a new backend. | ||
|
||
- Your users don't need to learn a new interface that is they can use `xr.open_dataset` with `engine` kwarg. | ||
- With little extra effort you can have lazy loading with Dask. you have to implement a function for reading blocks and Xarray will manage lazy loading with Dask for you | ||
- It's easy to implement: you don't need to integrate any code in Xarray | ||
Follow the tutorials on creating a new backend for binary files. | ||
|
||
```{tableofcontents} | ||
|
||
## Next | ||
``` | ||
|
||
See the [documentation](https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html) for more. | ||
### Links to internal backends | ||
|
||
Follow the tutorial on creating a new backend for binary files. | ||
- [netcdf4](https://pypi.org/project/netCDF4/) - netCDF4 | ||
- [scipy](https://scipy.org/) - netCDF3 | ||
- [zarr](https://pypi.org/project/zarr/) - Zarr | ||
- [pydap](https://pydap.github.io/pydap/) - Data Access Protocol (DAP/DODS/OPeNDAP) | ||
- [h5netcdf](https://h5netcdf.org/) - hdf5 | ||
JessicaS11 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```{tableofcontents} | ||
### Links to external backends (not comprehensive) | ||
|
||
``` | ||
- [cfgrib](https://github.com/ecmwf/cfgrib) - GRIB | ||
- [tiledb](https://github.com/TileDB-Inc/TileDB-CF-Py) - TileDB | ||
- [rioxarray](https://corteva.github.io/rioxarray/stable/) - GeoTIFF, JPEG-2000, ESRI-hdr, etc (via GDAL) | ||
- [xarray-sentinel](https://github.com/bopen/xarray-sentinel) - Sentinel-1 SAFE | ||
- ... |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.