Updating remote access notebook

[betolink]

I'm fixing a few typos in the notebook and adding an explicit way of inspecting the I/O behavior of the different caching strategies implemented in fsspec. I'm also mentioning the impact of chunking in access performance, I think this is now a self contained notebook, I guess we could include the internals of Zarr next.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[github-actions]

🎊 PR Preview fe9141c has been successfully built and deployed to https://xarray-contrib-xarray-tutorial-preview-pr-291.surge.sh

🕐 Build time: 0.011s

_{🤖 By surge-preview}

[betolink]

Hi @scottyhq! do you have suggestions on the failing checks? the link that is apparently broken is not really broken and the other is an example not a real link, on the spellcheck bot the fo is a reference to a file-like object.

[scottyhq]

Yeah, the link check unfortunately is finicky and I'm not sure how to exclude specific links. For the spellcheck, you can add fo to the ignore list

xarray-tutorial/.github/workflows/qaqc.yaml

Line 39 in 08e762e

ignore_words_list: hist,nd

Thanks for expanding the notebook, happy to do a full review!

[betolink]

Thanks @scottyhq the PR is ready for review.

[scottyhq]

Thanks @betolink ! This is really such a useful resource. I gave it another full read and added comments and suggestions

[scottyhq]

Going to leave some review comments on the whole notebook top to bottom rather than just the new changes since I didn't go over it carefully the first round!

1. Consider using Jupyter Book admonitions for your notes. Because jupyterlab-myst is in the environment these are rendered similarly on both the website and in jupyterlab

> It is important to note that there are...

```{note}
there are...
```

[scottyhq]

For this first note you say "use of a file handler and a cache" but I didn't see anything about the cache

[scottyhq]

Supported file formats by backend: It's not clear what BufferedIOBase and AbstractDataStore are and where they come from. Consider defining these in a bit more detail, or introduce them later on. What is a "buffer" vs a file?

[scottyhq]

"it’s really an anti pattern when we work with scientific data formats. Benchmarks show that any of the caching schemas will perform better than using the default read-ahead." -> "It's not ideal with common multidimensional binary data formats." Can you link to benchmarking results?

[scottyhq]

file = fsspec.open_local(f"simplecache::{uri}", filecache={'cache_storage': '/tmp/fsspec_cache'})

The keyword argument and uri should match here (filecache::{uri}). If I remember correctly filecache exposes a bit more control and the cache persists if you say close a notebook and come back to it, so I recommend that! I also like using same_names=True so that if you're working with multiple files you can do other things with them (like open in QGIS or other software) if you want.

[scottyhq]

This new ability to keep track of cache activity is super handy! I'm confused though, I think it's important to note that 'total requested bytes' != 'total transferred bytes' ? Also what is the cause of the cache hits and cache misses? :

        <BytesCache:
            block size  :   5242880
            block count :   0
            file size   :   4024972
            cache hits  :   175
            cache misses:   2
            total requested bytes: 4024972>
         
        <BlockCache:
            block size  :   8388608
            block count :   1
            file size   :   4024972
            cache hits  :   45
            cache misses:   1
            total requested bytes: 8388608>

[scottyhq]

"to cloud storage, but using the default caching." --> remove ", but using the default caching"? since you use blockcache in the code below

[scottyhq]

Remote data access and chunking: How about adding a tiny bit more here? For example, I think it's often a great starting point for people wanting to work with remote data to consider 1. the total file size (ds.nbytes) 2. is the file chunked (ds.sst.encoding) 3. what do you want to do with it (in particular are you needing to compute the mean of all pixels or just read a single pixel?).

Maybe a specific example? If I understand correctly, let's say you want the value of a single pixel from s3://sst.mnmean.nc (ds.isel(lon=0,lat=0,time=0)). Using defaults, Xarray dispatches to h5netcdf/h5py which tries to read one 'chunk' (1, 89, 180) containing your pixel of interest. So this request is translated by fsspec to read ~128kb via an HTTP range request to S3, but using the default caching an additional 5MB is read. The entire file size in this case is just 4MB, so for efficiency rather than fiddling with cache settings, it might be best to just use filecache:: so that all your Xarray computations read from the local file rather than possibly reading bits and pieces over the network.

[betolink]

Thanks for the thorough review @scottyhq! I'll address the suggestions early next week. Also, feel free to edit directly in the notebook!