Don't read entire chunks at a time

[JackKelly]

Because we're reading entire chunks at a time, we're reading and decompressing a lot more data than we need, which is almost certainly a major bottleneck.

Some ideas:

• Try again to use zarr.core.Array(partial_decompress=True) using FSStore & Blosc compression (we're already using Blosc zstd level 5 for NWPs) to read small chunks, especially for NWPs where we're using tiny 2x2 images.
• Try with uncompressed Zarr. It's possible that the performance increase from compression is far smaller than the performance increase of being able to precisely extract just the data we want from each chunk.

If it's possible to quickly load subsets of each chunk, then modify each Zarr DataSource so it no longer pre-reads entire chunks into memory, but instead, for each batch, loads each example separately using a different thread.

[JackKelly]

highly relevant:

• https://github.com/pystatgen/sgkit/issues/437

which includes this text (copy-and-pasted for convenience):

• zarr-specs#59 (zarr specs for partial chunk reads)
• zarr-python#40 (partial chunk reads feature request)
• zarr-python#521 (partial chunk reads feature request)
• zarr-python#584 (partial decompress draft PR)
• it sounds like even with partial decompress, we will need to fetch full compressed file

[JackKelly]

PR for partial_decompress, including lots of performance graphs: zarr-developers/zarr-python#667

[JackKelly]

This issue will be made obsolete (i.e. won't matter) if #58 works :)

[JackKelly]

No longer relevant. Made obsolete by #58