.zarr.zip on s3

[jeffpeck10x]

I searched and could not find an example of accessing a .zarr.zip from an s3 endpoint without having to first download it entirely. The provided zarr.storage.ZipStore only works on a local path (right?).

I experimented and found that this works:

import s3fs
import zarr
from fsspec import FSMap
from fsspec.implementations.zip import ZipFileSystem

def get_s3_zarr_zip(s3_path):
    s3 = s3fs.S3FileSystem()
    f = s3.open(s3_path)
    fs = ZipFileSystem(f, mode="r")
    store = FSMap("", fs, check=False)

    # optionally use lru cache or otherwise return zarr.group(store=store)
    cache = zarr.storage.LRUStoreCache(store, max_size=2**28)
    return zarr.group(store=cache)

I am wondering if this is alright. Is there anything that could be improved with this approach?

My use-case is read-only. I understand that this approach would not be able to handle updates without updating the entire .zarr.zip.

And more generally, if someone else is searching for a solution like this, I hope this helps!

[rabernat]

This is great @jeffpeck10x! Would you be willing to add this to the tutorial?

I wonder whether the built-in ZipStore has any use for reading from S3. Or is fsspec's ZipFileSystem always required here?

[jeffpeck10x]

I would be very happy to contribute this @rabernat . Please point me to where it would best fit in the tutorial and I will submit a PR.

I think what is novel here is that the ZipFileSystem will handle fetching parts of the zip file without pulling down the whole thing. And s3fs will translate those fetches into ranged requests.

My understanding is that the caveat to this formally being part of ZipStore is that it must be read-only. (Actually, I am not sure what would happen if I tried to pass mode="w" and tried to make an update...)

An alternative might be to update the FSStore to look for the .zip extension? 🤷

Either way, that this works with only a few lines of code perhaps means that it does not require its own storage class, as long as it is documented.

[rabernat]

Please point me to where it would best fit in the tutorial and I will submit a PR.

https://github.com/zarr-developers/zarr-python/blob/main/docs/tutorial.rst

There is already a section addressing Zip stores here - https://zarr.readthedocs.io/en/stable/tutorial.html#storage-alternatives - but only on file storage, not object storage.

Zarr's built-in ZipStore class does support writing mode. However, that would never work on object storage, because object storage doesn't support partial writes to a single object. Instead the workflow for creating a zip store in the cloud seems to be:

• Create the data on a local disk
• Zip it up
• Upload to object storage
• Access using the method you described above

It would be nice to address this full life cycle in the documentation.

[joshmoore]

An additional option would be to use fsspec's handler chaining:

        store = zarr.storage.FSStore(
                url=f"zip::file://{filepath}",  mode="r"
        )

With thanks to @caviere (see https://github.com/zarr-developers/outreachy_2022_testing_zipstore/blob/main/real%20%20world%20data/main.py#L54)

Regardless, 👍 for your version and/or this one being in the tutorial location that would have helped you find them! Thanks.

[jeffpeck10x]

made a small pr to update docs: #1615