Build Out Store Offerings

[ilan-gold]

We currently only do file system store: https://github.com/ilan-gold/zarrs-python/blob/main/src/codec_pipeline_store_filesystem.rs but should expand to, at the minimum, HTTP.

• Integrate HTTP Store feat: add HTTP store support #61
• Flag for using Opendal (better performance? default for cloud urls like s3:// and gcs:// ?)
• Cache store? Somehow enabling https://docs.rs/zarrs/latest/zarrs/array/trait.ChunkCache.html if we detect an LRUCacheStore on the python side?
• Consolidated metadata: Add partial Zarr V3 consolidated metadata support zarrs#55
• Investigate exposing the zarr-python ByteGetter ByteSetter to Rust to potentially support any store (fallback only?)

[kylebarron]

👋 Long time no see!

I've been working on a library that may be helpful here: pyo3-object_store. It's designed to define builders once, so that many Python-Rust bindings can reuse the same ObjectStore integration.

It enables your functions that use object store to be as simple as:

#[pyfunction]
pub fn use_object_store(store: PyObjectStore) {
    let store: Arc<dyn ObjectStore> = store.into_inner();
}

I've been meaning to update and publish a new release of pyo3-object_store. It would be cool to prototype an integration here.

[ilan-gold]

Hi @kylebarron funny how that happens.

I think (@LDeakin can correct me) that one would need to implement the necessary traits on the object_store object from https://docs.rs/zarrs_storage/0.3.0/zarrs_storage/index.html to get a compatible store for zarrs (to work with https://github.com/ilan-gold/zarrs-python/blob/main/src/store/manager.rs#L17).

But where that happens and how is probably more a matter of discussion. That would be very cool though!

[kylebarron]

That's this, right? https://docs.rs/zarrs_object_store/latest/zarrs_object_store/

[ilan-gold]

Ah ok, @LDeakin is very comprehensive!

[LDeakin]

Cool stuff @kylebarron! Yeah, I'd be interested to see an integration