[titiler.xarray] Use obstore to access remote datasets in xarray_open_dataset

[maxrjones]

@sharkinsspatial suggested in our ODD checkin yesterday trying out obstore in the XarrayReader. Since titiler doesn't rely on any distributed frameworks, I believe this should be possible today using the fsspec integration to get a file handler for NetCDF/HDF files. I may try to run some quick benchmarks to see if this impacts performance in any way.

Once zarr-developers/zarr-python#1661 is merged, obstore could be used for Zarr stores with notable performance benefits.

[vincentsarago]

Note: Obstore support in zarr-python was introduce in zarr-python==3.0.7 but sadly zarr-python>=3.0 is only available for python >=3.11 https://github.com/zarr-developers/zarr-python/blob/v3.0.8/pyproject.toml#L33 😬

we won't be able to support this until we remove support for python 3.9/3.10 in titiler.xarray

ref https://devguide.python.org/versions/

[maxrjones]

we won't be able to support this until we remove support for python 3.9/3.10 in titiler.xarray

Do you support python versions through their end of life? Zarr follows NEP 29 - https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table.

[vincentsarago]

Do you support python versions through their end of life?

we don't have a strict rule but yeah we usually try to support as much as we can

[j08lue]

we usually try to support as much as we can

Applaudable not to drop support for Python versions pre-EOL lightly. Manifold performance gain here might be a strong enough case, though?

We could explore this as part of our EOPF Sentinel Zarr Explorer efforts. cc @emmanuelmathot

[vincentsarago]

FYI, we do not provide a titiler.xarray application, so any user who want to have a Obstore backend for Zarr can absolutely do it without requiring us to change the titiler.xarray module.

import obstore
from zarr.storage import ObjectStore

def open_dataset(src_path: str, **kwargs: Any):
    """Open Xarray dataset

    Args:
        src_path (str): dataset path.

    """
    store = obstore.store.from_url(src_path)
    zarr_store = ObjectStore(store=store, read_only=True)
    ds = xarray.open_dataset(
        zarr_store,
        decode_times=True,
        decode_coords="all",
        consolidated=True,
        engine="zarr",
    )
    return ds

☝️ to replace https://github.com/developmentseed/titiler/blob/main/src/titiler/xarray/titiler/xarray/io.py#L14

Which is the approach we're doing for titiler-eopf https://github.com/EOPF-Explorer/titiler-eopf

[maxrjones]

we usually try to support as much as we can

Applaudable not to drop support for Python versions pre-EOL lightly. Manifold performance gain here might be a strong enough case, though?

Since titiler.xarray requires xarray and xarray requires python>=3.11, I think titiler.xarray specification of python>=3.9 may already be misleading.

Given the above, why not use the approach from titiler-eopf in titiler.xarray itself?

[vincentsarago]

titiler.xarray requires xarray from rio-tiler and there is no version limit (https://github.com/cogeotiff/rio-tiler/blob/deb5f3b976a26b94677979a2f9c1b88d371446f0/pyproject.toml#L70-L73) which means that we depends on versions that were compatible with python<3.11

[maxrjones]

ah sorry, I'm wasn't following that you allow people to use very old version of xarray, of course that makes sense