I'm opening this issue to track and discuss how our data structure differs from xarray. Ideally I would close it when AnnData could easily be implemented via xarray.

Some previous discussion: #308

The idea

I often think of AnnData as a kind of "special case" of xarray Datasets. We just improve convenience by specializing on the 2d case, plus a few other features. It would be nice if I didn't just think of it that way, and we could actually just use their code here.

sgkit basically accomplishes this. It basically uses a very "anndata shaped"¹ xarray Dataset² for representing genomics data. These data structures and our goals with them are so similar that searching for open issues by the sgkit devs on the xarray repository is a great way to find compatibility issues for anndata.

Additionally, zarr and OME-zarr are quite aligned with xarray.

What's missing

Some things we need, which xarray does not currently provide:

• We have support for the fast sparse array library (ideally we can get pydata/sparse to become fast)
• We support categorical variables
• We support repeated dimensions (e.g. obsp, varp) Repeated coordinates leads to unintuitive (broken?) indexing behaviour pydata/xarray#3731
• We have a nested structure (though it's on the roadmap with datatree being implemented)
• We are actively working on support for awkward arrays (first attempt to support awkward arrays #647 Awkward array backend? pydata/xarray#4285, https://github.com/pystatgen/sgkit/issues/643)