Xarray bridge

[pp-mo]

Provide a bridge to convert data to+from Xarray

Enable use of Iris+Xarray together, with fast+efficient exchange between the two.

Primary Goals:

• lossless, file-less exchange of information between Iris and Xarray
  • (i.e. cubes <--> xr.Dataset)
• should be ~equivalent to file exchange : iris-save/xr-load, or the converse
• but...
  • fast and easy !
  • no-copy and lazy-preserving transfer of (large) data arrays

Roughly following the ideas (#4835 now F-B https://github.com/SciTools/iris/tree/FEATURE_xarray_readwrite )
And especially to enable possible solutions listed in this comment

Also relevant (to lazy data) : ideas in pp-mo/iris#73
But NB while functional, this approach is very clunky : see generally simpler scheme suggested here

[pp-mo]

#5024 is the beginnings of a new approach to this.

[dcherian]

lossless, file-less exchange of information between Iris and Xarray

Is there something missing in Xarray's to_iris and from_iris methods? See https://github.com/pydata/xarray/blob/cdab326bab0cf86f96bcce4292f4fae24bddc7b6/xarray/convert.py

[pp-mo]

lossless, file-less exchange of information between Iris and Xarray

Is there something missing in Xarray's to_iris and from_iris methods? See https://github.com/pydata/xarray/blob/cdab326bab0cf86f96bcce4292f4fae24bddc7b6/xarray/convert.py

Not really as such, but there is quite a list of features that we can't get from that which we'd like.
As stated, we'd really like to be able to exchange data freely between the two for a "best of both" approach.

I haven't really looked into the differences between iris.save(cubes) and xr.convert.from_iris(cubes).to_netcdf().
But I have looked extensively into differences between iris.load('var') and xr.convert.to_iris(xr.open_dataset()['var']
Apart from some corner cases and problems which could be addressed, there is a fairly long list of things which Iris recognises, which are lost in loading 'via' xarray in this way.
Here's a summary ...

• global attributes
• coordinate systems, attached via a 'grid_mapping' link
• ancillary variables and cell measures, which behave very much like auxiliary coordinates, but are not (and should not be) treated as coordinates by Xarray
• "some" auxiliary coordinates, not treated as coordinates by xarray -- notably, string data

We also think there are probably some tricky corners relating to masked/missing data and time-values handling (with more obscure non-standard calendars). But those one can mostly at least work around, by turning off interpretation in xarray.

So, "tactically" here, the elephant in the room is that to_iris operates on a DataArray not a Dataset, so the variables which are connected to data-variable by CF concepts are not available. Hence, no aux-coords, cell-measures, coordinate systems etc.
But also "strategically", as I already said, it's a fundamentally limited approach for Xarray to attempt to second-guess Iris's handling of CF concepts, which really belongs in Iris, since in that respect what Iris does can be seen as (perhaps should be ?!?) an extension to xarray's scope, to add CF handling. (Which sounds rather like cf-array, but that's another story).

[cpelley]

ping @TomekTrzeciak FYI

[DPeterK]

@pp-mo I have to say - and following on from @dcherian's very good point - that I'd prefer any work to improve the Iris-XArray bridge to be done as enhancements to the functionality that already exists within XArray, rather than being added as new functionality within Iris. I think it would be confusing for users to have multiple APIs for this at all (e.g. xr.dataset.to_iris() and cube.to_xarray_dataset()) let alone also have them return differing results...

@dcherian to add to @pp-mo's very detailed answer above, the main thing is that the round-trip (that is Iris - XArray - Iris) results in a different end object to what was started with. Apologies - we've been meaning to contribute to XArray's Iris bridge with Iris-perspective updates so that the round-trip is lossless for years, because the functionality is super useful, but unfortunately other work pressures have always blocked us doing so 😑

[dcherian]

Apologies - we've been meaning to contribute to XArray's Iris bridge with Iris-perspective updates so that the round-trip is lossless for years, because the functionality is super useful, but unfortunately other work pressures have always blocked us doing so

I fully understand :)

That said, given all the subtle details in @pp-mo's answer, perhaps this code would fit better, and be better maintained, in Iris than in Xarray. Note that pandas to xarray conversion code lives in xarray and pandas just calls that. So we could just have xarray.Dataset.to_iris call the iris method

[DPeterK]

ping @jacobtomlinson, to whom I mentioned this work earlier

[pp-mo]

Latest ideas on this, are for it to become a new independent repo "scitools/ncdata"
Xarray-Iris bridge proposal -- NcData.pdf

A key idea is that it exists in its own space, with no required dependency on either Xarray or Iris