Help with DataTree compatibility of NASA Earthdata netCDF4s

[itcarroll]

The DataTree design treats inheritance a bit differently than netCDF4, so of course we can find tons of "quirky" NASA Earthdata that does not work as well as we would like with DataTree! I'm wondering aloud on the best path forward for one particular but common case, and hoping for any input.

Here's a thing you can do (and is done, e.g. dimension "wavelength_3d" in these netCDF4s) in a netCDF4 file:

with netCDF4.Dataset("demo.nc", "w") as f:
    f.createDimension("x", 3)
    f.createGroup("a")
    f["a"].createVariable("y", float, ("x",))
    f["a"]["y"][:] = [0.1, 2.3, 4.5]
    f.createGroup("b")
    f["b"].createVariable("x", int, ("x",))
    f["b"]["x"][:] = [0, 1, 2]

In this file, the variable "y" in group "a" has the same dimension as the coordinate variable "x" in group "b". We know it is the same dimension because dimensions are inherited in netCDF4 and the "x" dimension was defined at a common ancestor.

DataTree design includes coordinate inheritance. So according to DataTree, the variable "y" in group "a" has no coordinates or index. This is the quirk, common in many real netCDF4s, that I am seeking to smooth out.

Right now, I'm working around this quirk by manually assigning coordinates; like this:

with xarray.open_datatree("demo.nc") as f:
    f["a"]["x"] = f["b"]["x"]

I have no idea if this copies data (hope not!), but the main difficulty is you have to know where to find each variable's coordinates.

Unless I'm missing something, a "fix" could be pursued as a change to DataTree or as a re-write the netCDF4 files.

• re-write netCDF4 files: I'm certain it would be a non-starter to pursue relocating "b/x" to "/x" in these files. It may be possible to pursue using HDF5 to create a link "/x" or "a/x" to "b/x" that DataTree would read as a coordinate without changes.
• change DataTree: It may be possible for DataTree to create inherited coordinates based on the presence of inherited dimensions. In "demo.nc", DataTree would see that "x" is the same in groups "a" and "b" based on netCDF4's inheritance rules and go ahead and let group "a" inherit the coordinates from group "b".

Is either change worth pursuing?

[rupesh2]

Hi @itcarroll , Thank you for raising this issue with grouped datasets. We also have examples of such quirky grouped datasets, such AVIRIS-3 L2A.

I understand that rewriting the files to move the coordinate variables to the root will be not an option for many cases. The second option - "changing DataTree" - is also not trivial, because netCDF4 allows the same variable name to appear in different groups. Building on your example, another variable x could be added to the "demo.nc" file within a new group "c", with the x variable appearing twice, as \b\x (int) and \c\x(float).

with netCDF4.Dataset("demo.nc", "w") as f:
    f.createDimension("x", 3)
    f.createGroup("a")
    f["a"].createVariable("y", float, ("x",))
    f["a"]["y"][:] = [0.1, 2.3, 4.5]
    f.createGroup("b")
    f["b"].createVariable("x", int, ("x",))
    f["b"]["x"][:] = [0, 1, 2]
    f.createGroup("c")
    f["c"].createVariable("x", float, ("x",))
    f["c"]["x"][:] = [8.3, 1.5, 2.4]

We do encourage the data providers to use the coordinates variable attributes in netCDFs and assign the relative paths of the coordinates variables, following the CF conventions. If those attributes are present, we can request changes to DataTree that the coordinates attributes be respected to find and inherit the correct coordinates. For example, the coordinates attributes for a variable in the PACE data you shared could be set as geophysical_data/Rrs:coordinates = "../navigation_data/longitude ../navigation_data/latitude";

[itcarroll]

Since "explicit is better than implicit", embedding correct "coordinates" attributes into the variables is definitely a good idea! This could proceed in parallel on the data producer's side while having xarray recognize "coordinates" attributes that use relative or absolute paths is implemented. Interestingly, it seems that recognizing the "coordinates" attribute comes with xarray rather than cf_xarray, which may create some headwind when it comes to an enhancement.

[itcarroll]

In your example, I'm not convinced there is a problem with having multiple coordinates for variable "/a/y". Currently, both "/b/x" and "/c/x" get an index created when using open_datatree, which is great. Both "/b/x" and "/c/x" could be listed as inherited coordinates. At the DataArray level, there's nothing wrong with multiple coordinates that each have an index (but you can't use them both at the same time).

da = xr.DataArray(
    [0.1, 2.3, 4.5],
    dims=["x"],
    coords={
    "b/x": ("x", [0, 1, 2]),
    "c/x": ("x", ["a", "b", "c"]),
    },
    name="y",
)
da = da.set_xindex("b/x")
da = da.set_xindex("c/x")

Currently the path-like naming has no significance.

[itcarroll]

One last thing, I'm not concerned at the moment about the 'latitude' and 'longitude' coordinates which are 2D. I'm starting with the simpler case of the 1D "wavelength_3d" coordinate in the PACE/OCI data I linked in my question.