Intake, catalogs, and datatree

[eni-awowale]

What is your issue?

Original issue in xarray-contrib/datatree#134 with additional comments.

Intake, catalogs, and datatree

Thanks @TomNicholas and sorry for creating issue noise. I guess I got a bit carried away with these comments in the readme:

• Has functions for mapping user-supplied functions over every node in the tree,
• Automatically dispatches some of xarray.Dataset's API over every node in the tree (such as .isel),

I was thinking that maybe the datatree abstraction could be a more formalised and ultimately 'xarray native' approach to the the problems that have been tackled by e.g. intake-esm and intake-thredds. Leaves in the tree could compositions over netcdf files, which may be aggregated JSON indexes. I guess I was thinking that some sort of formalism over a nested datastructure could help in dask computational graph composition. I have run into issues where the scheduler gets overloaded, or just takes forever to start for calculations across large datasets composed with i.e. mf_opendataset

Originally posted in xarray-contrib/datatree#97 (comment)