Internal refactor of label-based data selection

[benbovy]

Xarray label-based data selection now relies on a newly added xarray.Index.query(self, labels: Dict[Hashable, Any]) -> Tuple[Any, Optional[None, Index]] method where:

• labels is a always a dictionary with coordinate name(s) as key(s) and the corresponding selection label(s) as values
• When calling .sel with some coordinate(s)/label(s) pairs, those are first grouped by index so that only the relevant pairs are passed to an Index.query
• the returned tuple contains the positional indexers and (optionally) a new index object

For a simple pd.Index, labels always corresponds to a 1-item dictionary like {'coord_name': label_values}, which is not very useful in this case, but this format is useful for pd.MultiIndex and will likely be for other, custom indexes.

Moving the label->positional indexer conversion logic into PandasIndex.query(), I've tried to separate pd.Index vs pd.MultiIndex concerns by adding a new PandasMultiIndex wrapper class (it will probably be useful for other things as well) and refactor the complex logic that was implemented in convert_label_indexer. Hopefully it is a bit clearer now.

Working towards a more flexible/generic system, we still need to figure out how to:

• pass index query extra arguments like method and tolerance for pd.Index but in a more generic way
• handle several positional indexers over multiple dimensions possibly returned by a custom "meta-index" (e.g., staggered grid index)
• handle the case of positional indexers returned from querying >1 indexes along the same dimension (e.g., multiple coordinates along x with a simple pd.Index)
• pandas indexes don't need information like the names or shapes of their corresponding coordinate(s) to perform label-based selection, but this kind of information will probably be needed for other indexes (we actually need it for advanced point-wise selection using tree-based indexes in xoak).

This could be done in follow-up PRs..

Side note: I've initially tried to return from xindexes items for multi-index levels as well (not only index dimensions), but it's probably wiser to save this for later (when we'll tackle the multi-index virtual coordinate refactoring) as there are many places in Xarray where this is clearly not expected.

Happy to hear your thoughts @pydata/xarray.

[Illviljan]

This will compute lazy values. Is that what we want here? Or should we do like this?

Suggested change

	x = x.values
	x = x.data

[benbovy]

Not sure what is the use case for this, but I don't think many Xarray users have already tried providing to .sel() slice objects with elements that consist of large Variable or DataArray objects?

We could still check their dimensions (ndim == 0) before coercing it as Numpy arrays if necessary.

[shoyer]

This only comes up if somebody uses a slice object, e.g., ds.sel(x=slice(i, j)). In this case i and j would be loaded into memory.

I think this is probably OK (it is the existing behavior) but on the other hand it would also probably be slightly safer to use .data, which would not implicitly convert a dask array.

[benbovy]

Just using .data wouldn't work with the code below, which expects a numpy array. Is there an easy way to get the scalar value from a dask or any duck array? I don't think so? dask/dask#2961. But I guess we could check len(np.shape(x)) != 0 and raise before coercing to a numpy array?

[benbovy]

But I guess we could check len(np.shape(x)) != 0 and raise before coercing to a numpy array?

Hmm checking this for any x wouldn't work with multi-index and slice objects with tuple elements:

idx = pd.MultiIndex.from_tuples([(0, 'a'), (1, 'b')]
da = xr.DataArray(idx)
da.sel(dim_0=slice((0, 'a'), (1, 'b')))

np.shape((0, 'a'))  # returns (2,)

[benbovy]

Thanks everyone for your comments on this PR!

I'll leave this PR open for another few days in case someone has not yet had a chance to review it. Otherwise, if that's ok I plan to merge it sometime early next week so that I can move forward with the refactoring. In the next step, I'd like to try and see if we can update the data model (i.e., break the "1-dimension <-> 1-index" relationship) and get rid of multi-index virtual coordinates (use real coordinates instead) with minimal user-facing changes (or ideally none).