Pointwise indexing -- something like sel_points

[shoyer]

@hdail suggested that it would be useful to be able to index points in addition to indexing dimensions separately. Right now, you can do this via something like:
xray.concat([ds.sel(x=x, y=y) for x, y in pts], dim='station')

This would be handy for sampling particular points out of multiple dimensions, e.g., to compare gridded and station data.

It's also probably worth implementing in xray because it can be done quickly using numpy's fancy indexing (I think) which would be more efficient than a loop, though I suspect the implementation is probably somewhat complex.

[shoyer]

This operation is actually sort of like reindexing. So perhaps this should be spelled ds.reindex_like(other) or ds.reindex(other.coords). With labeled dimensions and variables there is enough metadata to make the reshaping unambiguous.

[WeatherGod]

Unless I am missing something about xray, that selection operation could only work if pts had values that exactly matched coordinate values in ds. In most scenarios, that would not be the case. One would have to first build pts from a computation of nearest-neighbor indexs between the stations and the model grid.

[shoyer]

@WeatherGod You are totally correct. The last dataset on which I have needed to do this was an unprojected grid with a constant increment of 0.5 degrees between points, so finding nearest neighbors was easy.

If you have a lot of points to select at, finding nearest neighbor points could be done efficiently with a tree, e.g., scipy.spatial.cKDTree: http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.KDTree.query.html#scipy.spatial.KDTree.query

[WeatherGod]

Just managed to implement this using your suggestion for my data:

from scipy.spatial import cKDTree as KDTree
kd = KDTree(zip(model['longitude'].values.ravel(), model['latitude'].values.ravel()))
dists, indx = kd.query(zip(obs['longitude'], obs['latitude']))
indx = np.unravel_index(indx, mod['longitude'].shape)
mod_points = xray.concat([mod.isel(x=x, y=y) for y, x in zip(*indx)], dim='station')

Not entirely certain why I needed to reverse y and x in that last part, but, oh well...

[shoyer]

@WeatherGod Very nice! I'm not entirely sure why you have to reverse y and x at the end, either -- what is the order of the dimensions on mod['longitude']?

[WeatherGod]

Starting using the above snippet for more datasets, some with interdependent coordinates and some without (so the coordinates would be 1-d). I think I have generalized it significantly...

def grid_to_points(grid, points, coord_names):
    not_spatial = set(grid.dims) - set(coord_names)
    spatial_selection = {n:0 for n in not_spatial}
    spat_only = grid.isel(**spatial_selection)
    coords = []
    for i, n in enumerate(spat_only.dims):
        if spat_only[n].ndim != len(spat_only.dims):
            # Needs new axes
            slices = [np.newaxis] * len(spat_only.dims)
            slices[i] = slice(None)
        else:
            slices = [slice(None)] * len(spat_only.dims)
        coords.append(spat_only[n].values[slices])
    coords = [c.flatten() for c in np.broadcast_arrays(*coords)]

    kd = KDTree(zip(*coords))
    _, indx = kd.query(zip(*[points[n].values for n in spat_only.dims]))
    indx = np.unravel_index(indx, spat_only.shape)

    return xray.concat((grid.isel(**{n:j for n, j in zip(spat_only.dims, i)})
                        for i in zip(*indx)), dim='station')

I can still imagine some situations where this won't work, such as a requested set of dimensions that are a mix of dependent and independent variables. Currently, if the dimensions are independent, then the number of dimensions of each one is assumed to be 1 and np.newaxis is used for the others. Meanwhile, if the dimensions are dependent, then the number of dimensions for each one is assumed to be the same as the number of dependent variables and is merely flattened (the broadcast is essentially no-op).

I should also note that this is technically not restricted to spatial coordinates even though the code says so. Just anything that can be represented in euclidean space.

[WeatherGod]

Oh, and it does take advantage of a bunch of python2.7 features such as dictionary comprehensions and generator statements, so...

[WeatherGod]

And, I think I just realized how I could generalize it even more. Right now, grid can only be a DataArray, but I would like this to work for a DataSet as well. I bet if I use .sel() instead of .isel() and access the elements of the broadcasted arrays, I could make this work very nicely for both DataArray and DataSet.

[WeatherGod]

And, actually, the example I gave above has a bug in the dependent dimension case. This one should be much better (not fully tested yet, though):

def grid_to_points2(grid, points, coord_names):
    if not coord_names:
        raise ValueError("No coordinate names provided")
    not_spatial = set(grid.dims) - set(coord_names)
    spatial_selection = {n:0 for n in not_spatial}
    spat_only = grid.isel(**spatial_selection)
    coords = []
    for i, n in enumerate(spat_only.dims):
        if spat_only[n].ndim != len(spat_only.dims):
            # Needs new axes
            slices = [np.newaxis] * len(spat_only.dims)
            slices[i] = slice(None)
        else:
            slices = [slice(None)] * len(spat_only.dims)
        coords.append(spat_only[n].values[slices])
    coords = np.broadcast_arrays(*coords)

    kd = KDTree(zip(*[c.flatten() for c in coords]))
    _, indx = kd.query(zip(*[points[n].values for n in spat_only.dims]))
    indx = np.unravel_index(indx, coords[0].shape)

    return xray.concat(
            (grid.sel(**{n:c[i] for n, c in zip(spat_only.dims, coords)})
             for i in zip(*indx)),
            dim='station')

[shoyer]

The only part that wouldn't work for a Dataset is spat_only.shape. On a dataset, you can get that information from the values of the dims dictionary (the difference between dims on a dataset and dataarray is definitely an ugly corner of the API).

Also, you probably want to use c.ravel() instead of c.flatten(), because the later always makes a copy.

[shoyer]

The main logic there -- it looks like this is a routine for broadcasting data arrays? I have something similar, but not exactly the same, in xray.core.variable.broadcast_variables. It's also very similar to the logic in xray.Dataset.to_dataframe, e.g., right now I think you could do the broadcasting by doing xray.Dataset.from_dataframe(spat_only.to_dataframe()).

[WeatherGod]

oooh, didn't realize that dims is different for DataSet and DataArray... Gonna have to fix that, too. I am checking out the broadcasting functions you pointed out. The one limitation I see right away with xray.core.variable.broadcast_variables is that it is limited to two variables (presumedly, I would be broadcasting N number of coordinates because the variables may or may not have extraneous dimensions that I don't care to broadcast)

[WeatherGod]

to/from_dateframe just ate up all my memory. I think I am going to stick with my broadcasting approach...