Allow concat() to drop/replace duplicate index labels?

[chunweiyuan]

Right now,

>>> coords_l, coords_r = [0, 1, 2], [1, 2, 3]
>>> missing_3 = xr.DataArray([11, 12, 13], [(dim, coords_l)])
>>> missing_0 = xr.DataArray([21, 22, 23], [(dim, coords_r)])
>>> together = xr.concat([missing_3, missing_0], dim='x')
>>> together
<xarray.DataArray 'missing_3' (x: 6)>
array([11, 12, 13, 21, 22, 23])
Coordinates:
  * x        (x) int64 0 1 2 1 2 3
>>> together.sel(x=1)
<xarray.DataArray 'missing_3' (x: 2)>
array([12, 21])
Coordinates:
  * x        (x) int64 1 1

Would it be OK to introduce a kwarg ("replace"?) that replaces cells of identical coordinates from right to left?

That would render

>>> together
<xarray.DataArray 'missing_3' (x: 6)>
array([11, 21, 22, 23])
Coordinates:
  * x        (x) int64 0 1 2 3

Some people might even want to drop all cells with coordinate collisions (probably not us). If that's the case then the kwarg would be ternary.....

[shoyer]

I don't think this is the best fit for concat. Rather, I would suggest a new method, something like combine_first. See these issues for more discussion: #835 #742

[chunweiyuan]

Is combine_first already implemented? I can't find it in the source code, neither could I find ops.choose.

[shoyer]

Sorry, nope not implemented yet. ops.choose also is not implemented yet, but it would be in the style of stack from core/ops.py.

[chunweiyuan]

So I fooled around with Pandas' combine_first:

>>> df1 = pd.DataFrame({'x':[1,2,3],'z':[4,5,6]}).set_index('x')
>>> df1      
   z
x   
1  4
2  5
3  6
>>> df2 = pd.DataFrame({'x':[1,12,13],'y':[0,5,6],'z':[7,8,9]}).set_index(['x','y'])
>>> df2
      z
x  y   
1  0  7
12 5  8
13 6  9
>>> df1.combine_first(df2)
        z
x  y     
1  0  4.0
12 5  8.0
13 6  9.0

and was surprised that the indexes were not "outer-joined". Is this the behavior xarray wants to emulate?

As a mockup for xr.combine_first(arr1, arr2), I was thinking about using align([arr1, arr2], join="outer") to set up the template, and then go into the template to set the values right. Is that a sound approach? Muchas gracias.

[shoyer]

Indeed, I think we definitely want to do an outer align first. That behavior for pandas looks very strange.

[chunweiyuan]

Here's my somewhat hacky stab at it, not optimized for speed/efficiency:

def combine_first(left, right):
    """
    Takes 2 data arrays, performs an outer-concat, using values from the right array to
    fill in for missing coordinates in the left array.
    """
    l_aligned, r_aligned = xr.align(left, right, join="outer")
    temp = l_aligned + r_aligned   # hack
    temp.values[temp.notnull().values] = np.nan  # now template is all nan
    # insert non-nan values from right array first
    temp.values[r_aligned.notnull().values] = r_aligned.values[r_aligned.notnull().values]
    # insert values from left array, overwriting those from right array
    temp.values[l_aligned.notnull().values] = l_aligned.values[l_aligned.notnull().values]
    return temp

And the result:

>>> ar1 = xr.DataArray([4,5,6],[('x',[1,2,3])])
>>> ar2 = xr.DataArray([[7,8,9],[10,11,12],[13,14,15]],[('x',[1,12,13]),('y',[0,5,6])])
>>> ar1
<xarray.DataArray (x: 3)>
array([4, 5, 6])
Coordinates:
  * x        (x) int64 1 2 3
>>> ar2
<xarray.DataArray (x: 3, y: 3)>
array([[ 7,  8,  9],
       [10, 11, 12],
       [13, 14, 15]])
Coordinates:
  * x        (x) int64 1 12 13
  * y        (y) int64 0 5 6
>>> temp = combine_first(ar1, ar2)
>>> temp
<xarray.DataArray (x: 5, y: 3)>
array([[  4.,   5.,   6.],
       [  4.,   5.,   6.],
       [  4.,   5.,   6.],
       [ 10.,  11.,  12.],
       [ 13.,  14.,  15.]])
Coordinates:
  * x        (x) int64 1 2 3 12 13
  * y        (y) int64 0 5 6
>>> temp = combine_first(ar2, ar1)
>>> temp
<xarray.DataArray (x: 5, y: 3)>
array([[  7.,   8.,   9.],
       [  4.,   5.,   6.],
       [  4.,   5.,   6.],
       [ 10.,  11.,  12.],
       [ 13.,  14.,  15.]])
Coordinates:
  * x        (x) int64 1 2 3 12 13
  * y        (y) int64 0 5 6

Would this be the behavior we want from xarray's combine_first? Not entirely analogous to the Pandas function. Maybe rename it?

[chunweiyuan]

Another test:

>>> left
<xarray.DataArray (x: 2, y: 2)>
array([[1, 1],
       [1, 1]])
Coordinates:
  * x        (x) |S1 'a' 'b'
  * y        (y) int64 -2 0
>>> right
<xarray.DataArray (x: 2, y: 2)>
array([[0, 0],
       [0, 0]])
Coordinates:
  * x        (x) |S1 'b' 'c'
  * y        (y) int64 0 2
>>> combine_first(left, right)
<xarray.DataArray (x: 3, y: 3)>
array([[  1.,   1.,  nan],
       [  1.,   1.,   0.],
       [ nan,   0.,   0.]])
Coordinates:
  * x        (x) object 'a' 'b' 'c'
  * y        (y) int64 -2 0 2
>>> combine_first(right, left)
<xarray.DataArray (x: 3, y: 3)>
array([[  1.,   1.,  nan],
       [  1.,   0.,   0.],
       [ nan,   0.,   0.]])
Coordinates:
  * x        (x) object 'a' 'b' 'c'
  * y        (y) int64 -2 0 2

[shoyer]

Pandas implements this like: result = where(notnull(left), left, right). That's probably how we want want to do it -- much simpler than my proposal in the other issue.

[shoyer]

I think we should still call it combine_first -- the only difference is the alignment behavior.

[chunweiyuan]

Thanks for the reply. Would it make things easier if c, d = xr.align(a, b, join="outer") explicitly broadcasts missing dimensions in all returned values? Currently, if b is missing a dimension, then d would also miss a dimension:

>>> a
<xarray.DataArray (x: 2, y: 2)>
array([[1, 1],
       [1, 1]])
Coordinates:
  * x        (x) |S1 'a' 'b'
  * y        (y) int64 -2 0
>>> b
<xarray.DataArray (x: 3)>
array([4, 5, 6])
Coordinates:
  * x        (x) |S1 'a' 'b' 'd'
>>> c, d = xr.align(a, b, join="outer")
>>> c
<xarray.DataArray (x: 3, y: 2)>
array([[  1.,   1.],
       [  1.,   1.],
       [ nan,  nan]])
Coordinates:
  * x        (x) object 'a' 'b' 'd'
  * y        (y) int64 -2 0
>>> d
<xarray.DataArray (x: 3)>
array([4, 5, 6])
Coordinates:
  * x        (x) object 'a' 'b' 'd'

[shoyer]

Use xarray.broadcast to do xarray style broadcasting (it also aligns). This will be easier after xarray.apply is merged

For now, maybe use Dataset._binary_op - take look at how fillna is implemented. This is actually almost exactly the same as fillna except using an outer join (instead of a left join).

[chunweiyuan]

So I spent some time looking at dataset._calculate_binary_op but couldn't quite come up with what I wanted. After banging my head against the wall a bit this is what I have:

def combine_first(left, right):
    la, ra = xr.align(left, right, join='outer', copy=False)  # should copy=True?
    la, ra = la.where(ra.isnull() | ra.notnull()), ra.where(la.isnull() | la.notnull()) 
    ra.values[la.notnull().values] = la.values[la.notnull().values]
    return ra

And it seems to work. My test cases are

>>> l_2d
<xarray.DataArray (x: 2, y: 2)>
array([[1, 1],
       [1, 1]])
Coordinates:
  * x        (x) |S1 'a' 'b'
  * y        (y) int64 -2 0
>>> r_2d
<xarray.DataArray (x: 2, y: 2)>
array([[0, 0],
       [0, 0]])
Coordinates:
  * x        (x) |S1 'b' 'c'
  * y        (y) int64 0 2
>>> ar_1d
<xarray.DataArray (x: 3)>
array([4, 5, 6])
Coordinates:
  * x        (x) |S1 'a' 'b' 'd'

and here are the results:

>>> combine_first(l_2d, r_2d)
<xarray.DataArray (x: 3, y: 3)>
array([[  1.,   1.,  nan],
       [  1.,   1.,   0.],
       [ nan,   0.,   0.]])
Coordinates:
  * x        (x) object 'a' 'b' 'c'
  * y        (y) int64 -2 0 2
>>> combine_first(r_2d, l_2d)
<xarray.DataArray (x: 3, y: 3)>
array([[  1.,   1.,  nan],
       [  1.,   0.,   0.],
       [ nan,   0.,   0.]])
Coordinates:
  * x        (x) object 'a' 'b' 'c'
  * y        (y) int64 -2 0 2
>>> combine_first(l_2d, ar_1d)
<xarray.DataArray (x: 3, y: 2)>
array([[ 1.,  1.],
       [ 1.,  1.],
       [ 6.,  6.]])
Coordinates:
  * x        (x) object 'a' 'b' 'd'
  * y        (y) int64 -2 0
>>> combine_first(ar_1d, l_2d)
<xarray.DataArray (x: 3, y: 2)>
array([[ 4.,  4.],
       [ 5.,  5.],
       [ 6.,  6.]])
Coordinates:
  * x        (x) object 'a' 'b' 'd'
  * y        (y) int64 -2 0

I don't like the fact that I have to access .values, and the use of .where is slightly wonky. But this is definitely the cleanest working solution I have thus far. Any suggestion to improve is welcome.

[chunweiyuan]

Any suggestion on improvement?

[chunweiyuan]

hmm, so what would be an expected behavior of ds.combine_first?

If I have

>>> ds0
<xarray.Dataset>
Dimensions:  (x: 2, y: 2)
Coordinates:
  * x        (x) |S1 'a' 'b'
  * y        (y) int64 -1 0
Data variables:
    ds0      (x, y) int64 0 0 0 0
>>> ds1
<xarray.Dataset>
Dimensions:  (x: 2, y: 2)
Coordinates:
  * x        (x) |S1 'b' 'c'
  * y        (y) int64 0 1
Data variables:
    ds1      (x, y) int64 1 1 1 1

I get

>>> ds0.combine_first(ds1)
<xarray.Dataset>
Dimensions:  (x: 3, y: 3)
Coordinates:
  * x        (x) object 'a' 'b' 'c'
  * y        (y) int64 -1 0 1
Data variables:
    ds0      (x, y) float64 0.0 0.0 nan 0.0 0.0 nan nan nan nan
    ds1      (x, y) float64 nan nan nan nan 1.0 1.0 nan 1.0 1.0

and changing the order to ds1.combine_first(ds0) just flips the order of the data_vars, but the cell values of the data_vars remain the same.

This is done essentially by adding a _combine_first to ops.py that mimics _fillna, except join='outer'.

[shoyer]

What about simply adding a keyword argument join to fillna? Then you could you write join='outer' to accomplish combine_first.