Inconsistent output in GroupBy.apply returning a DataFrame

[TomAugspurger]

This is a continuation of #13056, #14927, and #13056, which were closed by #31613. I think that PR ensured that we consistently take one of two code paths. This issue is to verify that we actually want the behavior on master.

Focusing on a specific pair of examples that differ only in whether the returned index is the same or not:

# master

In [10]: def f(x):
    ...:     return x.copy()  # same index


In [11]: def g(x):
    ...:     return x.copy().rename(lambda x: x + 1)  # different index

In [12]: df = pd.DataFrame({"A": ['a', 'b'], "B": [1, 2]})

In [13]: df.groupby("A").apply(f)
Out[13]:
   A  B
0  a  1
1  b  2

In [14]: df.groupby("A").apply(g)
Out[14]:
     A  B
A
a 1  a  1
b 2  b  2

# 1.0.4
In [8]: df.groupby("A").apply(f)
Out[8]:
     A  B
A
a 0  a  1
b 1  b  2

In [9]: df.groupby("A").apply(g)
Out[9]:
     A  B
A
a 1  a  1
b 2  b  2

So the 1.0.4 behavior is to always prepend the group keys to the result as an index level.
In pandas 1.1.0, whether the group keys are prepended depends on whether the udf returns a dataframe with an identical index. Do we want that kind of value-dependent behavior?

@jorisvandenbossche's notebook from #13056 (comment) might be helpful, though it might be out of date.

[WillAyd]

I don't know that I have a strong enough preference to change anything with where things are now. Inconsistency is annoying, but I don't know that the old behavior is necessarily correct enough to revert back to, and that itself is inconsistent when you consider that the method being applied here is shape-preserving (e.g. a typical groupby.transform wouldn't prepend the groupings)

[jorisvandenbossche]

The difference becomes a bit more apparent with a slightly larger dataframe (repeating the "a" key):

def f(x):
    return x.copy()  # same index

def g(x):
    return x.copy().rename(lambda x: x + 1)  # different index

df = pd.DataFrame({"A": ['a', 'b', 'a'], "B": [1, 2, 3]})

In [6]: df.groupby("A").apply(f) 
Out[6]: 
   A  B
0  a  1
1  b  2
2  a  3

In [7]: df.groupby("A").apply(g) 
Out[7]: 
     A  B
A        
a 1  a  1
  3  a  3
b 2  b  2

Personally I agree with the retoric question "Do we want that kind of value-dependent behavior?". Such a drastic different output (also the row order differs, not just the index) based on a subtle difference in the function / output isn't that user friendly IMO.
Quoting myself from #13056 (comment): "I would personally deprecate this, and point users to groupby().transform(..)"

Now, as @WillAyd already said, tranform doesn't prepend the groupings as the index. But, it also doesn't include the grouping column at all (also not as a normal column, like apply does in some cases):

In [8]: df.groupby("A").transform(f) 
Out[8]: 
   B
0  1
1  2
2  3

Which makes it potentially a bit difficult to point users towards that, unless we add a keyword or something to ensure the grouping column is optionally included in the output.

Another related thought: for DataFrame.apply we have a result_type keyword, to specify what kind of function you are passing / how the result should be interpreted (as a reduction, vs preserve original shape, etc). Something similar might also be useful for the groupby version.

[TomAugspurger]

In my specific case .transform isn't an option because the function being applied needs to be applied to the table (it's doing things with multiple columns).

[TomAugspurger]

Rereading #34809 (comment), and I think that the suggestion to include a result_type-like keyword to control whether the function should be viewed as an aggregation / transform / other makes sense. I think that's the only possible solution for my use case: a distributed groupby.nunique() that's implemented through a .groupby(grouper).apply(lambda x: x.drop_duplicates()) on each chunk. We want consistent shapes for the result of each chunk. On master, we get value-dependent behavior depending on whether there are any duplicate values or not.

I'll try to put something together in the next couple days.

[quasiben]

Wanted to leave a general note about what cudf does. cudf has a limited groupby-apply support and generally it's not super performant for GPUs. Additionally cudf generally has tried to match pandas grouping and sorting behavior where possible but here we probably will find some challenges:

Also wanted to note that rename does not support index changes

In [154]: def f(x):
     ...:     return x.copy()  # same index
     ...:
     ...: def g(x):
     ...:     import cudf
     ...:     idx = range(50, 100)
     ...:     return x.copy().set_index(x.index + 1)  # different index
     ...:
     ...: cdf = cudf.DataFrame({"A": ['a', 'b', 'a'], "B": [1, 2, 3]})

In [155]: cdf.groupby("A").apply(f)
Out[155]:
   A  B
0  a  1
1  b  2
2  a  3

In [156]: cdf.groupby("A").apply(g)
Out[156]:
   A  B
1  a  1
2  b  2
3  a  3

[TomAugspurger]

Thanks. It looks like this is essentially inconsistent handling of the existing group_keys argument, which #34998 is trying to clean up. Hopefully this doesn't make things too much more complicated for cudf.