BUG: Agg and Apply with custom function improperly ignore column labels when function returns a dict-like

[aevangeline]

Hello folks! I've discovered an odd corner case regression that I hope can be fixed.

The problem is that if you are running an aggregation/apply on a dataframe using a custom function - if that function returns a dict-like object, it doesn't use the appropriate column labels in the original dataframe.

As a simple replication

df = pd.DataFrame([[1, 2, 3],
                   [4, 5, 6],
                   [7, 8, 9],
                   [np.nan, np.nan, np.nan]],
                  columns=['A', 'B', 'C'])

print(df.agg(lambda x : {"sum" :np.sum(x)}))

prints

0    {'sum': 12.0}
1    {'sum': 15.0}
2    {'sum': 18.0}

instead of the expected

A    {'sum': 12.0}
B    {'sum': 15.0}
C    {'sum': 18.0}

This is a regression/breaking change since in prior versions of pandas (I believe 1.0.X and earlier) the expected results were produced.

[arw2019]

Driveby comment: it's always better to use .sum instead of an aggregation performance-wise

[aevangeline]

Absolutely! This is a toy example I made just for the purposes of demonstration - This bug cropped up when I was trying to build up a dictionary of qualitative tags that appeared in a column mapped to the number of times they appeared. The data for that is unfortunately locked behind an IRB and some confidentiality agreements, hence the very silly example.

[ma3da]

Hi,
from what I understand, when passing to DataFrame.agg a func that returns a dict, the result is in fine provided by Frame[Row/Column]Apply.wrap_results_for_axis. In the 'Row' case, the index of the returned Series is not set, which looks like a simple omission.
In the 'Column' case, we get the expected behaviour. Reusing the example given by @aevangeline,

df.index = ("i0", "i1", "i2", "i3")
print(df.agg(lambda x : {"sum" :np.sum(x)}, axis=1))

outputs

i0     {'sum': 6.0}
i1    {'sum': 15.0}
i2    {'sum': 24.0}
i3     {'sum': 0.0}
dtype: object