[BUG] Python groupby rolling aggregations return index inconsistent with pandas

[beckernick]

Python groupby rolling aggregations return a single Index that corresponds to the original row position of the element, but in pandas return a MultiIndex that includes both the groupby key(s) and original row position.

This is not currently blocking any behavior with Dask + cuDF, as grouped rolling operations are blocked by #10173

import pandas as pd
import cudf
import numpy as np
​
df = cudf.datasets.randomdata(nrows=100000)
pdf = df.to_pandas()
​
print(pdf.groupby(['id']).rolling(window=3).x.mean().head())
print(df.groupby(['id']).rolling(window=3).x.mean().head())
id        
879  43605   NaN
881  3941    NaN
882  29855   NaN
884  14616   NaN
     70864   NaN
Name: x, dtype: float64
43605    <NA>
3941     <NA>
29855    <NA>
14616    <NA>
70864    <NA>
Name: x, dtype: float64

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[beckernick]

Took a quick look at this issue again. I suspect that the reason this occurs is due to the __getitem__ call (.x). RollingGroupby inherits from Rolling, which has the getitem implementation.

This implementation returns a standard Rolling object:

cudf/python/cudf/cudf/core/window/rolling.py

Lines 204 to 211 in 3bacb12

	def __getitem__(self, arg):
	if isinstance(arg, tuple):
	arg = list(arg)
	return self.obj[arg].rolling(
	window=self.window,
	min_periods=self.min_periods,
	center=self.center,
	)

So the call to mean() goes down the standard Rolling._apply_agg path rather than the RollingGroupby._apply_agg. This path doesn't create a MultiIndex:

cudf/python/cudf/cudf/core/window/rolling.py

Lines 264 to 284 in 3bacb12

	def _apply_agg_dataframe(self, df, agg_name):
	return cudf.DataFrame._from_data(
	{
	col_name: self._apply_agg_column(col, agg_name)
	for col_name, col in df._data.items()
	},
	index=df.index,
	)
	def _apply_agg(self, agg_name):
	if isinstance(self.obj, cudf.Series):
	return cudf.Series._from_data(
	{
	self.obj.name: self._apply_agg_column(
	self.obj._column, agg_name
	)
	},
	index=self.obj.index,
	)
	else:
	return self._apply_agg_dataframe(self.obj, agg_name)

Unlike the RollingGroupby._apply_agg path:

cudf/python/cudf/cudf/core/window/rolling.py

Lines 545 to 559 in 3bacb12

	def _apply_agg(self, agg_name):
	if agg_name == "count" and not self._time_window:
	self.min_periods = 0
	index = cudf.MultiIndex.from_frame(
	cudf.DataFrame(
	{
	key: value
	for key, value in itertools.chain(
	self._group_keys._data.items(),
	self.obj.index._data.items(),
	)
	}
	)
	)

[beckernick]

To make this concrete:

import cudf

df = cudf.datasets.randomdata()

print("getitem call on the RollingGroupby object")
print(df.groupby(["id"]).rolling(window=3).x.mean().head())

print("getitem call before the RollingGroupby is created")
print(df.groupby(["id"]).x.rolling(window=3).mean().head())
getitem call on the RollingGroupby object
3    <NA>
8    <NA>
6    <NA>
7    <NA>
9    <NA>
Name: x, dtype: float64
getitem call before the RollingGroupby is created
id    
918  3    <NA>
963  8    <NA>
967  6    <NA>
990  7    <NA>
     9    <NA>
Name: x, dtype: float64

[beckernick]

Ran into this today while using cudf.pandas.

cc @galipremsagar @vyasr , would this be a good first issue for a new contributor?

[vyasr]

Yes, I think so!