-
Notifications
You must be signed in to change notification settings - Fork 651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FIX-#3571: fallback to 'sort=True' on categorical keys in groupby #3715
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2894,7 +2894,7 @@ def value_counts( | |
): | ||
if subset is None: | ||
subset = self._query_compiler.columns | ||
counted_values = self.groupby(by=subset, sort=False, dropna=dropna).size() | ||
counted_values = self.groupby(by=subset, dropna=dropna, observed=True).size() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Example>>> df = pandas.DataFrame({"a": [1, 1, 2, 2], "b": [2, 2, 3, 3]}, dtype="category")
>>> df
a b
0 1 2
1 1 2
2 2 3
3 2 3
>>> df.value_counts()
a b
1 2 2
2 3 2
dtype: int64
>>> df.groupby(["a", "b"]).size()
a b
1 2 2
3 0
2 2 0
3 2
dtype: int64
>>> df.groupby(["a", "b"], observed=True).size()
a b
1 2 2
2 3 2
dtype: int64 |
||
if sort: | ||
counted_values.sort_values(ascending=ascending, inplace=True) | ||
if normalize: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1731,6 +1731,32 @@ def test_not_str_by(by, as_index): | |
eval_general(md_grp, pd_grp, lambda grp: grp.first()) | ||
|
||
|
||
@pytest.mark.parametrize("sort", [True, False]) | ||
@pytest.mark.parametrize("is_categorical_by", [True, False]) | ||
def test_groupby_sort(sort, is_categorical_by): | ||
# from issue #3571 | ||
by = np.array(["a"] * 50000 + ["b"] * 10000 + ["c"] * 1000) | ||
random_state = np.random.RandomState(seed=42) | ||
random_state.shuffle(by) | ||
Comment on lines
+1738
to
+1740
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the bug is partitioning dependent, that's why I'm carrying the same data all around There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To clarify - you mean the shuffle is in place so that the bug gets tripped (bc if we didn't shuffle and the values weren't replicated across the partitions, we wouldn't hit the bug?) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Right. For this data shuffled with this specific seed we know for sure that there is a problem caused by the incorrect order of categories in the reduce phase of groupby (see the issue explanation). This problem might not be visible when the values are in a different order, so I'm using exact data from the #3571 report as a solid reproducer of the issue. |
||
|
||
data = {"key_col": by, "data_col": np.arange(len(by))} | ||
md_df, pd_df = create_test_dfs(data) | ||
|
||
if is_categorical_by: | ||
md_df = md_df.astype({"key_col": "category"}) | ||
pd_df = pd_df.astype({"key_col": "category"}) | ||
|
||
md_grp = md_df.groupby("key_col", sort=sort) | ||
pd_grp = pd_df.groupby("key_col", sort=sort) | ||
|
||
modin_groupby_equals_pandas(md_grp, pd_grp) | ||
eval_general(md_grp, pd_grp, lambda grp: grp.sum()) | ||
eval_general(md_grp, pd_grp, lambda grp: grp.size()) | ||
eval_general(md_grp, pd_grp, lambda grp: grp.agg(lambda df: df.mean())) | ||
eval_general(md_grp, pd_grp, lambda grp: grp.dtypes) | ||
eval_general(md_grp, pd_grp, lambda grp: grp.first()) | ||
|
||
|
||
def test_sum_with_level(): | ||
data = { | ||
"A": ["0.0", "1.0", "2.0", "3.0", "4.0"], | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
modin_frame._dtypes
attribute isNone
most of the time, its computation requires calling MapReduce and actual data materialization. I don't think we want to trigger this on everygroupby(sort=False)