DEPR: support axis=None in DataFrame reductions #52042

jbrockmendel · 2023-03-17T03:17:19Z

closes Support axis=None in all reductions #21597 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Haven't yet added tests or docs, want to see if anyone cares about the warnings we get in the groupby tests when we do e.g. gb.apply(np.sum)

…-reductions

rhshadrach · 2023-03-19T22:41:43Z

Will users also see this message when doing np.sum(df)? Might make sense say DataFrame.sum or numpy.sum. Then the error message would also make sense in the groupby apply case.

jbrockmendel · 2023-03-20T19:22:29Z

Looks like this does produce the warning for np.sum(df), will update

jbrockmendel · 2023-03-24T16:04:30Z

Hmm looks like the warning also shows up for the built-in sum, but not min (while applying np.min already operates over axis=None)

topper-123

Generally LGTM. There are some tests failing, but I guess that's fixable

mroeschke · 2023-05-24T17:54:48Z


pandas/core/series.py:6050: error: Argument 1 of "prod" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6050: note: This violates the Liskov substitution principle
pandas/core/series.py:6050: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/series.py:6081: error: Argument 1 of "sem" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6081: note: This violates the Liskov substitution principle
pandas/core/series.py:6081: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/series.py:6092: error: Argument 1 of "var" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6092: note: This violates the Liskov substitution principle
pandas/core/series.py:6092: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/series.py:6103: error: Argument 1 of "std" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6103: note: This violates the Liskov substitution principle
pandas/core/series.py:6103: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11001: error: Argument 1 of "sum" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11001: note: This violates the Liskov substitution principle
pandas/core/frame.py:11001: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11013: error: Argument 1 of "prod" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11013: note: This violates the Liskov substitution principle
pandas/core/frame.py:11013: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11044: error: Argument 1 of "sem" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11044: note: This violates the Liskov substitution principle
pandas/core/frame.py:11044: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11055: error: Argument 1 of "var" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11055: note: This violates the Liskov substitution principle
pandas/core/frame.py:11055: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11066: error: Argument 1 of "std" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11066: note: This violates the Liskov substitution principle
pandas/core/frame.py:11066: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11096: error: Incompatible types in assignment (expression has type "Callable[[Optional[Union[int, Literal['index', 'columns', 'rows']]], bool, bool, int, KwArg(Any)], Any]", base class "NDFrame" defined the type as "Callable[[Union[int, None, Literal['index', 'columns', 'rows', _NoDefault.no_default]], bool, bool, int, KwArg(Any)], Any]")  [assignment]
Found 11 errors in 2 files (checked 1418 source files)

mypy (stubtest)............................................................Failed
- hook id: stubtest
- duration: 117.05s
- exit code: 1

error: not checking stubs due to mypy build errors:
pandas/core/series.py:6039: error: Argument 1 of "sum" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6039: note: This violates the Liskov substitution principle
pandas/core/series.py:6039: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/series.py:6050: error: Argument 1 of "prod" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6050: note: This violates the Liskov substitution principle
pandas/core/series.py:6050: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/series.py:6081: error: Argument 1 of "sem" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6081: note: This violates the Liskov substitution principle
pandas/core/series.py:6081: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/series.py:6092: error: Argument 1 of "var" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6092: note: This violates the Liskov substitution principle
pandas/core/series.py:6092: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/series.py:6103: error: Argument 1 of "std" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/series.py:6103: note: This violates the Liskov substitution principle
pandas/core/series.py:6103: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11001: error: Argument 1 of "sum" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11001: note: This violates the Liskov substitution principle
pandas/core/frame.py:11001: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11013: error: Argument 1 of "prod" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11013: note: This violates the Liskov substitution principle
pandas/core/frame.py:11013: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11044: error: Argument 1 of "sem" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11044: note: This violates the Liskov substitution principle
pandas/core/frame.py:11044: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11055: error: Argument 1 of "var" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11055: note: This violates the Liskov substitution principle
pandas/core/frame.py:11055: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11066: error: Argument 1 of "std" is incompatible with supertype "NDFrame"; supertype defines the argument type as "Union[Union[int, Literal['index', 'columns', 'rows']], None, Literal[_NoDefault.no_default]]"  [override]
pandas/core/frame.py:11066: note: This violates the Liskov substitution principle
pandas/core/frame.py:11066: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pandas/core/frame.py:11096: error: Incompatible types in assignment (expression has type "Callable[[Optional[Union[int, Literal['index', 'columns', 'rows']]], bool, bool, int, KwArg(Any)], Any]", base class "NDFrame" defined the type as "Callable[[Union[int, None, Literal['index', 'columns', 'rows', _NoDefault.no_default]], bool, bool, int, KwArg(Any)], Any]")  [assignment]

jbrockmendel · 2023-05-25T14:56:23Z

updated+green

mroeschke · 2023-05-25T16:13:32Z

Thanks @jbrockmendel

* DEPR: support axis=None in DataFrame reductions * test, whatsnew * catch in apply(sum) * Fix defaults * catch warnings * dont check stacklevel * mypy fixup * catch warning

jbrockmendel added 2 commits March 16, 2023 20:15

DEPR: support axis=None in DataFrame reductions

2f8666f

Merge branch 'main' into depr-reductions

5bfe28a

mroeschke added Deprecate Functionality to remove in pandas Reduction Operations sum, mean, min, max, etc. labels Mar 17, 2023

Merge branch 'main' of https://github.com/pandas-dev/pandas into depr…

395b91d

…-reductions

jbrockmendel added 5 commits March 20, 2023 12:23

Merge branch 'main' into depr-reductions

aa592d6

Merge branch 'main' into depr-reductions

71f70eb

test, whatsnew

5b8e078

Merge branch 'main' into depr-reductions

b07fc3b

Merge branch 'main' into depr-reductions

b1a1b1c

topper-123 reviewed Apr 9, 2023

View reviewed changes

jbrockmendel mentioned this pull request Apr 29, 2023

BUG: sum with axis=None does not apply the aggregation across both axes. #52977

Closed

3 tasks

jbrockmendel added 10 commits April 29, 2023 09:07

catch in apply(sum)

6a99ba5

Merge branch 'main' into depr-reductions

8d8703a

Merge branch 'main' into depr-reductions

09a46ac

Merge branch 'main' into depr-reductions

cd6879e

Merge branch 'main' into depr-reductions

c35b206

Fix defaults

b1d4ab7

Merge branch 'main' into depr-reductions

54f90e4

catch warnings

adafb33

Merge branch 'main' into depr-reductions

13bb07d

dont check stacklevel

3b2f589

jbrockmendel added 4 commits May 24, 2023 10:57

Merge branch 'main' into depr-reductions

fe2b690

mypy fixup

b335e65

Merge branch 'main' into depr-reductions

05e991b

catch warning

ecef601

mroeschke added this to the 2.1 milestone May 25, 2023

mroeschke approved these changes May 25, 2023

View reviewed changes

mroeschke merged commit d6ee9ad into pandas-dev:main May 25, 2023

jbrockmendel deleted the depr-reductions branch May 25, 2023 16:14

jbrockmendel mentioned this pull request May 29, 2023

DEPR: Special casing of NumPy and Python builtin functions #53425

Closed

rhshadrach mentioned this pull request Feb 29, 2024

CLN: Enforce deprecation of axis=None in DataFrame reductions #57684

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEPR: support axis=None in DataFrame reductions #52042

DEPR: support axis=None in DataFrame reductions #52042

jbrockmendel commented Mar 17, 2023

rhshadrach commented Mar 19, 2023

jbrockmendel commented Mar 20, 2023

jbrockmendel commented Mar 24, 2023

topper-123 left a comment

mroeschke commented May 24, 2023

jbrockmendel commented May 25, 2023

mroeschke commented May 25, 2023

DEPR: support axis=None in DataFrame reductions #52042

DEPR: support axis=None in DataFrame reductions #52042

Conversation

jbrockmendel commented Mar 17, 2023

rhshadrach commented Mar 19, 2023

jbrockmendel commented Mar 20, 2023

jbrockmendel commented Mar 24, 2023

topper-123 left a comment

Choose a reason for hiding this comment

mroeschke commented May 24, 2023

jbrockmendel commented May 25, 2023

mroeschke commented May 25, 2023