Skip to content

DEPR: __invert__ on object dtype (GH#51567) - #67372

Open
jbrockmendel wants to merge 2 commits into
pandas-dev:mainfrom
jbrockmendel:bug-51567
Open

DEPR: __invert__ on object dtype (GH#51567)#67372
jbrockmendel wants to merge 2 commits into
pandas-dev:mainfrom
jbrockmendel:bug-51567

Conversation

@jbrockmendel

@jbrockmendel jbrockmendel commented Aug 30, 2026

Copy link
Copy Markdown
Member

closes #51567
closes #16873
closes #31035

~ on object dtype dispatches elementwise to the objects' own __invert__, so a boolean mask that has been cast to object silently becomes integers instead of being negated:

>>> ~pd.Series([True, False], dtype=object)
0   -2
1   -1
dtype: object

This is the cause of GH-16873 (~ser.shift(1).dropna() on a bool Series returns [-1, -1, -2], since shift returns object) and GH-31035 ((~s).astype(bool) on an object bool Series returns all True). It is also why ser.where(cond) accepts an object-dtype boolean cond but ser.mask(cond) raises TypeError: Boolean array expected for the condition, not int64mask computes ~cond.

GH-31035 asks for the opposite resolution — that ~ on an object-dtype boolean Series return correct booleans. This deprecates the operation instead; the replacement is ~obj.astype(bool).

Object dtype is the only thing left to deprecate here. ~ currently succeeds on exactly three families — bool-like (bool, boolean, bool[pyarrow], Sparse[bool]), integer-like (numpy, masked, arrow, Sparse[int64]), and object. datetime64, timedelta64, period, interval, category, float/complex, string, decimal and every non-integer arrow type already raise, so "deprecate everything besides int and bool" and "deprecate object" are the same change.

Object dtype is not even self-consistent today, which is why the bug hides so well — np.bool_ elements give the correct logical NOT while Python bool elements give integers, and the two are indistinguishable to the user:

>>> ~pd.Series(np.array([True, False], dtype=object))
0   -2
1   -1
dtype: object
>>> ~pd.Series(np.array([np.True_, np.False_], dtype=object))
0    False
1     True
dtype: object

Worth noting that CPython reached the same conclusion independently: Python 3.12+ emits DeprecationWarning: Bitwise inversion '~' on bool is deprecated and will be removed in Python 3.16, so the boolean-mask case will raise on its own in 3.16 regardless.

The warning is emitted from four entry points (NDFrame.__invert__, Index.__invert__, NumpyExtensionArray.__invert__, SparseArray.__invert__). SparseArray matters because Sparse[object] with an object fill_value succeeds and reproduces the bug; with the default nan fill_value it happens to raise.

Integer dtypes are deliberately left alone — that behavior matches Python and numpy, is tested through pd.eval, and was documented with worked examples in GH-65291.

AI disclosure: drafted with Claude Code (claude opus 5 (high)), which surveyed the current per-dtype behavior, instrumented the test suite to find every affected test in advance, and wrote the implementation and tests under my review.

~ on object dtype dispatches elementwise to the objects' own __invert__,
so a boolean mask that has been cast to object silently becomes integers
instead of being negated.

Object is the only dtype left to deprecate: ~ currently succeeds only on
bool-like, integer-like, and object dtypes. datetime64, timedelta64,
period, interval, category, float, complex, string, decimal and every
non-integer arrow type already raise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jbrockmendel jbrockmendel added Numeric Operations Arithmetic, Comparison, and Logical operations Deprecate Functionality to remove in pandas labels Aug 30, 2026
tests/arithmetic holds binary-operator suites; move the deprecation tests
to the per-container unary homes instead.  Adds coverage for the
SparseArray[object] call site, which tests/extension/test_sparse.py never
reached (it only parametrizes float dtypes).

Also reference GH#16873 and GH#31035 in the whatsnew note; both are
resolved by this deprecation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jbrockmendel
jbrockmendel marked this pull request as ready for review August 30, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Deprecate Functionality to remove in pandas Numeric Operations Arithmetic, Comparison, and Logical operations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DEPR: __invert__ on non-bool dtypes Incorrect boolean bitwise conversion when dtype=object ERR: Remove __invert__ operations on dtype=object?

1 participant