Fix binding difference between callables and callback protocols

[hauntsaninja]

Fixes #15926 (comment), fixes #16200

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/arrays/_ranges.py:94: error: Too many arguments for "__call__" of "_SignedIntOp"  [call-arg]
+ pandas/core/arrays/_ranges.py:94: error: Too many arguments for "__call__" of "_UnsignedIntOp"  [call-arg]
+ pandas/core/arrays/_ranges.py:94: note: Left operand is of type "int | signedinteger[_64Bit] | unsignedinteger[_64Bit]"
+ pandas/core/arrays/_ranges.py:136: error: Too many arguments for "__call__" of "_UnsignedIntOp"  [call-arg]
+ pandas/core/arrays/_ranges.py:150: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/arrays/_ranges.py:150: note: Possible overload variants:
+ pandas/core/arrays/_ranges.py:150: note:     def __call__(self) -> bool_
+ pandas/core/arrays/_ranges.py:150: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/arrays/_ranges.py:150: note:     def __call__(self) -> Any
+ pandas/core/arrays/_ranges.py:150: error: Unsupported operand types for <= ("int" and "unsignedinteger[_64Bit]")  [operator]
+ pandas/core/arrays/_ranges.py:178: error: Too many arguments for "__call__" of "_SignedIntOp"  [call-arg]
+ pandas/core/arrays/_ranges.py:181: error: Too many arguments for "__call__" of "_SignedIntOp"  [call-arg]
+ pandas/core/arrays/_ranges.py:201: error: Too many arguments for "__call__" of "_UnsignedIntOp"  [call-arg]
+ pandas/core/arrays/_ranges.py:203: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "unsignedinteger[_64Bit]"  [operator]
+ pandas/core/arrays/_ranges.py:203: note: Possible overload variants:
+ pandas/core/arrays/_ranges.py:203: note:     def __call__(self) -> bool_
+ pandas/core/arrays/_ranges.py:203: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/arrays/_ranges.py:203: note:     def __call__(self) -> Any
+ pandas/core/arrays/_ranges.py:204: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "unsignedinteger[_64Bit]"  [operator]
+ pandas/core/arrays/_ranges.py:204: note: Possible overload variants:
+ pandas/core/arrays/_ranges.py:204: note:     def __call__(self) -> bool_
+ pandas/core/arrays/_ranges.py:204: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/arrays/_ranges.py:204: note:     def __call__(self) -> Any
+ pandas/core/arrays/_ranges.py:204: error: Too many arguments for "__call__" of "_UnsignedIntOp"  [call-arg]
+ pandas/util/_validators.py:443: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/util/_validators.py:443: note: Possible overload variants:
+ pandas/util/_validators.py:443: note:     def __call__(self) -> bool_
+ pandas/util/_validators.py:443: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/util/_validators.py:443: note:     def __call__(self) -> Any
+ pandas/util/_validators.py:443: note: Left operand is of type "int | integer[Any]"
+ pandas/util/_validators.py:444: error: Too many arguments for "__call__" of "_NumberOp"  [call-arg]
+ pandas/util/_validators.py:444: note: Left operand is of type "int | integer[Any]"
+ pandas/core/util/hashing.py:76: error: Too many arguments for "__call__" of "_UnsignedIntOp"  [call-arg]
+ pandas/core/nanops.py:731: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/nanops.py:731: note: Possible overload variants:
+ pandas/core/nanops.py:731: note:     def __call__(self) -> bool_
+ pandas/core/nanops.py:731: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/nanops.py:731: note:     def __call__(self) -> Any
+ pandas/core/nanops.py:731: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1264: error: Too many arguments for "__call__" of "_FloatOp"  [call-arg]
+ pandas/core/nanops.py:1264: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1272: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/nanops.py:1272: note: Possible overload variants:
+ pandas/core/nanops.py:1272: note:     def __call__(self) -> bool_
+ pandas/core/nanops.py:1272: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/nanops.py:1272: note:     def __call__(self) -> Any
+ pandas/core/nanops.py:1272: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1275: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/nanops.py:1275: note: Possible overload variants:
+ pandas/core/nanops.py:1275: note:     def __call__(self) -> bool_
+ pandas/core/nanops.py:1275: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/nanops.py:1275: note:     def __call__(self) -> Any
+ pandas/core/nanops.py:1275: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1345: error: Too many arguments for "__call__" of "_FloatOp"  [call-arg]
+ pandas/core/nanops.py:1345: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1346: error: Too many arguments for "__call__" of "_FloatOp"  [call-arg]
+ pandas/core/nanops.py:1346: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1347: error: Too many arguments for "__call__" of "_FloatOp"  [call-arg]
+ pandas/core/nanops.py:1347: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1359: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/nanops.py:1359: note: Possible overload variants:
+ pandas/core/nanops.py:1359: note:     def __call__(self) -> bool_
+ pandas/core/nanops.py:1359: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/nanops.py:1359: note:     def __call__(self) -> Any
+ pandas/core/nanops.py:1359: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1373: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/nanops.py:1373: note: Possible overload variants:
+ pandas/core/nanops.py:1373: note:     def __call__(self) -> bool_
+ pandas/core/nanops.py:1373: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/nanops.py:1373: note:     def __call__(self) -> Any
+ pandas/core/nanops.py:1373: note: Left operand is of type "floating[Any] | ndarray[Any, dtype[floating[Any]]]"
+ pandas/core/nanops.py:1565: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/nanops.py:1565: note: Possible overload variants:
+ pandas/core/nanops.py:1565: note:     def __call__(self) -> bool_
+ pandas/core/nanops.py:1565: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/nanops.py:1565: note:     def __call__(self) -> Any
+ pandas/core/dtypes/cast.py:1755: error: Unsupported operand types for <= ("int" and "integer[Any]")  [operator]
+ pandas/core/dtypes/cast.py:1755: note: Right operand is of type "int | integer[Any] | float"
+ pandas/core/dtypes/cast.py:1755: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/dtypes/cast.py:1755: note: Possible overload variants:
+ pandas/core/dtypes/cast.py:1755: note:     def __call__(self) -> bool_
+ pandas/core/dtypes/cast.py:1755: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/dtypes/cast.py:1755: note:     def __call__(self) -> Any
+ pandas/core/dtypes/cast.py:1755: note: Left operand is of type "int | integer[Any] | float"
+ pandas/core/missing.py:215: error: Unsupported operand types for - ("int" and "signedinteger[Any]")  [operator]
+ pandas/core/arrays/datetimelike.py:1350: error: Too many arguments for "__call__" of "_NumberOp"  [call-arg]
+ pandas/core/arrays/datetimelike.py:1350: note: Left operand is of type "int | integer[Any]"
+ pandas/core/arrays/datetimelike.py:1408: error: Too many arguments for "__call__" of "_NumberOp"  [call-arg]
+ pandas/core/arrays/datetimelike.py:1408: note: Left operand is of type "int | integer[Any]"
+ pandas/io/parsers/readers.py:555: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/io/parsers/readers.py:555: note: Possible overload variants:
+ pandas/io/parsers/readers.py:555: note:     def __call__(self) -> bool_
+ pandas/io/parsers/readers.py:555: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/io/parsers/readers.py:555: note:     def __call__(self) -> Any
+ pandas/io/parsers/readers.py:555: note: Left operand is of type "int | integer[Any]"
+ pandas/io/parsers/python_parser.py:1378: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/io/parsers/python_parser.py:1378: note: Possible overload variants:
+ pandas/io/parsers/python_parser.py:1378: note:     def __call__(self) -> bool_
+ pandas/io/parsers/python_parser.py:1378: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/io/parsers/python_parser.py:1378: note:     def __call__(self) -> Any
+ pandas/io/parsers/python_parser.py:1378: note: Left operand is of type "int | integer[Any]"
+ pandas/core/indexing.py:1607: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ pandas/core/indexing.py:1607: note: Possible overload variants:
+ pandas/core/indexing.py:1607: note:     def __call__(self) -> bool_
+ pandas/core/indexing.py:1607: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ pandas/core/indexing.py:1607: note:     def __call__(self) -> Any

... (truncated 108 lines) ...

arviz (https://github.com/arviz-devs/arviz)
+ arviz/tests/base_tests/test_stats.py:381: error: Too many arguments for "__call__" of "_FloatOp"  [call-arg]

spark (https://github.com/apache/spark)
+ python/pyspark/ml/linalg/__init__.py:634: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ python/pyspark/ml/linalg/__init__.py:634: note: Possible overload variants:
+ python/pyspark/ml/linalg/__init__.py:634: note:     def __call__(self) -> bool_
+ python/pyspark/ml/linalg/__init__.py:634: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ python/pyspark/ml/linalg/__init__.py:634: note:     def __call__(self) -> Any
+ python/pyspark/ml/linalg/__init__.py:639: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ python/pyspark/ml/linalg/__init__.py:639: note: Possible overload variants:
+ python/pyspark/ml/linalg/__init__.py:639: note:     def __call__(self) -> bool_
+ python/pyspark/ml/linalg/__init__.py:639: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ python/pyspark/ml/linalg/__init__.py:639: note:     def __call__(self) -> Any
+ python/pyspark/pandas/indexing.py:1658: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ python/pyspark/pandas/indexing.py:1658: note: Possible overload variants:
+ python/pyspark/pandas/indexing.py:1658: note:     def __call__(self) -> bool_
+ python/pyspark/pandas/indexing.py:1658: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ python/pyspark/pandas/indexing.py:1658: note:     def __call__(self) -> Any
+ python/pyspark/pandas/indexing.py:1658: note: Left operand is of type "int | signedinteger[_64Bit] | signedinteger[_32Bit]"
+ python/pyspark/pandas/indexing.py:1671: error: No overload variant of "__call__" of "_ComparisonOp" matches argument type "int"  [operator]
+ python/pyspark/pandas/indexing.py:1671: note: Possible overload variants:
+ python/pyspark/pandas/indexing.py:1671: note:     def __call__(self) -> bool_
+ python/pyspark/pandas/indexing.py:1671: note:     def __call__(self) -> ndarray[Any, dtype[bool_]]
+ python/pyspark/pandas/indexing.py:1671: note:     def __call__(self) -> Any
+ python/pyspark/pandas/indexing.py:1671: note: Left operand is of type "int | signedinteger[_64Bit] | signedinteger[_32Bit]"
+ python/pyspark/pandas/base.py:264: error: Too many arguments for "__call__" of "_TD64Div"  [call-arg]

bokeh (https://github.com/bokeh/bokeh)
+ src/bokeh/util/serialization.py: note: In function "convert_timedelta_type":
+ src/bokeh/util/serialization.py:163:22: error: Too many arguments for "__call__" of "_TD64Div"  [call-arg]
+ src/bokeh/util/serialization.py:210:22: error: Too many arguments for "__call__" of "_TD64Div"  [call-arg]

materialize (https://github.com/MaterializeInc/materialize)
+ misc/python/materialize/feature_benchmark/filter.py:31: error: Unsupported operand types for * ("int" and "floating[Any]")  [operator]

[ilevkivskyi]

Hm, it looks like all new errors are technically correct. NumPy stubs take into account current (inconsistent) mypy's behavior. I am not sure what is the best way forward: even if we fix the NumPy stubs, everyone who doesn't pin mypy will get errors until they update NumPy stubs, and vice versa.

[RonnyPfannschmidt]

would it be feasible to have a per module configuration option that helps to annotate whether or not to do the check,

that way one could opt-in to the new way and the numpy specs could gradually fix their types

ideally this would warn in a way that helps people figure what flags to set at either definition or usage site

[RonnyPfannschmidt]

im working around this using a Concatenate and mypy_extensions.Arg for now

[RonnyPfannschmidt]

i thik i now understand the underlying issue

its structurally incorrect to do the first parameters strop off

instead its necessary to be aware of the method descriptor protocol and applying it

so any callable protocol would need a __get__ that returns the correct method version for the callable

and for callables its also a bug to pretend they bind, in particular when __call__ using objects would be valid pass ins that do in fact not run any __get__

i'd like if there was a typing helper for this so one could spell Binding/nonBinding locations for those

[RonnyPfannschmidt]

i learned that the workaround with mypy_extensions no longer works