Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Case of generic decorators not working on generic callables (ParamSpec) #17621

Open
kcdodd opened this issue Aug 1, 2024 · 2 comments
Open
Labels
bug mypy got something wrong topic-paramspec PEP 612, ParamSpec, Concatenate

Comments

@kcdodd
Copy link

kcdodd commented Aug 1, 2024

This describes an issue where a generic decorator that returns a generic sub-type of a "callable" (using __call__ and ParamSpec) cannot be applied to a generic function. In the example below, Callable[_P, _R] -> Traceable[_P, _R] works, but Callable[_P, _R] -> Decorated[_P, _R] does not. It seems to work if either the decorated function is not generic (E.G. radius() instead of apply() in the example), or if the return type of the decorator is a super-type of its argument (E.G. Traceable instead of Decorated).

Relevant closed issues

To Reproduce

from collections.abc import Callable
from typing_extensions import (
  TypeVar,
  ParamSpec,
  Generic,
  Protocol,
  reveal_type)

_P = ParamSpec('_P')
_R = TypeVar('_R', covariant=True)

_P2 = ParamSpec('_P2')
_R2 = TypeVar('_R2', covariant=True)

class Traceable(Protocol[_P, _R]):
  def __call__(self, *args: _P.args, **kwargs: _P.kwargs) -> _R: ...

class Decorated(Traceable[_P, _R]):
  target: Traceable[_P, _R]

  def __init__(self, target: Traceable[_P, _R]):
    self.target = target

  def __call__(self, *args: _P.args, **kwargs: _P.kwargs) -> _R:
    return self.target(*args, **kwargs)

def decorator1(target: Callable[_P, _R]) -> Traceable[_P, _R]:
  return Decorated(target)

def decorator2(target: Traceable[_P, _R]) -> Decorated[_P, _R]:
  return Decorated(target)

def apply(
    func: Callable[_P2, _R2],
    *args: _P2.args,
    **kwargs: _P2.kwargs) -> _R2:

  return func(*args, **kwargs)

@decorator1
def apply_decorated1(
    func: Callable[_P2, _R2],
    *args: _P2.args,
    **kwargs: _P2.kwargs) -> _R2:

  return func(*args, **kwargs)

@decorator2 # error: Argument 1 to "decorator2" has incompatible type "Callable[[Callable[_P2, _R2], **_P2], _R2]"; expected "Traceable[Never, Never]"  [arg-type]
def apply_decorated2(
    func: Callable[_P2, _R2],
    *args: _P2.args,
    **kwargs: _P2.kwargs) -> _R2:

  return func(*args, **kwargs)

@decorator2
def radius(x: float, y: float) -> float:
  return (x**2 + y**2)**0.5
reveal_type(apply) # Revealed type is "def [_P2, _R2] (func: def (*_P2.args, **_P2.kwargs) -> _R2`-2, *_P2.args, **_P2.kwargs) -> _R2`-2"
reveal_type(apply_decorated1) # Revealed type is "def [_P2, _R] (func: def (*_P2.args, **_P2.kwargs) -> _R`6, *_P2.args, **_P2.kwargs) -> _R`6"
reveal_type(apply_decorated2) # Revealed type is "typehint_decorator.Decorated[Never, Never]"
reveal_type(radius) # Revealed type is "typehint_decorator.Decorated[[x: builtins.float, y: builtins.float], builtins.float]"

r00 = radius(1.0, 0.0)
reveal_type(r00) # Revealed type is "builtins.float"

r01 = apply(radius, 1.0, 0.0)
reveal_type(r01) # Revealed type is "builtins.float"

r1 = apply_decorated1(radius, 1.0, 0.0)
reveal_type(r1) # Revealed type is "builtins.float"

r2 = apply_decorated2(radius, 1.0, 0.0) # error: Argument 1 to "__call__" of "Decorated" has incompatible type "Decorated[[float, float], float]"; expected "Never"  [arg-type]
reveal_type(r2) # Revealed type is "Any"

f2: Decorated[[float, float], float] = radius
f1: Traceable[[float, float], float] = f2
f0: Callable[[float, float], float] = f1

g1: Traceable[[Callable[[float, float], float], float, float], float] = apply_decorated1
g0: Callable[[Callable[[float, float], float], float, float], float] = g1

h2: Decorated[[Callable[[float, float], float], float, float], float] = apply_decorated2
h1: Traceable[[Callable[[float, float], float], float, float], float] = h2
h0: Callable[[Callable[[float, float], float], float, float], float] = h1

reveal_type(h2) # Revealed type is "typehint_decorator.Decorated[[def (builtins.float, builtins.float) -> builtins.float, builtins.float, builtins.float], builtins.float]"

https://mypy-play.net/?mypy=latest&python=3.12&gist=a8f681e6c14ec013bf3ae56c81fe94b2

Expected Behavior

Expected variables transferred from input generic callable to returned generic callable, even if the return is not a super-type of the input.

Actual Behavior

ParamSpec variables are not used to parameterize the returned generic if it is not a super-type of the input.

LOG:  Mypy Version:           1.12.0+dev.a0dbbd5b462136914bb7a378221ae094b6844710
LOG:  Config File:            Default
...
typehint_decorator.py:48: error: Argument 1 to "decorator2" has incompatible type "Callable[[Callable[_P2, _R2], **_P2], _R2]"; expected "Traceable[Never, Never]"  [arg-type]
typehint_decorator.py:60: note: Revealed type is "def [_P2, _R2] (func: def (*_P2.args, **_P2.kwargs) -> _R2`-2, *_P2.args, **_P2.kwargs) -> _R2`-2"
typehint_decorator.py:61: note: Revealed type is "def [_P2, _R] (func: def (*_P2.args, **_P2.kwargs) -> _R`6, *_P2.args, **_P2.kwargs) -> _R`6"
typehint_decorator.py:62: note: Revealed type is "typehint_decorator.Decorated[Never, Never]"
typehint_decorator.py:63: note: Revealed type is "typehint_decorator.Decorated[[x: builtins.float, y: builtins.float], builtins.float]"
typehint_decorator.py:66: note: Revealed type is "builtins.float"
typehint_decorator.py:69: note: Revealed type is "builtins.float"
typehint_decorator.py:72: note: Revealed type is "builtins.float"
typehint_decorator.py:74: error: Need type annotation for "r2"  [var-annotated]
typehint_decorator.py:74: error: Argument 1 to "__call__" of "Decorated" has incompatible type "Decorated[[float, float], float]"; expected "Never"  [arg-type]
typehint_decorator.py:74: error: Argument 2 to "__call__" of "Decorated" has incompatible type "float"; expected "Never"  [arg-type]
typehint_decorator.py:74: error: Argument 3 to "__call__" of "Decorated" has incompatible type "float"; expected "Never"  [arg-type]
typehint_decorator.py:75: note: Revealed type is "Any"
typehint_decorator.py:88: note: Revealed type is "typehint_decorator.Decorated[[def (builtins.float, builtins.float) -> builtins.float, builtins.float, builtins.float], builtins.float]"
LOG:  Deleting typehint_decorator typehint_decorator.py typehint_decorator.meta.json typehint_decorator.data.json
LOG:  No fresh SCCs left in queue
LOG:  Build finished in 0.419 seconds with 50 modules, and 14 errors
Found 5 errors in 1 file (checked 1 source file)

Your Environment

  • Mypy version used: 1.12.0+dev*
  • Mypy command-line flags: python -m mypy -v typehint_decorator.py
  • Python version used: 3.12.4
@kcdodd kcdodd added the bug mypy got something wrong label Aug 1, 2024
@Shaun-Ho
Copy link

Shaun-Ho commented Aug 2, 2024

I think I have a similar problem when trying to create class-based decorators with alternate constructors:

from __future__ import annotations

import collections.abc
import typing

P = typing.ParamSpec("P")
T_co = typing.TypeVar("T_co", covariant=True)


class MyDecorator(typing.Generic[P, T_co]):
    def __init__(
        self,
        func: collections.abc.Callable[P, T_co],
        *,
        option: str | None = None,
    ) -> None:
        self._attribute = option

    @classmethod
    def construct_with_configuration( # Argument 1 has incompatible type "Callable[[int], int]"; expected "Callable[[VarArg(Never), KwArg(Never)], Never]"  [arg-type]
        cls,
        option: str,
    ) -> collections.abc.Callable[[collections.abc.Callable[P, T_co]], MyDecorator[P, T_co]]:
        def decorator(func: collections.abc.Callable[P, T_co]) -> MyDecorator[P, T_co]:
            return cls(func, option=option)
        return decorator
    
@MyDecorator.construct_with_configuration(
    option="a",
)
def a_function(a: int) -> int:
    return a + 1

typing.reveal_type(a_function) # Revealed type is "MyDecorator[Never, Never]"

@ilevkivskyi ilevkivskyi added the topic-paramspec PEP 612, ParamSpec, Concatenate label Aug 4, 2024
@sebastian-goeldi
Copy link

sebastian-goeldi commented Aug 19, 2024

I encountered a very similar problem when trying to overload a decorator to support it with kwargs or without. I think it all boils down to mypy resolving T in a return type Callable[[Callable[..., T]], Callable[..., T]] to Never unless there are other indicators for how to resolve T in any args/kwargs. It would be amazing if mypy could support it.

Example for completness
from __future__ import annotations

from collections.abc import Callable
from typing import ParamSpec, Protocol, TypeVar, overload

P = ParamSpec("P")
AB = TypeVar("AB", bound="A", covariant=True)
T = TypeVar("T")


class A:
    def __init__(self) -> None:
        return


class B(A):
    pass


class ABFunc(Protocol[P, AB]):
    __name__: str

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> AB: ...


@overload
def decorator(_func: ABFunc[P, AB]) -> ABFunc[P, AB]: ...


@overload
def decorator(
    *, flag_a: bool = False, flag_b: tuple[AB, ...] = tuple()
) -> Callable[[ABFunc[P, AB]], ABFunc[P, AB]]: ...


def decorator(
    _func: ABFunc[P, AB] | None = None,
    *,
    flag_a: bool = False,
    flag_b: tuple[AB, ...] = tuple(),
) -> ABFunc[P, AB] | Callable[[ABFunc[P, AB]], ABFunc[P, AB]]:
    def decorator(f: ABFunc[P, AB]) -> ABFunc[P, AB]:
        print(flag_a)
        print(flag_b)
        return f

    return decorator if _func is None else decorator(_func)


@decorator(flag_a=True)  # works if e.g. 'flag_b=(B(),)' is added
def myfunc() -> B:
    return B()


# reveal_type(myfunc)  # 'ABFunc[[], B]' if flag_b is set otherwise 'ABFunc[Never, Never]'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug mypy got something wrong topic-paramspec PEP 612, ParamSpec, Concatenate
Projects
None yet
Development

No branches or pull requests

4 participants