How to type the self argument in a Protocol for a decorator?

[DanielNoord]

Originally asked this on Discord, but nobody was able to help there. How should I type the following:

import functools
from typing import Protocol


class Method[S: SomeClass, R, **P](Protocol):
    def __call__(_, self: S, *args: P.args, **kwargs: P.kwargs) -> R: ...


def decorator[S: SomeClass, R, **P](f: Method[S, R, P]) -> Method[S, R, P]:
    @functools.wraps(f)
    def wrapper(self: S, *args: P.args, **kwargs: P.kwargs) -> R:
        return f(self, *args, **kwargs)

    return wrapper


class SomeClass:
    @decorator
    def a_method(self) -> int: ...


instance = SomeClass()
instance.a_method()

I want to indicate that the decorator only works on methods of SomeClass or its subclasses, but pyright complains that self is missing from the call to a_method with this code.

[erictraut]

This is a known limitation of ParamSpec when it's used with a protocol class. Type checkers retain internal information about methods such as whether they are instance methods, class methods, or static methods. This information is effectively lost when passing the callable through a callback protocol. There are some times where it might make sense to retain this information, but there are other times where this would be problematic. The typing spec doesn't provide any guidance about when to retain it and when not.

Pyright will retain this internal information when using a Callable rather than a callback protocol, so my recommendation in this case is to avoid using a callback protocol and replace it with a simple Callable.

def decorator[S: SomeClass, R, **P](f: Callable[Concatenate[S, P], R]) -> Callable[Concatenate[S, P], R]: ...

This retains the information that the decorated method is an instance method and should be bound when accessed through an instance of SomeClass.

[DanielNoord]

Thanks, there was one last changed needed to satisfy both pyright and mypy in strict mode:

import functools
from collections.abc import Callable
from typing import Concatenate


def decorator[S: SomeClass, R, **P](
    f: Callable[Concatenate[S, P], R],
) -> Callable[Concatenate[S, P], R]:
    @functools.wraps(f)
    def wrapper(self: S, /, *args: P.args, **kwargs: P.kwargs) -> R:
        return f(self, *args, **kwargs)

    return wrapper


class SomeClass:
    @decorator
    def a_method(self) -> int: ...


instance = SomeClass()
instance.a_method()

Which is making self positional only.
Otherwise it would generate:

test.py:13: error: Incompatible return value type (got "_Wrapped[[S, **P], R, [S, **P], R]", expected "Callable[[S, **P], R]")  [return-value]
test.py:13: note: "_Wrapped[[S, **P], R, [S, **P], R].__call__" has type "Callable[[Arg(S, 'self'), **P], R]"

Could you explain why pyright accepts the code without the / and mypy doesn't?

[erictraut]

Could you explain why pyright accepts the code without the / and mypy doesn't?

Pyright treats self and cls as positional-only in this case. This hasn't yet been clarified in the typing spec, but there seems to be general consensus that it should be. See here.

[DanielNoord]

Another follow up question, I now have one additional argument in the method signature. How should I express this:

from __future__ import annotations

import functools
from collections.abc import Callable
from typing import Concatenate


class OtherClass: ...


def inserts_class[S: SomeClass, R, **P](
    f: Callable[Concatenate[S, P], R],
) -> Callable[Concatenate[S, P], R]:
    @functools.wraps(f)
    def wrapper(self: S, /, *args: P.args, **kwargs: P.kwargs) -> R:
        kwargs["other"] = OtherClass()
        return f(self, *args, **kwargs)

    return wrapper


def decorator[S: SomeClass, O: OtherClass, R, **P](
    f: Callable[Concatenate[S, O, P], R],
) -> Callable[Concatenate[S, O, P], R]:
    @functools.wraps(f)
    def wrapper(self: S, other: O, *args: P.args, **kwargs: P.kwargs) -> R:
        return f(self, other, *args, **kwargs)

    return wrapper


class SomeClass:
    @inserts_class
    @decorator
    def a_method(self, other: OtherClass) -> int: ...


instance = SomeClass()
instance.a_method()

As you can see the codebase I'm dealing with is quite creative in its use of decorators.
Basically insert_class ensures that certain keyword arguments are given and decorator relies on both self and other to be of a certain type. Adding the / after other will break the code as insert_class inserts the class as keyword argument (and I'm not a liberty to change this).

[erictraut]

ParamSpec and Concatenate don't support the concatenation of keyword arguments. You can read about why this limitation was imposed by looking at the Rejected Ideas section of PEP 612.

I recommend stepping back and seeing if there's another approach to what you're trying to do here.

While I don't mind answering general questions about the type system, it would probably be better for you to post these types of questions to the Python typing forum. That way, they'll get attention from a broader audience.

[DanielNoord]

Thanks Eric, appreciate all the answers!

Completely agree with stepping back 😄 It's just that the monolith webserver at work can't really be refactored that easily and this is the last step towards turning on strict and pyright. I'd hoped to fix these last issues without having to redo all of the decorators. I think I have done so now.

I tried the Discord but got some less than helpful answers 😅 Is there a place where stuff like this can be documented? It doesn't really belong in the Spec (yet) as it clearly depends on underspecced behaviour, but I personally have been looking for good "Method procotols" multiple times already and never could find the right documentation. Is there a place for official "This is probably how you should do this" type of documentation?

[erictraut]

There is a "typing guides" here. Contributes are welcome!