Cannot infer type argument with TypeVarTuple and Callback protocol

[cdce8p]

To Reproduce

# mypy: enable-incomplete-feature=NewGenericSyntax
from collections.abc import Callable
from typing import Protocol

class ActionType(Protocol):
    def __call__(self, var: str, context: int = 2) -> None: ...

class Job[*_Ts]:
    def __init__(self, target: Callable[[*_Ts], None]) -> None:
        self.target = target

def run_job[*_Ts](job: Job[*_Ts], *args: *_Ts) -> None: ...

def a1(action: ActionType) -> None:
    job = Job(action)
    run_job(job, "Hello")  # -> error

Actual Behavior

error: Cannot infer type argument 1 of "run_job"  [misc]

Expected Behavior
No error. Mypy should be able to tell that context has a default value and is thus optional. It already works from pure Callables (without the intermediate generic class).

def run_job_2[*_Ts](action: Callable[[*_Ts], None], *args: *_Ts) -> None: ...

def a2(action: ActionType) -> None:
    run_job_2(action, "Hello")  # works fine

Your Environment

• Mypy version used: mypy 1.11.0+dev.177c8ee7b8166b3dcf89c034a676ef5818edbc38 (compiled: no) (current master)
• Mypy command-line flags: --enable-incomplete-feature=NewGenericSyntax (the bug exists with the old generic syntax as well)
• Python version used: 3.12

[ilevkivskyi]

It seems to me you are confusing this with ParamSpec. TypeVarTuple doesn't have any notion of argument kinds, so after you assigned job = Job(action), the type of job is simply Job[str, int]. I guess what you want is this:

class Job[**P]:
    def __init__(self, target: Callable[P, None]) -> None:
        self.target = target

def run_job[**P](job: Job[P], *args: P.args, **kwargs: P.kwargs) -> None: ...

Btw with the new syntax underscores are not needed, and not recommended.

[erictraut]

@ilevkivskyi, I agree this is a gray area in the typing spec. We should collectively decide whether this is something that should be supported for TypeVarTuples.

I managed to support the above code in pyright (inspired in part by your work in mypy), but we may want to ultimately disallow this in the spec. I don't have a strong opinion one way or the other.

In the off chance that this is helpful, I'll explain how I implemented this in pyright. I'm not that familiar with mypy's internals, so I don't know if the approach I used in pyright is feasible for you. When pyright records the specialized type arguments for a tuple, it stores not only the type of the type argument but also a flag is_unbounded that indicates whether the entry is unbounded (i.e. followed by a ...). To support the capture of signatures with default arguments by a TypeVarTuple, I added a second flag is_optional that indicates whether the type argument is required or optional. In effect, I'm storing a bit of information about the parameter that was captured by the TypeVarTuple. When specializing Job above, the second type argument (with type int) is marked as "optional". With this information, I'm able to support parameters with default values when later expanding the specialized TypeVarTuple value for *args: *Ts.

Code sample in pyright playground

from typing import Callable, Protocol

class ActionType(Protocol):
    def __call__(self, var: str, context: int = 2) -> None: ...

class Job[*_Ts]:
    def __init__(self, target: Callable[[*_Ts], None]) -> None:
        self.target = target

def run_job[*_Ts](job: Job[*_Ts], *args: *_Ts) -> None: ...

def a1(action: ActionType) -> None:
    job = Job(action)
    run_job(job, "Hello")  # OK
    run_job(job, "Hello", 1)  # OK
    run_job(job, "Hello", "1")  # Type error

[cdce8p]

It seems to me you are confusing this with ParamSpec.

No. I can't use ParamSpec in that case unfortunately. The function only accepts *args so I need to use TypeVarTuple for it.

TypeVarTuple doesn't have any notion of argument kinds, so after you assigned job = Job(action), the type of job is simply Job[str, int].

Strictly speaking, you're correct. However as Eric already pointed out, you implemented something like this for Callables already. The run_job_2 example works with mypy even though that also depends on default arguments.

I agree this is a gray area in the typing spec. We should collectively decide whether this is something that should be supported for TypeVarTuples.

I managed to support the above code in pyright (inspired in part by your work in mypy), but we may want to ultimately disallow this in the spec. I don't have a strong opinion one way or the other.

I'm strongly in favor of keeping the existing support for default arguments. Without that the usefulness of TypeVarTuples for Callable and *args typing would be severely limited. There are quite a few functions (especially in the asyncio module) that already depend on that.
python/typeshed#11015
https://peps.python.org/pep-0646/#type-variable-tuples-with-callable

[ilevkivskyi]

you implemented something like this for Callables already.

No, I didn't. ActionType is simply a subtype of Callable[[str], None], and callables were always put in the second pass of inference, this is why example two works.

I know the callables in Python are a mess. I would personally prefer if they were all like def fn(x, y, /, *, z, t) (~ the shell semantics that is already familiar to everyone), but we are where we are. Anyway, even in such ideal world I don't see how this can be consistently/intuitively supported. For example:

class ActionType(Protocol):
    def __call__(self, var: str, context: int = ...) -> None: ...

class Job[*Ts]:
    attr: tuple[*Ts]
    def __init__(self, target: Callable[[*Ts], None]) -> None: ...

action: ActionType
job = Job(action)
reveal_type(job.attr)  # What should this be? tuple[str], tuple[str, int], or tuple[str] | tuple[str, int]?

Of course this is an artificial example, but it shows how we quickly get into the formal kind vs actual kind confusion again. Btw I did implement ~similar horrible special-casing for ParamSpec (even going beyond/against what is in the PEP), but ParamSpec is already inherently ad-hoc and is strongly tied to callables. I want to keep TypeVarTuple more "generic" (i.e. not tied to callables semantics).