Bug: overloads not matched when using `TypeVar` with `default` (where Pyright matches correctly)

[MarcoGorelli]

Bug Report

In the example below, mypy reveals the type to be Any, whereas Pyright correctly detects Bar

Note that replacing T | None with T makes Mypy also infer the type correctly

Noticed while working on pandas-stubs

To Reproduce

from typing import Any, Callable, Generic, Protocol, Self, overload, reveal_type

from typing_extensions import TypeVar

T = TypeVar("T", bound=str | int, default=Any)


class Foo(Generic[T]):
    def __init__(self, t: T) -> None:
        self.t = t


class Bar(Protocol):
    @overload
    def apply(self, f: Callable[..., Foo]) -> Self: ...
    @overload
    def apply(self, f: Callable[..., T | None]) -> Foo[T]: ...


def func() -> Foo:
    return Foo(3)


def main(b: Bar) -> None:
    reveal_type(b.apply(func))

https://mypy-play.net/?mypy=latest&python=3.12&gist=6df33ff32887ddfab4c1ae43edd62b2d

Expected Behavior

that reveal_type would show Bar (for reference, this is what Pyright does)

Actual Behavior

note: Revealed type is "Any"

Your Environment

• Mypy version used: 1.15.0
• Mypy command-line flags:
• Mypy configuration options from mypy.ini (and other config files):
• Python version used: 3.12

[sterliakov]

I'm not sure this is even a bug. This:

T = TypeVar("T", bound=Union[str, int], default=Any)

means "T can be anything compatible with str | int". default=Any encourages mypy to fall back to Any when typevar can't be solved. When constraints are solved to Never, we check the results against the upper bound, and if that fails we also try the default. Initially T is solved to Never in the second overload, so we fall back to default and that obviously succeeds (mypy.applytype.get_target_type is responsible for that)

Here's a minimal reproducer without overloads:

from typing import Any, Callable, Generic

from typing_extensions import TypeVar

T = TypeVar("T", bound=str | int, default=Any)


class Foo(Generic[T]):
    def __init__(self, t: T) -> None:
        self.t = t


class Bar:
    def apply(self, f: Callable[..., T | None]) -> Foo[T]: ...


def func() -> Foo:
    return Foo(3)


def main(b: Bar) -> None:
    reveal_type(b.apply(func))  # N: Revealed type is "__main__.Foo[Any]"

Could you elaborate on your use case where default=Any makes sense? Usually it's a bad idea, mypy produces a diagnostic when generic's typevar is not filled, and default=Any essentially just disables that check.

As a result, both overloads are matched, and since they have nothing in common we reduce the final result to Any.

I don't know whether this should or will be fixed, but a simple workaround exists: do not use default=Any at least for protocols where correct inference is important. This reveals Bar as you expect:

from typing import Any, Callable, Generic, Protocol, Self, overload, reveal_type

from typing_extensions import TypeVar

T = TypeVar("T", bound=str | int, default=Any)
U = TypeVar("U", bound=str | int)


class Foo(Generic[T]):
    def __init__(self, t: T) -> None:
        self.t = t


class Bar(Protocol):
    @overload
    def apply(self, f: Callable[..., Foo]) -> Self: ...
    @overload
    def apply(self, f: Callable[..., U | None]) -> Foo[U]: ...


def func() -> Foo:
    return Foo(3)


def main(b: Bar) -> None:
    reveal_type(b.apply(func))

[erictraut]

In the minimal producer without overloads, mypy doesn't detect a type error for the call expression b.apply(func), but I think it should. That seems to be the root of the problem here.

The argument func has the type Callable[[], Foo[Any]]. For the call to succeed, this type must be assignable to Callable[..., T | None] which means Foo[Any] must be assignable to T | None. Since T has an upper bound of str | int, that means Foo[Any] must be assignable to str | int | None, which it is not. That means there's a type violation. This is independent of whether T can be solved or not, so the default value shouldn't come into play here.

If I remove the default from the type parameter T, mypy correctly detects this error.

[sterliakov]

@erictraut Any is a valid solution for T. mypy does not suggest it by default, but using default=Any typevars makes mypy use Any as explicitly allowed value. With T = Any, the call is def apply(f: Callable[..., Any | None]) -> Foo[Any], and thus apply(func) is allowed.

While I probably agree that using default only when there were no constraints at all would be more reasonable, [] represents both "no constraints" and "constraints can't be built" internally, and that won't be an easy fix. I'm not sure whether default=Any is worth such work - essentially this is just another escape hatch, and Any is rather contagious anyway.

[erictraut]

Argument assignability must be satisfied before attempting to solve type variables. The type Foo[Any] is not assignable to T. There are cases where assignability is satisfied but a type variable still cannot be solved. Only in those cases should the default be applied.

Consider this code sample:

T = TypeVar("T", bound=int, default=bool)

def func(x: T | list[str]) -> T: ...

reveal_type(func(1))  # int
reveal_type(func([""]))  # bool
func("")  # Error, even though a solution of T=Any would satisfy it

I agree that default=Any is an odd construct and probably should be discouraged. However, I think mypy's current behavior here is noncompliant with the typing spec and therefore should be considered a bug.

Interestingly, mypy works as I would expect in other cases that involve Any within the default type. You mentioned that "default=Any makes mypy use Any as an explicitly allowed value", but that doesn't seem to extend to default types other than Any. In the example below, would mypy then consider list[Any] an acceptable solution for T? It generates an error for the call func([""]), so that doesn't appear to be the case.

T = TypeVar("T", bound=Sequence[int], default="list[Any]")

def func(x: T) -> T: ...

func([1])  # OK
func([""])  # Error

[sterliakov]

From the spec:

type checkers may use a type parameter’s default when the type parameter cannot be solved to anything.

I think mypy is conformant here: there's no solution for T in the OP example, so mypy is correct to use the provided default of Any?

Your last example is convincing, though: it produces a diagnostic even if T is just default=Any:

T = TypeVar("T", bound=Sequence[int], default=Any)

def func(x: T) -> T: ...

func([1])  # OK
func([""])  # E: Value of type variable "T" of "func" cannot be "list[str]"  [type-var]

So our approach is indeed at least inconsistent.

[erictraut]

I think mypy is conformant here

I can understand why you might find the current wording in the spec to be confusing or ambiguous. As I mentioned above, assignability of arguments is a prerequisite. If the type of an argument isn't assignable to the corresponding parameter type, it should be considered a type violation. Only in cases where the arguments are assignable does it make sense to solve type variables. As I point out above, there are situations where arguments are assignable to parameters but there is still no solution for the type variable because of insufficient constraints. That's what the spec is referring to here where it says "when the type parameter cannot be solved to anything". If I remember correctly, I suggested the current wording in PEP 696 when I reviewed an early draft, so you can probably blame me for its lack of clarity, but I can confirm that was the intended meaning of this phrase.

In any case, it's important for type checkers to be consistent here because, as the OP points out, this affects overload resolution. We've been working hard to pin down the overload resolution specification to improve type checker consistency and make it easier for library and stub maintainers.

[MarcoGorelli]

Thanks all for responses

Could you elaborate on your use case where default=Any makes sense?

In pandas, Series is generic in the data type. For example, Series[int], Series[float], .... There are cases when the inner type can't be determined statically due to value-dependent behaviour (😭), and so Series[Any] is used. I was exploring whether defaulting Series to Series[Any] was feasible

[jorenham]

Could you elaborate on your use case where default=Any makes sense? Usually it's a bad idea, mypy produces a diagnostic when generic's typevar is not filled, and default=Any essentially just disables that check.

In NumPy this is used quite a lot, for example

• floating defaults to floating[Any], (since 2.1)
• dtype defaults to dtype[Any] (since 2.3), and
• ndarray defaults to ndarray[tuple[Any, ...], dtype[Any]] (since 2.3)

There are also some examples of this in the typeshed stdlib stubs, e.g. builtins.slice.