Type[T] -> T has a strange behaviour

[ltworf]

This bug report is coming from #8946 and ltworf/typedload#132

Basically the pattern

def f(t: Type[T]) -> T: ...

Works for some types but not others, and I can't see a pattern. For example it will work for a NamedTuple but not for a Tuple.

See this example:

from typing import *
from dataclasses import dataclass
T = TypeVar('T')

@dataclass
class P:
    a: int

class Q(NamedTuple):
    a: int

def create(t: Type[T]) -> T:    ...

# These ones work
reveal_type(create(int))
reveal_type(create(str))
reveal_type(create(List[int]))
reveal_type(create(Dict[int, int]))
reveal_type(create(Q))
reveal_type(create(P))

# These do not
reveal_type(create(Tuple[int]))
reveal_type(create(Union[int,str]))
reveal_type(create(Optional[int]))

Now in the older mypy, the ones that do not work would just be understood as Any, which is absolutely not the intended behaviour. In the latest release instead they fail with error: Argument 1 to "create" has incompatible type "object"; expected "Type[<nothing>]" (and I have no idea of what this means).

Anyway in my view, all of the examples provided should work.

In case I'm wrong, then none of them should work, and probably a way to achieve this is needed.

Background: I am working on a library to load json-like things into more typed classes, and it promises to either return an object of the wanted type, or fail with an exception.

[JelleZijlstra]

This is similar also to #8992.

[ltworf]

Ah yes, sorry for not linking that one too. But that is about the change that happened in mypy. The change is possibly good since the type isn't being inferred. I'm thinking it should either be or not be in all cases.

[JukkaL]

List[int] and Dict[int, int] should have the inferred type object in an expression context, since they can't be used as type objects at runtime.

The error messages given are pretty confusing, but it can be tricky to generate a significantly better error message here.

[ltworf]

Why can't they be used at runtime?

This works

from typedload import load
from typing import *

a = load(('1', 1, 1.1), List[int])
assert a == [1, 1, 1]

[ltworf]

Just now I encountered another strange failure caused by this issue:

from attr import attrs, attrib
from typing import Optional


def validate(*args, **kwargs) -> None:
    ...

@attrs
class Qwe:
    v = attrib(type=Optional[int], default=None, validator=lambda i, _, v: validate(i.address, v))

@attrs
class Asd:
    v: Optional[int] = attrib(default=None, validator=lambda i, _, v: validate(i.address, v))

Both classes do the same thing, however the 1st one fails and the 2nd one is ok with mypy.

The error only seem to happen if the validator parameter is a lambda, just passing a funciton doesn't trigger it.

[glyph]

Is there a new, better way to say "thing that mypy treats as a type, at runtime" that isn't Type[]? I understand why Type is misleading given that it implies a specific run-time protocol, and it would be cool to get safety around that, but if one has lots of code that can already deal with Unions et. al., it's not quite clear how to describe its signatures any more.

[glyph]

For now, I'm resorting to this:

from typing import TYPE_CHECKING, Generic, TypeVar

T = TypeVar("T")

if TYPE_CHECKING:
    class MyType(Generic[T]):
        "A MyPy type object."
else:
    class _ProtoMyType(object):
        def __getitem__(self, t):
            return lambda: t
    MyType = _ProtoMyType()

...

def create(t: Union[Type[T], MyType[T]) -> T:
    "runtime magic lives here"

This seems pretty hacky though.

[wyfo]

I've another issue that seems to be related.

from typing import Type, TypeVar, Union
T = TypeVar("T")
def foo(obj: Union[Type[T], T]) -> T:
    ...
class Bar:
    baz: int
foo(Bar).baz
# Mypy: <nothing> has no attribute "baz"
# Mypy: Argument 1 to "foo" has incompatible type "Type[Bar]"; expected "Type[<nothing>]"

My current workaround is to use overload with

@overload
def foo(obj: Type[T]) -> T:
    ...
@overload
def foo(obj: T) -> T:
    ...

and error disappear.
Hope it help.

[davidfstr]

Background: I am working on a library to load json-like things into more typed classes, and it promises to either return an object of the wanted type, or fail with an exception.

I'm working on the same kind of library (see my implementation here!) and running into this same issue.

For example I want to be able to have a function with signature:

def trycast(type: Type[T], value: object) -> Optional[T]: ...

And then call it like this:

response_json: object
int_or_str = trycast(Union[int, str], response_json)
if int_or_str is not None:
    print('Got a Union[int, str]!')  # int_or_str should be narrowed to Union[int, str] here
else:
    print('Got something else!')

I've even implemented such a function that works at runtime, but mypy doesn't like me passing a Union[int, str] object to trycast as a Type[T]. It fails on the trycast(Union[int, str], response_json) call with:

error: Argument 1 to "trycast_union" has incompatible type "object"; expected "Type[<nothing>]"

Having the following items recognized by mypy as a Type[T] would be especially helpful in type-annotating functions that can manipulate generic type objects at runtime:

• Union[T1, T2, ...]
• Optional[T]
• List[T], list[T]
• Dict[K, V], dict[K, V]
• Literal['foo', 'bar', ...]
• T extends TypedDict, for some T

The "MyType" workaround given earlier in this thread doesn't seem to work to recognize a Union[...] type despite apparently working to recognize an Optional[...] type.

[CarliJoy]

It seems that even very basic usage of Type[T] -> T does not work.

Similiar to the function of @davidfstr I have:

from __future__ import annotations
from typing import TypeVar, Any, Type

TargetType = TypeVar("TargetType", int, float, str)

def convert_type(input_var: Any, target_type: Type[TargetType]) -> TargetType:
    if target_type == str:
        return str(input_var)
    if target_type == int:
        return int(input_var)
    if target_type == float:
        return float(input_var)
    raise NotImplementedError("This Target Type is not supported")

convert_type("1", int) + convert_type("1.2", float)

resulting in

mypy_test_variable_returns2.py:8: error: Incompatible return value type (got "str", expected "int")
mypy_test_variable_returns2.py:8: error: Incompatible return value type (got "str", expected "float")
mypy_test_variable_returns2.py:10: error: Incompatible return value type (got "int", expected "str")
mypy_test_variable_returns2.py:12: error: Incompatible return value type (got "float", expected "int")
mypy_test_variable_returns2.py:12: error: Incompatible return value type (got "float", expected "str")

a fix would be very much appreciated

[ltworf]

Check the PEP that @davidfstr is working on…

[CarliJoy]

@ltworf Thanks for pointing out.
But still I am not sure if that is the issue here.
I would expect the code above to be more or less equivalent to the code below:

from __future__ import annotations
from typing import TypeVar, Any, Type, overload, Union

TargetType = TypeVar("TargetType", int, float, str)

@overload
def convert_type(input_var: Any, target_type: Type[int]) -> int:
    ...

@overload
def convert_type(input_var: Any, target_type: Type[float]) -> float:
    ...

@overload
def convert_type(input_var: Any, target_type: Type[str]) -> str:
    ...

def convert_type(input_var: Any, target_type):
    if target_type == str:
        return str(input_var)
    if target_type == int:
        return int(input_var)
    if target_type == float:
        return str(input_var)  # should raise error but doesn't
    raise NotImplementedError("This Target Type is not supported")

convert_type("1", int) + convert_type("1.2", float)

But as you can see, mypy is not checking that properly either...
And as far I understood in #9773, Type[] should work with everything that you can input in isinstance
So even the basic example doesn't work... :-/

Actually it seems that overloading even basic types also doesn't work as expected:

@overload
def combine(a: str, b: str) -> str:
    ...

@overload
def combine(a: str, b: int) -> int:
    ...

def combine(a: str, b: Union[str, int]) -> Union[str, int]:
    if isinstance(b, str):
        return str(int(a)+int(b))
    if isinstance(b, int):
        return str(int(a) + b) # Also should raise an error but doesn't
    raise NotImplementedError

Or am I off track here?

PS: I know that I could use Single Dispatch for the second example, but they are only minimal working examples for much more complex functions, with much more possible variants (include None, etc...)

[DevilXD]

@CarliJoy In your first example, you are using target_type == ..., which won't trigger MyPy to do any type narrowing. The only proper way to narrow down the types right now, is with isinstance usage.

In your second example, you are correctly using isinstance, meaning that b's type of Union[str, int] is narrowed down to just int. Since int(a) is also of type int, and int + int => int, there is no error returned - it is working as expected in this case.

The underlaying issue is that Type[...] usage right now, is only valid for things that can work as a second argument to isinstance - from the documentation:

The only legal parameters for Type are classes, Any, type variables, and unions of any of these types.

While this is fair and valid for most use-cases, Type[T] won't accept things like:

• Singletons: None, NotImplemented and Ellipsis
• Union or Literal
• List[int], or any parametrized base class from the typing module really
• and many others

See also: #9773

PS: The official documentation actually does a pretty poor job at wording - "unions" are not valid when assigned to Type:

Incompatible types in assignment (expression has type "object", variable has type "Type[Any]")

The correct word there, would be "tuples" of any of these types. I guess I could make a PR that fixes this.

[CarliJoy]

In your second example, you are correctly using isinstance, meaning that b's type of Union[str, int] is narrowed down to just int. Since int(a) is also of type int, and int + int => int, there is no error returned - it is working as expected in this case.

Stupid me. Your are right. Thanks for pointing out.

@CarliJoy In your first example, you are using target_type == ..., which won't trigger MyPy to do any type narrowing. The only proper way to narrow down the types right now, is with isinstance usage.

Yes which is actually the problem here writing a typed converter function that expects a target type that should be returned.
Is there an issue for this already?
Because cast or isinstance won't work in this case.

The underlaying issue is that Type[...] usage right now, is only valid for things that can work as a second argument to isinstance - from the documentation:

The only legal parameters for Type are classes, Any, type variables, and unions of any of these types.

I really hope that #9773 will be implemented in the near future.

[davidfstr]

I really hope that #9773 will be implemented in the near future.

Not until at least Python 3.11. Window for Python 3.10 is closed now. Still finishing PEP 655 before returning to TypeForm.

…

On May 3, 2021, at 10:26 AM, Kound ***@***.***> wrote:  In your second example, you are correctly using isinstance, meaning that b's type of Union[str, int] is narrowed down to just int. Since int(a) is also of type int, and int + int => int, there is no error returned - it is working as expected in this case. Stupid me. Your are right. Thanks for pointing out. @CarliJoy In your first example, you are using target_type == ..., which won't trigger MyPy to do any type narrowing. The only proper way to narrow down the types right now, is with isinstance usage. Yes which is actually the problem here writing a typed converter function that expects a target type that should be returned. Is there an issue for this already? Because cast or isinstance won't work in this case. The underlaying issue is that Type[...] usage right now, is only valid for things that can work as a second argument to isinstance - from the documentation: The only legal parameters for Type are classes, Any, type variables, and unions of any of these types. I really hope that #9773 will be implemented in the near future. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[efokschaner]

I tried to use the workaround in #9003 (comment) but I wasn't able to get it working correctly. It did help me to arrive at a similar workaround I felt like sharing for feedback or in case it helps anyone else.

In my case, I was trying to wrap cattrs.structure() method, so conceptually a signature like my_func(return_type: Type[T]) -> T.

This workaround requires callers to wrap the non-type types from typing with a wrapper called Returns. It also allows the use of None as the parameter, in a similar way that None is allowed in type signatures.

from typing import TYPE_CHECKING, Generic, Optional, Type, TypeVar, Union, overload

T = TypeVar("T")

class Returns(Generic[T]):
    if not TYPE_CHECKING:

        def __class_getitem__(cls, item: object) -> object:
            """Called when Returns is used as an Annotation at runtime.
            We just return the type we're given"""
            return item


@overload
def my_func(return_type: Type[Returns[T]]) -> T:
    pass

@overload
def my_func(return_type: Type[T]) -> T:
    pass

@overload
def my_func(return_type: None) -> None:
    pass

def my_func(return_type: Union[Type[T], Type[Returns[T]], None]) -> Optional[T]:
    # Actually returns a T using cattr
    return None


class SomeClass:
    pass

foo: int = my_func(int) # Infered as returning int
bar: SomeClass = my_func(SomeClass) # Infered as returning SomeClass
baz: Optional[int] = my_func(Returns[Optional[int]]) # Infered as returning Optional[int]

# Error AS INTENDED
# error: Incompatible types in assignment (expression has type "Optional[int]", variable has type "str")
qux: str = my_func(Returns[Optional[int]])

Interactive version at: https://mypy-play.net/?mypy=latest&python=3.8&flags=strict%2Cno-implicit-optional&gist=b4862022f1c7429b603b23c2939cc71e