Can we have a generic Type[C]?

[NYKevin]

The type object occupies a rather strange place in the type hierarchy:

>>> type(type) is type
True

(I'm pretty sure that's a flat lie, since you can't instantiate something from itself, but regardless...)

>>> isinstance(type, object)
True
>>> isinstance(object, type)
True
>>> isinstance(type, type)
True

In Java, the (very rough) equivalent is a class, specifically Class<T>. It's also generic; the type variable refers to the instance. Java has it easy because they don't support metaclasses. Classes are not first class in Java, so their type hierarchy doesn't have to deal with the strange loops shown above.

I realize metaclasses in their full generality are out of scope (at least for now), but a generic Type[T] like Java's would be nice to have. So far as I can tell from the Mypy documentation, it doesn't currently exist.

Here's some example code which might like to have this feature:

def make_foo(class_: Type[T]) -> T:
    # Instantiate and return a T

[ilevkivskyi]

@NYKevin, One of the ideas could be to make first-class classes invariant and add a keyword, something like Type[Foo, nonstrict=True] that would make them covariant. The word nonstrict sounds scary, so that people who are going to use it, will be prepared to have some errors at runtime :-)

[agronholm]

I don't think that is valid Python syntax.

[ilevkivskyi]

Good point :-) One can of course define a constant nonstrict in typing.py and write Type[Foo, nonstrict], but I believe all these are details. One needs first to implement it (Type[Foo]) in any form then do some fixes.

@JukkaL I think Callable[..., X] is a good first approximation for a value with type Type[X].

[NYKevin]

@agronholm The only reason Java calls the superclass implementation in that case is because it supports overloading. Python doesn't, so I would expect LSP adherence in practice. It may be difficult to enforce statically, however (cf. the Circle-Ellipse problem). But that doesn't mean you can get away with violating it for non-static methods. Things will break, whether the linter catches it or not.

@ilevkivskyi That seems a rather odd syntax to me. I'm not sure invariant Type[T] is necessarily all that useful anyway. What's the point of a type variable if you know statically what its value will be?

[ilevkivskyi]

@NYKevin, I agree, it is not very useful, but one can use it for something like def foo(cls: Type[T]) -> T: ... as you originally proposed.

[gvanrossum]

This seems to run into some of the same objections as handling the classmethod first argument does in #292, where we also have the problem of how to type-check calling the class, since we don't know what the signature of a subclass's constructor is.

Maybe we can cut through these objections using a similar approach: Never mind the non-Liskov properties of constructor signatures, at least in the first round. If someone defines a class C with a certain constructor signature, and they define an API that takes an argument c annotated with Type[C], then suitable argument values for c are the class C and any subclass thereof. In the API, one can call class methods of C on the argument c, and one can also call c(), with the same signature as C().

In the future we can come up with some way to handle subclasses of C with a different constructor signature. Perhaps we could give Type an optional second parameter that constrains the constructor signature, so that e.g. Type[C, [int, int]] would mean subclasses of C whose constructor takes two integers. Hm... maybe it should be Type[[int, int], C] so it matches Callable? In any case, Type[C] should probably then be limited to subclasses whose constructor signature matches that of C. (To unconstrain the constructor signature, Type[C, ...] or Type[..., C] could be used, similar to Callable[..., C].)

[NYKevin]

Perhaps we could give Type an optional second parameter that constrains the constructor signature, so that e.g. Type[C, [int, int]] would mean subclasses of C whose constructor takes two integers.

If I needed something like that, I would just write Callable[[int, int], C], which is nicely duck-typed. I'm not sure it's a great idea to overcomplicate things here.

I'd much rather assume something like this: "Well-behaved constructors support cooperative multiple inheritance; they should consume the arguments they recognize and pass unrecognized arguments up the inheritance chain via super()." Not everybody does that, unfortunately, but it does seem like good design for larger or more complex class hierarchies. Unfortunately, I'm not sure that actually limits the non-Liskov tendencies enough to make this amenable to static analysis.

Another option would be to ban calling types entirely. If you want a callable, you should take a callable. Type[T] would then be limited to purely Liskov-compatible operations like calling classmethods (which may well be a better design for this kind of thing anyway). It might then also make sense to have a CallableType[[args], T] as you suggest, but I'm not actually aware of any use case where you specifically need both those things at once, and you can always just multiply inherit from Type and Callable on a case-by-case basis.

[gvanrossum]

I would just write Callable[[int, int], C]

That's great if all you need is a factory, and if that's all we needed we wouldn't need Type[C] at all. But what if you're also using it as a class? E.g. call a class method, use it with isinstance() or issubclass(), or possibly even use it as a base class for a dynamically created new class.

Well-behaved constructors support cooperative multiple inheritance

IMO that's a dangerous myth -- it is essentially claiming a very specific protocol for constructors as the one true way. That's okay for a library or framework, but I wouldn't want Python itself to enforce it, and I don't think it's appropriate for a type checker either. (Different frameworks may have different religions about constructors and a type checker should be able to support them all equally well.)

(Also, why shouldn't a recognized argument also be passed via super()? That's how non-constructor methods do it after all.)

Another option would be to ban calling types entirely

I originally started out thinking that's a reasonable alternative, but looking at actual code where developers have asked for Type[C] I found that they are often constructing instances (but not just that, so Callable isn't a substitute).

However: I am totally fine with punting on all this and just supporting Type[C] for now. (I am also considering making it so that at runtime you can call Type[] with any number of parameters -- this would make it easier to evolve the rules in mypy without having to distribute a new version of typing.py.)

[NYKevin]

(Also, why shouldn't a recognized argument also be passed via super()? That's how non-constructor methods do it after all.)

Because object.__init__() doesn't take arguments. Every argument has to be swallowed at some point in the MRO before you reach object, or it errors out. There are exactly two sensible ways to do that: swallow them as you recognize them, or swallow them all at once right before object. The latter requires some root type that always appears second-to-last in the MRO, which is needless complexity. It also makes it significantly harder to detect too-many-arguments, where as swallow-as-you-go gives you that for free.

(Of course, in practice, a lot of code out there takes the third approach of "swallow everything immediately and don't even call super().__init__()," which I call bad design but you might just as easily call "YAGNI." This is definitely a matter of opinion.)

(Different frameworks may have different religions about constructors and a type checker should be able to support them all equally well.)

Fair enough, though I believe the system I just described is implicitly endorsed by the design of super(). But some people might reasonably disagree, and even if I am right, most developers won't want to go back and change their constructors just to make the type checker happy.

I originally started out thinking that's a reasonable alternative, but looking at actual code where developers have asked for Type[C] I found that they are often constructing instances (but not just that, so Callable isn't a substitute).

I would refactor that code into a classmethod, if possible.

But again, we don't want to demand that developers refactor their code in arbitrary ways just to use the type checker, so we'd need to support this use case even if we were sure it was always a bad idea (which we're not).

(I am also considering making it so that at runtime you can call Type[] with any number of parameters -- this would make it easier to evolve the rules in mypy without having to distribute a new version of typing.py.)

I like this, as a compromise. Maybe throw in a warning in strict mode (is strict mode still a thing?).

[gvanrossum]

Because object.__init__() doesn't take arguments. Every argument has to be swallowed at some point in the MRO before you reach object, or it errors out. [...] The latter requires some root type that always appears second-to-last in the MRO, which is needless complexity.

Why would such a root class be a bad idea? To me it sounds more sensible that assuming you can just multiply-inherit from anything that inherits from object. That's patently false anyways (try MI from list and dict :-).

MI is a great idea but it needs to be tamed. We used to have a meme that mix-in classes were the only way to tame it. That was probably true in the Python 2 days of classic classes. But nowadays a framework-specific root class sounds way better. You're never going to be successful using MI with two base classes that weren't designed to cooperate. Sharing a framework-specific root class sounds like a perfect way to signal that you're expecting to cooperate, and that you're using the framework's protocol for constructor super-calls. The MRO mostly d

Fair enough, though I believe the system I just described is implicitly endorsed by the design of super().

The design of super() is primarily meant to support "Liskov" for ordinary (non-constructor) methods. It is additionally constrained by the need for backwards compatibility, e.g. you can choose not to use it. If you don't call your super method you're implicitly constraining yourself to single inheritance. If you do call your super method you may support MI, if you do it right. For ordinary methods that's simple. For constructors, Python doesn't provide any additional support, but the needs and conventions are nevertheless different. Again, for SI you can do it any way you like (Liskov is not needed), and for MI you have to agree on a protocol. But I don't think that that protocol has to be the one implemented by object.

Type[] with any number of parameters

I like this, as a compromise. Maybe throw in a warning in strict mode (is strict mode still a thing?).

The idea is that at runtime it's completely silent, but a type checker can require a specific format. So if mypy doesn't support Type[X, Y] it should certainly flag that as an error. And we should all agree on what Type[X, Y] means before supporting it -- but my hope is that we won't be constrained by what the runtime library in 3.5.2 supports, since that library couldn't care less. (It's much easier to roll out a new version of mypy than to roll out a new version of typing.py, since the latter is baked into the stdlib of 3.5.x, and a PyPI package can't override a stdlib module.)

[NYKevin]

I'm still running into a block on callable types. Let me explain again why I don't like callable types.

LSP was not designed for first-class classes, and only very roughly supports MI. There is no such thing as an "LSP-correct constructor," and with MI it isn't even possible to reason about "the superclass" of a given class (even if that class was itself written with SI). Combined with Python's lack of overload support, it's completely impossible to determine at compile time whether a given constructor call is legal without knowing the identity of the class.

In many cases, you actually do know this; if a type literal (i.e. the direct name of a class, without any runtime lookup etc.) is passed to something taking a Type[T], you know T at compile time, and can verify the constructor calls are safe. The problem is with the other cases.

You have two options for those cases. You can be lenient, and allow code which is not actually type safe, or you can be strict, and prohibit code that is safe. I favor the latter, since it can be overridden with type: ignore and avoided using classmethods and other LSP-respecting patterns instead of directly calling a constructor.

And if you really need callable types, you can always write this:

T = TypeVar('T', bound=SomeClass)
class SomeClassType(Type[T], Callable[[args], T]):
    ...

(replacing args with whatever SomeClass's constructor is supposed to take.)

Given that, I don't see why mypy/typing.py needs built-in support for callable types, unless there is some use case where you would have to write out a lot of SomeClassType declarations, which I find difficult to imagine.

[JukkaL]

Just making Type[...] support specifying argument types would only partially solve the issue. We couldn't use that safely for the cls argument of class methods without some assurances that any derived classes will have a compatible __init__ signature. One option would be to require cls to be annotated if a class method wants to construct an instance. Something like this:

class A:
    def __init__(self, s: str) -> None: ...

    @classmethod
    def factory(cls: Type[[str], A], x: int) -> A: 
        a = cls(str(x))
        a.clsmethod()  # ok
        return a

    @classmethod
    def clsmethod(cls) -> None: 
        ...  # not possible to call cls() here since no arg types provided

class B(A):
    def __init__(self, x: int) -> None: ...  # error, because not compatible with `factory`

Now the fact that A.factory calls the cls argument with a single str argument is part of the public interface of the class. The type of cls would just default to Type[A], which would not allow cls to be called. This has some clear tradeoffs:

• In a class with many class methods, the cls annotation may have to be repeated many times. Using a type alias would be reasonable to make this less of an issue.
• It's not possible to specify default argument values or keyword arguments for __init__ via Type[...]. If those are needed, ... args must be used. Not sure how common a requirement this would be.
• Derived classes don't need to have a __init__ signature that is compatible with the base class -- it just needs to be compatible with all class methods.
• Different class methods can require different __init__ signatures, but if this is the case, they can't call each other.

[gvanrossum]

I like Jukka's plan for phase 2. Explicitly marking up class methods with the expected signature of the cls argument and enforcing that subclasses honor that signature sounds right.

But I don't want to do this in phase 1; it's a complex feature to implement and our immediate requirements aren't focused on worry about the constructor signature.

For phase 1, I propose Type[C] to allow calling the class with the same signature as C, and to also do this for class methods. This covers a very common case in the legacy code I've seen.

[JukkaL]

Sounds reasonable.

[gvanrossum]

FWIW I changed the topic to Type[C] because it's already useful in a non-generic fashion. But of course it should still be generic too.

Non-generic example:

class C: ...
def factory(cls: Type[C]) -> C: ...
class D(C): ...
d = factory(D)  # inferred type is D

Generic example:

def factory(cls: Type[T]) -> T: ...
class E: ...
e = factory(E)  # inferred type is E

[NYKevin]

Do we want Type to be covariant? Maybe I'm misreading, but it looks like
your example says yes.

[gvanrossum]

Mention of covariance usually makes my brain hurt. I wrote exactly what I would expect in those two examples, trying to avoid covariance-induced headaches. Which part made you conclude that Type is covariant? And do you think that's a good or a bad idea?

[NYKevin]

My apologies, I replied too quickly and without fully analyzing the example.

Before I get into my analysis, I'd like to clarify what co/contra/invariance would mean for Type. Assume that C is a literal class (as opposed to a type variable or some other thing). Then Type[C] matches one of the following:

• If Type is invariant, it matches the type object C and nothing else. This isn't terribly useful, but that's only because we used a literal class. If we had used a type variable, it would match the exact type object that was passed or returned.
• If Type is covariant, it matches the type object C together with its subclasses (and their subclasses all the way down the tree).
• If Type is contravariant, it matches the type object C together with its base classes (and their base classes all the way up to object).

Now, your example:

class C: ...
def factory(cls: Type[C]) -> C: ...
class D(C): ...
d = factory(D)  # inferred type is D

If Type is invariant or contravariant, then the fourth line is illegal, because you were asked for a Type[C] but you passed a Type[D], and the latter is not a subtype of the former. Under invariance, C is the only legal argument to this function, while contravariance would also permit any of its (direct or indirect) base classes.

If Type is covariant, then the fourth line is legal, but d is deduced to be a C instance, because that's what factory() is declared as returning. C is a class, not a type variable. C always means C.

Here's what you wanted instead:

class C: ...
T = TypeVar('T', bound=C)
def factory(cls: Type[T]) -> T: ...
class D(C): ...
d = factory(D)  # inferred type is D

Unlike the previous example, this is perfectly legal regardless of the co/contra/invariance of Type. However, if Type is covariant, it's not immediately obvious to me that the implementation of factory() is forbidden from returning a C instance, because that would be legal under the interpretation where T = C; if Type is not covariant, then that interpretation fails because you can't pass D to a Type[C] function. This is most easily prevented by requiring the type checker to deduce the most specific type possible. However, since static analysis is fundamentally impossible, that may not be a realistic requirement.

I tried to figure out how this example works under contravariance, and gave myself a headache. It does seem entirely unworkable if we require most-specific-type-deduction, since we can deduce T as any subclass of D (which could be a very large and unreasonable set of classes). I imagine you'd have to reverse the deduction requirement under contravariance, and require most-general-type-deduction. But then you have to specify that you don't want to accidentally deduce a union type or some other crazy thing.

It should be noted that, as far as I can tell, co/contravariance do not add anything to this example. They solely serve to make it more complex and harder to reason about.

Meanwhile, this example appears to do exactly what you think it does, assuming invariance:

def factory(cls: Type[T]) -> T: ...
class E: ...
e = factory(E)  # inferred type is E

Under co/contravariance, it's less obvious to me that this is correct since T could be inferred as various things which are not E. Again, a specificity requirement might overcome this problem.

In conclusion: I am not convinced that Type[C] is a useful thing to talk about, where C is a literal class. Type[T], on the other hand, would be useful. Therefore, I think Type should be invariant to ensure that the type variable is always exactly the value passed or returned, and not some other value.

[gvanrossum]

Wow. Thanks. I only now realize that my example with Type[C] was nonsense -- if anything it would have made more sense written with

def factory(cls: Type[C]) -> cls: ...

but that would up the ante quite a bit (references to parameter values in other annotations!), so your version with

T = TypeVar('T', bound=C)
def factory(cls: Type[T]) -> T: ...

makes much more sense. Phew!

I believe you are arguing that Type[] should be invariant, and since that's in general the default I am fine with that. And apparently it saves everyone headaches!

I still think there's a use case for Type[C](with C a literal class) -- not for a factory but for other uses. For example the dict mapping strings to classes with a given interface can have its type specified as Dict[str, Type[C]] for some concrete class C. This dict might well be a global variable where using a type variable with bound=C doesn't work. A simplified example:

class Base: ...
type_map = {}  # type: Dict[str, Type[Base]]
def register(cls: Type[Base], key: str) -> None:
    type_map[key] = cls
def make(key: str) -> Base:
    return type_map[key]()

You might well argue that type_map's values should really have type Callable[[], Base], but there might well be other uses of those values -- e.g. calling class methods, using it with isinstance() etc. There might also be a check issubclass(cls, Base) in register() for the benefit of dynamic checking (there might be unchecked legacy code that also calls register()).

[NYKevin]

class Base: ...
type_map = {}  # type: Dict[str, Type[Base]]
def register(cls: Type[Base], key: str) -> None:
    type_map[key] = cls
def make(key: str) -> Base:
    return type_map[key]()

Sorry, but that does not do what you think it does. If Type is invariant, Type[Base] means "only allow Base here", in much the same way as None (or equivalently types.NoneType) means "only allow None here"; it is a singleton type, or a type with exactly one instance (namely Base). So you can't call register() on any class other than Base, which is (if I understand what you're trying to do) entirely useless.

Fixing this example is harder. I don't believe it's actually legal to put a type variable in the type comment of a global variable (though an instance or local variable may be an entirely different story). I'd probably just write Type[Any] and do this:

class Base: ...
type_map = {}  # type: Dict[str, Type[Any]]
T = TypeVar('T', bound=Base)
def register(cls: Type[T], key: str) -> None:
    type_map[key] = cls
def make(key: str) -> Base:
    return type_map[key]()

If I understand the behavior of Any correctly, that would have two implicit type coercions, one from Type[X] to Type[Any] for some subclass X of Base, and another from Type[Any] to Type[Base]. I'm not certain PEP 484 actually guarantees those conversions; if it does not, we have to use Dict[str, Any] instead (yuk!).

This is an example in which covariance would help, since we're not trying to return instances of something more specific than Base, and we could basically use your code unmodified.

Unfortunately, I think you may need to choose between a useful Type[T] (invariant, T is a type variable) and a useful Type[C] (covariant, C is a class). I'm not sure you can nontrivially have both, since type deduction does weird things when you try to combine covariance with type variables. You could special-case Type to choose between these two behaviors dynamically, but I don't like the added complexity; it's clearly incompatible with the relatively simple model espoused by PEP 484, and I don't like moving away from that without a compelling reason, which this corner case really isn't.

[gvanrossum]

I guess this is why we didn't do the sooner... I find the "fix" of using Type[Any] in the declaration of type_map is unacceptable. I will have to try and actually understand your reasoning why Type[] shouldn't be covariant, and hopefully I'll find a hole in it. But I need to sit down and think more.

[JukkaL]

@NYKevin I believe that Type[T] can be covariant, and thus Type[C] would be a reasonable thing to say. I'd like to see an example that shows otherwise. Mypy already seems to be able to deal with similar ambiguity in many places. Your example seems similar to code like this, which is no problem:

T = TypeVar('T')

def id(x: T) -> T: 
    return x

a = id(1)  # type of a?

We could theoretically infer any supertype of int for a but we'll just pick int because it's the least surprising thing to do, as it's the most specific type we can infer.

[NYKevin]

I withdraw my objection to covariance, both because of JukkaL's comment and because it matches up with the behavior of metaclasses: If D is a subclass of C, then D's metaclass is a subclass of (or equal to) C's metaclass.

[gvanrossum]

OK, with that settled I can add a minimal Type class to typing.py.

I think I'll give it minimal functionality and maximum flexibility, so that
in the future we can adjust the PEP without having to update typing.py, and
type checkers have freedom to experiment.

You will be able to write Type[T], Type[T, S], Type[int], Type[sys,
[sys.stdin, sys.stdout, None], sys.modules], or any other crazy stuff; it
will just record the argument values received for whoever wants to
introspect them. Type checkers will reject arguments they don't like.

It should probably reject isinstance(..., Type[...]) and issubclass(..,
Type[...]).

Should I make it final, or do you foresee use cases where one would inherit
from Type[...]? I guess that could be useful for declaring a metaclass --
just like current metaclasses derive from type. With a concrete parameter,

class C: ...
class M(Type[C]): ...

creates a metaclass M that is only usable with subclasses of C. We could
even make M the metaclass of C:

class M(Type['C']): ...
class C(metaclass=M): ...

Potentially useful or not? (Even if we decide not to allow that in mypy,
maybe typing.py still shouldn't reject it? We're getting a lot of trouble
for being too strict with generic base classes, see #85/#115.

[JukkaL]

Not sure whether being able to inherit Type[...] would be useful, but I see no clear reason to disallow that at runtime, so my vote goes for not rejecting it in typing.

[NYKevin]

class M(Type['C']): ... 
class C(metaclass=M): ...

Oohh, that is slick. Here's a real use case:

class EnumMeta(Type['Enum']): ...
class Enum(metaclass=EnumMeta): ...

Skimming Lib/enum.py, it seems obvious to me that EnumMeta and Enum are tightly coupled, and you'd basically never want to write your own separate class with metaclass=EnumMeta. The two classes are not separate components; they are two parts of a single whole. Separating them is unlikely to accomplish anything, and making it a type error sends a nice "don't do that" signal (without actually prohibiting anything at runtime).

[gvanrossum]

Hm, the EnumMeta/Enum pattern occurs a lot, but it usually is just an implementation technique (often the metaclass isn't even public -- or even if it is, it's undocumented and not listed in `__all__`). I don't think that enforcing it in the type checker is going to make much of a difference.

[NYKevin]

You may need a public metaclass, or else some forms of multiple inheritance will break.

# In some library:
class MA(Type['A']): ...
class A(metaclass=MA): ...
class MB(Type['B']): ...
class B(metaclass=MB): ...
# In application code:
class C(A, B): ... # Error: No metaclass

The way to fix this is to make a new metaclass MC which inherits from MA and MB. If they've both been written defensively and don't do anything too wild, MC may be trivial. But the application code "can't" do that if MA or MB is private (to the extent that privacy is a thing, which it really isn't).

You can declare this form of multiple inheritance outside your public API, but personally, I try to support it where reasonable.

[bintoro]

Was Class ever considered?

Type[C] is reminiscent of type(c), so it has that going for it. But if the project aims to maintain some level of distinction between types and classes, Class would be the more accurate choice. Class[C] may be more self-explanatory as well when encountered in code.

[sametmax]

Are types and classes different ? Would we need Type to accept stuff like str, dict, int and custom classes, while Class would accept only classes ?

[gvanrossum]

Hmm... The distinction between "type" and "class" as we're trying to distinguish in PEP 484 is that a class is a runtime concept while a type is a concept in the type checker. So str, dict etc. are classes. They are also types, but only because basically every class is considered a type -- and then there are some things that are types but not classes, such as Any, Union[int, str], Tuple[int, int], and type variables. (However, List[int] is both, though for fairly obscure reasons.)

I don't recall whether we considered Class[C]; I kind of like Type[C] because we can make Type equal to Type[object] and which is exactly what type means. OTOH you've got a point, an argument annotated with Type[C] is really a class object that subclasses C, and giving it a type that's not a class makes little sense -- I wouldn't want to be allowed to allow Type[Any] or Type[Union[...]].

Still I'm reluctant to give up the pun or rhyme with type.

[bintoro]

Right, the ability to substitute Type for type in an annotation is pretty neat — didn't think of that.

[gvanrossum]

Everyone of python-ideas seems to prefer Type[C] over Class[C], so let's not fret over the naming.

Next we need to come up with some text for PEP 484, and a PR for typing.py (both the Python 3 version and the Python 2 backport). All preferably before PyCon, i.e. before May 28 or so (because right after PyCon I'll be depeleted and the 3.5.2 RC will be on June 12, i.e. really soon afterwards).

[gvanrossum]

I think my text about covariance "Type[T] should be considered covariant, since for a concrete class C , Type[C] matches C and any of its subclasses" is bogus.

Looking at the actual class definition in the PR, it seems the type variable used (CT) represents a metaclass. The "variance" allowed here is subclassing the metaclass. I think it's still correct to state that it's covariant, but the text needs to clarify that this refers to the metaclass. HELP!!

[bintoro]

With generic collections, the type parameter denotes the type of the contained elements. But for Type, the type parameter must denote the runtime value (the class) itself, not its type (the metaclass).

What to do here may depend on the definition of a "generic class". If Generic[X] is supposed to always mean that an acceptable runtime value encloses an instance of X, then Type can't be generic.

FWIW, I assumed all along that Type would be a special case like Tuple.

[ilevkivskyi]

I think the problem here is about classes vs types. It is not safe to talk about variance with respect to subclassing, one should talk about variance of subtyping. Type[t1] clearly does not represent a proper class. But it is also not a proper metaclass, in the same way as any generic type is not a proper class. However, Type[t1] is a well defined type.

Therefore I would propose wording like this:

"Type is covariant in its parameter, because Type[Derived] is a subtype of Type[Base]. Indeed, all members of the former type, i.e. classes that subclass Derived are also members of the latter type, since they all also subclass Base."

[ilevkivskyi]

@bintoro Although Tuple is not a generic type, it still behaves perfectly covariant in its parameters. Also I would propose to avoid using term "generic class" to avoid confusions. Only MyGeneric[Any] (or equivalently MyGeneric without parameter) should be called a class, MyGeneric[int], etc. are all types.

[bintoro]

@ilevkivskyi I did not refer to Tuple for its behavior.

I would propose to avoid using term "generic class" to avoid confusions.

I wouldn't because generic classes are a thing. A class that inherits from Generic[T] is a generic class in my books. I would propose to avoid calling them types to avoid confusion.

[ilevkivskyi]

@gvanrossum In addition to my first explanation, I think CT does not represent metaclass, it still represents a type, you could even substitute a Union in place of it.

[gvanrossum]

Hm, I agree CT does not represent a metaclass. The example Type[User] makes this pretty clear. It's Type itself that's like a metaclass (it inherits from 'type'). This is also pretty clear from the new_user() example:

def new_user(user_class: XXX):
    ...
new_user(User)

Compare this to

def foo(arg: int):
    ...
foo(42)

Here, 42 is an instance of int, just like in the previous example User is an instance of type. So where the annotation for arg is a class, the annotation for user_class is a metaclass.

Now back to variance. For this we need a different "regular" example. Let's use

def bar(args: Sequence[int]):
    ...

If we had a subclass of int, MyInt, then Sequence[MyInt] would be a valid type for a call to bar(), and that's what covariance means.

So in our Type example, we have Type[User] as the argument's annotation. For Type to be covariant would mean that if we have another function with an argument declared as Type[ProUser], we could call new_user() with that argument:

def new_pro_user(pro_user_class: Type[ProUser]):
    user = new_user(pro_user_class)
    ...

Indeed that sounds fine to me, so I agree intuitively (without having proven so rigorously) that Type is indeed covariant. (That's a big deal because previously I was just repeating "Type is covariant" because that's what the experts said; I hadn't actually visualized what that statement meant. :-)

Having gotten this far, I like Ivan's first sentence:

"Type is covariant in its parameter, because Type[Derived] is a subtype of Type[Base]."

I don't think the second sentence adds much (even though I agree it's true).

[rwbarton]

I agree that Type should be covariant. Here's my logic, which is basically an expanded version of your "if we have another function with an argument declared as Type[ProUser], we could call new_user() with that argument: [...] Indeed that sounds fine to me".

To test whether Type[Derived] should be a subtype of Type[Base], we need to check whether a value of type Type[Derived] is legal in any context where a value of type Type[Base] is. In order to do that we should start by writing down the typing rules for using values of type Type[T]. My suggestion is:

• If c has type Type[T], C is a class, and T is a subtype of C, then if the expression C(...) type checks, then c(...) also type checks and has type T. [Construction via class variable]
• If c has type Type[T], C is a class, and T is a subtype of C, then if the expression C.m(...) type checks, then c.m(...) also type checks and has the same type as C.m(...). [Invocation of class method via class variable]

(Why do these rules not start simply "If c has type Type[C], then if..."? The argument to Type might not be a class, but instead a Union or a type variable. In the case of a type variable, we need to be able to invoke class methods of the upper bound of the type variable (if it is a class). So I think we are forced into rules of this form.)

Now suppose S is a subtype of T and d has type Type[S]. Take the first rule and substitute d for c leaving the rest of the statement unchanged. Then S is a subtype of C, because S <: T <: C, and so we can apply the typing rule for d and S to conclude that d(...) has type S. Again since S <: T, d(...) also has type T. So, the value d also satisfies the typing rule for T, meaning it can be substituted for c. Similar logic applies to the second rule. So according to these rules, Type should indeed be covariant.

[rwbarton]

One problem with the typing rules above is that they don't actually correspond to python's runtime semantics. For example, every class is a subclass of object, but object's constructor can take 0 arguments, which a typical class's constructor cannot. So we expect a = A to have type Type[A], and then a() should be legal according to the above rule (with C = object), but it could fail at runtime. This is the LSP issue mentioned earlier in this thread. And this issue is unavoidable; we want the new_user example from the PEP to type check, but it really could fail at runtime if BasicUser's constructor isn't compatible with User's.

The language in the PEP about compatibility of method signatures is the assumption that is needed for the typing rule to match the runtime reality. mypy currently checks compatibility for ordinary methods, but not for the constructor. The true covariance rule for runtime behavior is something like "Type[D] is a subtype of Type[C] when D is a subclass of C which overrides the constructor and methods of C in a compatible manner".

Operationally I think mypy should implement these rules by just picking C to be the most specific class that is known to be a supertype of the type T, since that is the class that c is most likely to be compatible with (more so than any of C's superclasses, like object).

[rwbarton]

Also, I quite like @gvanrossum's type_map example. A related example would be code that does something like

class A: ...
class AImpl1(A): ...
class AImpl2(A): ...

if some_condition:
    aImpl = AImpl1  # type: Type[A]
else:
    aImpl = AImpl2

# use class methods on aImpl

Technically this does not require variance since the rule for typing classes could be "C has type Type[T] if T is a supertype of C", but it's nicer to just say "C has type Type[C]" and make use of covariance.

[gvanrossum]

OK, thanks for the (semi-?)formal proof, it helps to know that I didn't miss a case (reasoning about variance just doesn't come naturally to me, I'm probably a closet Eiffel programmer :-).

Of course the weakness of the whole scheme is that the constructor signature of a subclass doesn't have to match that of the base class -- but that's not specific to the argument for covariance. There's already text in the PEP that promises to get back to this issue in the future.

The other thought that this triggered for me is that a class method might itself be a factory that uses Type[T] in its return type, for some type variable T whose upper bound is C. Then c.m() needn't have the same type as C.m() -- wherever the return type of C.m() uses C, the return type of c.m() uses c. But honestly I don't want to weigh the PEP down with this formalism anyways, so we can hand-wave this away.

[rwbarton]

It should probably reject isinstance(..., Type[...]) and issubclass(.., Type[...]).

I'm not sure which "it" you're referring to here: mypy or typing.py's runtime implementation. If these are rejected at runtime, then I would tend to prefer that they be rejected by mypy too.

Trying this out, mypy already gives an error

x/type.py:7: error: Generic type is prohibited as a runtime expression (use a type alias or '# type:' comment)

but with a type alias mypy accepts the isinstance call since Type[T] is a subtype of type. Seems like it would take a special case for mypy to reject this form, so maybe it should be allowed at runtime too?

Running under python gave a baffling error, looks like an unrelated issue?:

Traceback (most recent call last):
  File "x/type.py", line 7, in <module>
    isinstance(1, Type_int)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/typing.py", line 996, in __instancecheck__
    return self.__subclasscheck__(instance.__class__)
TypeError: __subclasscheck__() takes exactly one argument (0 given)

For context, that __instancecheck__ is a method of GenericMeta and the full program I am running is

from typing import TypeVar, Generic

T = TypeVar('T')
class Type(type, Generic[T], extra=type): pass

Type_int = Type[int]
isinstance(1, Type_int)

[rwbarton]

The other thought that this triggered for me is that a class method might itself be a factory that uses Type[T] in its return type, for some type variable T whose upper bound is C. Then c.m() needn't have the same type as C.m() -- wherever the return type of C.m() uses C, the return type of c.m() uses c. But honestly I don't want to weigh the PEP down with this formalism anyways, so we can hand-wave this away.

Something like this crossed my mind too; I think it might fit better with the SelfType stuff discussed elsewhere and we should not worry about it right now.

[gvanrossum]

(Our remarks about constructor signatures and LSP crossed, but I think we're in agreement. Note that object is even more special than most other classes, because it also has a wacko rule about compatibility between __new__ and __init__ if you implement one of them but not the other. But I digress.)

[gvanrossum]

I'm not sure which "it" you're referring to here

Me neither, but most likely I was thinking about the type checker. Or perhaps the PEP. Since at runtime isinstance() involving special stuff is almost always forbidden, and issubclass() is forbidden by the PEP (though the runtime still allows it -- ripping it out is still a task (#136) but it's stalled by some unforeseen problems).

[ilevkivskyi]

@gvanrossum I agree that my second sentence does not add much, but I do not see a shorter way of explaining the covariance than @rwbarton did. I think probably it would be better to simply add a short example, illustrating that a value of type Type[Derived] is legal in a context where a value of type Type[Base] is. Something like this:

``Type`` is covariant in its parameter, because ``Type[Derived]`` is a subtype of ``Type[Base]``::
  def new_pro_user(pro_user_class: Type[ProUser]):
      user = new_user(pro_user_class) # OK
    ...

[gvanrossum]

Done! I've also added a similar call to new_user() to the earlier Union example (implicitly acknowledging covariance there without mentioning it, to avoid scaring people unnecessarily).

[gvanrossum]

Closing -- both the PEP and typing.py have been updated. We're waiting for mypy but that's also very close (and not essential to this issue): python/mypy#1569