Mark Shannon's presentation at the 2017 Language Summit

[gvanrossum]

@ilevkivskyi @markshannon.

Mark observed that the typing module uses classes to represent types. This can be expensive, since e.g. the type List[int] really ought to be the tuple (List, int) but it's actually a class object which has a fair amount of overhead (though not as much as early versions of typing.py, since we now cache these class objects).

If we changed to tuples (or at least objects simpler than class object), we'd have a problem: The simpler object couldn't be subclassed from. But who subclasses List[int]? Then again, maybe simpler objects aren't the point?

Mark also pointed out that after

from typing import List
class C(List[int]): pass
print(C.__mro__)

We find that C.__mro__ has 17 items!

I confirmed this. The roughly equivalent code using collections.abc

from collections.abc import MutableMapping
class C(MutableMapping): pass
print(C.__mro__)

has only 7 items. And subclassing builtins.list

class C(list): pass
print(C.__mro__)

has only three.

This affects performance, e.g. which is faster?

class C(list, Sequence[int]): pass
C().append(1)
class D(Sequence[int], list): pass
D().append(1)

One append() call is 10% faster than the other.

[JukkaL]

I agree with Mark that this is a problem and can be a blocker for adopting type annotations for some projects. We are mostly shielded from the problem at Dropbox because we use comment annotations, but once we migrate to Python 3 we'd potentially have problems with the current approach as well.

Inheriting from List[int] (or, say Dict[str, Foo]) is occasionally useful and probably worth supporting, but Mark's suggestion to use a different syntax for this seems like a reasonable compromise. I think his suggestion was to do something like this, since List[int] is not a class:

from typing import implements, List

@implements(List[int])
class MyList(list): ...

It's unclear whether this would also affect user-defined generic classes. For example, consider this code that works currently:

class Box(Generic[T]): ...

class SpecialBox(Box[int]): ...

class AnotherBox(Box[T]): ...

def do_stuff(box: Box[int], box2: AnotherBox[str]) -> None: ...

If I understood things right, this would be written like this based on Mark's proposal:

class Box(Generic[T]): ...

@implements(Box[int])
class SpecialBox(Box): ...

@implements(Box[T])
class AnotherBox(Box): ...

def do_stuff(box: Box[int], box2: AnotherBox[str]) -> None: ... # No change

This would still mean that user-defined classes may trigger metaclass conflicts due to Generic. This would mostly happen with user-defined generic classes -- just inheriting from Iterable[str], for example, would no longer imply any metaclass restrictions.

These things should probably also work:

@implements(Tuple[int, str])
class MyTuple(tuple): ...

@implements(Tuple[int, ...])
class MyUniformTuple(tuple): ...

@implements('List[int]')  # Still slightly more efficient
class MyList(list): ...

[gvanrossum]

Personally I think this part of the proposal is not workable. E.g. inheriting from Sequence (an existing ABC repurposed as a type) also adds some concrete default implementations, such as __iter__ and __contains__. You'd have to write

@implements(typing.Sequence[T])
class MyList(collections.abc.Sequence):
    ...

[JukkaL]

Maybe that could written like this:

@implements(typing.Sequence[T])
class MyList(typing.Sequence):
    ...

There would still be a duality -- Sequence and other ABCs would be both classes (for things like inheritance, isinstance, etc.) and types. However, indexing a generic class would result in an object that is only a type, not a class. So these would all be true:

• typing.Sequence is both a class and a type (the latter would be equivalent to typing.Sequence[Any] in a type context).
• typing.Sequence[int] is a type but not a class.
• An annotation, a cast and @implements would work with arbitrary types.
• Base classes must all be classes. They can only be types if the type is also a class.
• A custom generic class MyList would be both a class and a type, but Mylist[int] would only be a type.
• Generic type aliases like List[int] and also bare List are only types, not classes.
• Each context where a type or a class is valid would always be either a type or a class context, not both. We'd have to document all of these. For example, the second argument to isinstance would be a class context, whereas the first argument to cast would be a type context.
• String literal escaping is only valid in a type context.
• Any, Union[...], None, Optional[...] and Callable[...] are types but not classes.

This is still a usability regressions and a backward compatibility break, as defining subclasses of generic classes would become different and slightly harder. However, maybe we can live with this, since after we have protocols, using Iterable[x] and other common ABCs as base classes would no longer be necessary. Also, by making the distinction between types and classes explicit, things may be less confusing for users.

Finally, generic types such as Dict[int, T] would have to be real objects, since we need to be able to index them, due to generic type aliases.

[markshannon]

Does anyone actually define their own generics, or even inherit from a type?

As a data point, Zulip makes no use of inheritance to define a class and a type at the same time. In other words there is no code of the form class C(T): ... where T is a class in the typing module does not exist in the Zulip code base.

https://lgtm.com/query/1957210066/project:140750185/lang:python/

[dmoisset]

I've done both things (define own generics and inheriting from something lice Dict[str, str]) in the context of writing stubs for popular libraries, and I've done the second (inheriting from a specific realization of a generic) in application code

[markshannon]

How often, though? I expect that it is a very rare thing to do.
For the first case (stub files):
Stub files are special, we never run them, so it might be OK to allow a type as a base class in stub files.
For the second (application code):
Is the code open source? It would be good to have some real world examples.

[JukkaL]

Mypy itself has three examples:

$ ag 'class.*\(Dict' mypy
mypy/binder.py
18:class Frame(Dict[Key, Type]):

mypy/nodes.py
2351:class SymbolTable(Dict[str, SymbolTableNode]):

mypy/test/testsemanal.py
215:class TypeInfoMap(Dict[str, TypeInfo]):

Also our internal Dropbox codebases have dozens of examples of dict subclasses. Subclassing list seems pretty rare though.

[JelleZijlstra]

As a user, I'm not too bothered with the current state of things, since it's rare that I need to subclass a generic. (I found only a single inheritance from Generic[T] in my codebase.) I hope we won't switch to suboptimal APIs like @implements just to improve performance in rare cases.

[markshannon]

@JelleZijlstra
It isn't just performance. Less code, means fewer bugs.
Also, keeping types and classes distinct in the implementation helps user maintain mental separation.

[markshannon]

@JukkaL
class Frame(Dict[Key, Type]): would be better (IMO) as

@implements(Mapping[Key, Type])
class Frame(dict):

That way the type (mapping) can be kept separate from the implementation (dict).

Similarly for the other two examples.

[JukkaL]

@markshannon What would then happen to methods defined in dict but not in Mapping, such as __init__? In particular:

1. Are they available through Frame (I suppose so, as otherwise there's no way to construct an instance)?
2. What will the signature of __init__ etc. be (with respect to type variables of dict)?

@gvanrossum I wonder if there would be a way to simplify the MROs even if we inherit from List[int] etc.? Could we filter out the cruft, and maybe have a separate __typed_mro__ attribute or something that would have all the generic base classes? Then we could use an inspection API to access the full MRO. This wouldn't directly help with the space usage of generic type objects, though, but maybe this would help with the slower method calls.

[ilevkivskyi]

Here are my comments:

• Is there an actual proposal, what exactly is proposed to change and why? This is completely unclear from the current discussion.
• It is probably too late for any backward incompatible changes. I just checked and FWIW typing is downloaded from PyPI at around 580k downloads/month, and searching for typing imports on GitHub gives more than 25k files. Also my experience is that people complained even about small incompatible changes in internal API in 3.6.0.
• I already mentioned few times in February that I did some profiling and have some ideas about how to speed-up typing. Unfortunately, I didn't have time to implement them.
• It looks like protocols will resolve several problems mentioned above. For example, it will be not necessary to subclass Mapping[int, str] if a class implements it.

[JukkaL]

Yeah, migration would already be tricky. On the other hand, two years from now it would be still much harder, so if we are going change something fundamental, we should do it as soon as possible.

[markshannon]

First of all a bit of history. In the run up to accepting PEP 484, in an email thread with @gvanrossum and @JukkaL I specifically requested that all isinstance() and issubclass() implementations be removed. Looking back, however, it looks as if I didn't make that public. My bad.

I have consistently been of the opinion that equating types and classes is a bad idea.
What has changed is that we now have evidence that doing so is complex and slow and that inheriting from a type is very rare.

@ilevkivskyi The specific proposal is that no classes representing types should inherit from type.
This would simplify the typing module and reduce its performance impact by a large amount.
But performance is not the only problem; coupling types to classes impairs understanding of an already subtle topic.

As the use of type-hints spread, I expect that applications that declare types will remain a (small?) minority, but that applications that use at least one module that uses type-hints will become common.
Therefore, most applications will pay the performance cost of loading the typing module plus the cost of creating types in their library code. We should keep that cost as small as possible, ideally zero.

[gvanrossum]

FWIW isinstance() was indeed removed, per your request. issubclass() remains because there were problems with removing it. There's still an issue open about removing it.