Reconsider making Mapping covariant in the key type

[JukkaL]

Confusion with invariant container types is one of the more common questions mypy users have. Often these can be easily worked around by using a covariant type such as Sequence instead of List. However, there is no fully covariant dictionary-like type -- Mapping is invariant in the key type, since the key type is used as an argument type of __getitem__ and get. This is particularly problematic in Python 2, as mappings with unicode keys are common, and a mapping with str keys is not compatible with them, even though they are fine at runtime (at least if the keys are ascii only).

I can see a few things we could do.

1) Don't care about unsafety

We'd make Mapping covariant even though it's known to be unsafe (#273). The argument is that the potential unsafety is less of a problem than the confusing user experience. This wouldn't help with certain related things like using a unicode key to access Mapping[str, X], which is arguably okay.

Note that this was proposed and rejected earlier.

2) Use object in Mapping argument types

We'd make Mapping covariant and use object in the argument type of __getitem__ and get. This would result in less effective type checking, but I'd consider this to still be correct at least for get, since it can always fall back to the default. Also, __getitem__ would typically only generate a KeyError if given a key of an unrelated type, and this is arguably not a type error. Indexing operations on mappings can already fail with KeyError, and this isn't considered a type safety issue.

We could also recommend that type checkers special case type checking of Mapping.__getitem__ and Mapping.get (and also the corresponding methods in subclasses such as Dict). A reasonable rule would be to require that the actual key type in code like d[k] overlaps with the declared key type. This would allow things like using unicode keys for Mapping[str, int], which should perhaps be fine in Python 2. We could still reject things like d[1] if the declared key type is str, since int and str are not overlapping.

Subclasses of Mapping that only support specific runtime key types would now be rejected. For example, this wouldn't work any more:

class IntMap(Mapping[int, int]):
    def __getitem__(self, k: int) -> int: ...   # needs an object argument
    ...

However, it would be easy enough to refactor this to type check:

class IntMap(Mapping[int, int]):
    def __getitem__(self, k: object) -> int: ...
        assert isinstance(k, int)  # sorry, can't deal with non-int keys
        ...
    ...

I think that this is reasonable -- the required assert (or cast) makes it clear that this is potentially unsafe.

3) Just teach users to work around using unions or type variables

Maybe keeping Mapping is invariant in the key type is reasonable. Often the lack of covariance can be worked around by using other type system features. Here are some examples:

• Use Union[Mapping[str, int], Mapping[unicode, int]] instead of Mapping[unicode, int].
• Use Mapping[AnyStr, int] in Python 2 and Mapping[str, int] in Python 3 using a conditional type alias.
• Define a type variable like AnyText = TypeVar('AnyText', str, Text) and use Mapping[AnyText, int]. This is a little awkward, since in Python 3 the values of the type variable would be str and str.

I don't like these for a few reasons:

• These are not intuitive.
• These are inconsistent with other containers such as Sequence.
• These are more verbose than just Mapping[unicode, int].

[ilevkivskyi]

We could also recommend that type checkers special case type checking of Mapping.__getitem__ and Mapping.get (and also the corresponding methods in subclasses such as Dict).

I think this is a crucial point. I don't see a good solution without adding some internal special-casing in type checkers.

[JukkaL]

Pinging @rowillia, since he had issues with invariance recently.

[ilevkivskyi]

@JukkaL
There appeared an idea, see http://bugs.python.org/issue25988 to have a base class of Mapping that would be contravariant in the key type. It seems to me that sometimes contravariance rather than covariance is something that is wanted. For example (see b.p.o. issue):

def func(arg: BaseMap[str, int]) -> int: ...
x: Dict[Union[str, unicode], int]
func(x) # people want this to work, but it doesn't with invariant Mapping instead of BaseMap

[JukkaL]

It's unclear to me how useful BaseMap would be. For any new feature, I'd like to see how it would help with real-world code. For example, maybe actual use cases often iterate over the keys, which wouldn't be supported by BaseMap. If that's the case, using Mapping[AnyStr, int] or Union[Mapping[str, int], Mapping[unicode, int]] would be more reasonable, and both of these already work.

[ilevkivskyi]

Actually the original proposal in http://bugs.python.org/issue25988 was to add an ABC with two methods __getitem__ and __iter__, but it would be invariant, so this will not help with this problem. TBH my main motivation here is to have a protocol in typing that has __getitem__ and is not "bulky". Now we have Mapping and Sequence but those have 7-8 methods each.

[JukkaL]

Well, it's probably only worth adding such a protocol if it's useful often enough -- defining one manually is easy in case it would only be used occasionally. I don't know whether this is the case, though.

[ilevkivskyi]

OK, right, let us postpone this then. It will be easy to add one later, if we will see that it is actually needed.
Do we need to say something about this in the rejected/postponed section of PEP 544?

[JukkaL]

This is small enough that it probably won't need to be mentioned.

[LunarLanding]

Hi, this might add to the cases where the key not being covariant fails, not sure:
Python 3.8 using pylance for typechecking.

from __future__ import annotations
from typing import *
A=TypeVar('A',bound=Hashable)
B=TypeVar('B')
class C(MutableMapping[A, B], Generic[A, B]):
	def __init__(self):
		self.d = cast(Dict[A,B],dict())
	def test(self,a:A):
		return self.d.get(1) #should not typecheck