TypedDict: `__required_keys__` and `__optional_keys__` can be wrong in the presence of inheritance

[JelleZijlstra]

Bug report

Bug description:

from typing import NotRequired, Required, TypedDict

class A(TypedDict):
    a: NotRequired[int]
    b: Required[int]

class B(A):
    a: Required[int]
    b: NotRequired[int]

print(B.__required_keys__)
print(B.__optional_keys__)

This will print (tried on 3.11 and current-ish main):

frozenset({'b', 'a'})
frozenset({'b', 'a'})

But obviously, a single key should only be either Required or NotRequired, not both. The child class's version should prevail.

@alicederyn and I discovered this while discussing the implementation of PEP-705 (python/typing_extensions#284). cc @davidfstr for PEP 655.

I also see some problems with how type checkers handle this case, but i'll post about that separately.

CPython versions tested on:

3.11, CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-112509: Fix keys being present in both required_keys and optional_keys in TypedDict #112512
• [3.12] gh-112509: Fix keys being present in both required_keys and optional_keys in TypedDict (GH-112512) #112530
• [3.11] gh-112509: Fix keys being present in both required_keys and optional_keys in TypedDict (GH-112512) #112531

[alicederyn]

This example is illegal according to PEP-589.

A TypedDict type with a required key is not consistent with a TypedDict type where the same key is a non-required key, since the latter allows keys to be deleted.

Concretely, B cannot be a subclass of A as it "removes" the ability to delete key 'b' (breaking substitutability) and also adds None to the possible values returned by get('a') (also breaking substitutability).

[JelleZijlstra]

@alicederyn you may be right that this is illegal at type checking time; I posted python/typing#1516 about this. But as Alex pointed out on the PR, backward compatibility means we can't just prohibit this at runtime. PEP 705 is actually a good illustration of why it's good to be lenient at runtime: it creates a mechanism that does let you combine NotRequired and Required for the same key in a type-safe manner. If we had prohibited that at runtime, it would make it harder for people to use PEP 705 on older Python versions.