final TypedDict

[crusaderky]

Reopened from #7845 as requested

mypy seems to ignore the @final decorator when it is applied to a TypedDict. This issue becomes material when one invokes the items() or values() methods, or passes a generic string to get(); then the output will be incorrectly upcast to object since mypy thinks it may be dealing with a subclass.

from typing_extensions import final, TypedDict

@final
class Point(TypedDict):
    x: float
    y: float
    z: float


p: Point
x: str
reveal_type(p.values())
reveal_type(p.items())
reveal_type(p.get(x, 0.0))

Output:

12: note: Revealed type is 'typing.ValuesView[builtins.object*]'
13: note: Revealed type is 'typing.AbstractSet[Tuple[builtins.str*, builtins.object*]]'
14: note: Revealed type is 'builtins.object*'

Expected output:

12: note: Revealed type is 'typing.ValuesView[builtins.float*]'
13: note: Revealed type is 'typing.ItemsView[builtins.str*, builtins.float*]'
14: note: Revealed type is 'builtins.float*'

[crusaderky]

Related: #7849, #7865

[JukkaL]

The proposed semantics seem to be that a final TypedDict object doesn't have any extra keys beyond those included in the definition. Also, a final TypedDict can't be used as a base to define derived TypedDicts.

I haven't thought about this carefully, but it may be possible to define this in a sound fashion. A final TypedDict would only be compatible with another final TypedDict, and they must have the same keys and required keys, and the key types must be compatible.

Since this seems to work at runtime, this could be implemented without changes to typing. However, I'm not sure how useful this would be. I'd be interested in hearing if anybody has real-world use cases where this would be helpful.

[ilevkivskyi]

A final TypedDict would only be compatible with another final TypedDict, and they must have the same keys and required keys, and the key types must be compatible.

Exactly, it looks like this is the only way to make this sound. This however may be not very useful, as one would need to have same typed dict all the way down the call stack. On the other hand this may be OK for new codebases.

[crusaderky]

The same problem is on the keys. .keys() and .__iter__() yield str according to mypy, but the final decorator could make it change that to Literal.

Real life use case:

from typing_extensions import TypedDict, final


@final
class Counters(TypedDict):
    counter_1: int
    blah_blah: int
    something_else: int
    and_another: int
    one_more: int


def reset_counters(c: Counters) -> None:
    for k in c:
        c[k] = 0  # error: TypedDict key must be a string literal; expected one of ('counter_1', 'blah_blah', 'something_else', 'and_another', 'one_more')

[intgr]

I would find the implementation propsed by @JukkaL quite useful. I have a TypedDict whose all values are List[something] and I would like to iterate over the dict's items() in a generic manner.

[JosiahKane]

I would also find this useful in annotating real world code, as indicated in the linked closed issue.

My actual usecase is a Computer Vision type problem where we're using TypedDict to package together corresponding images. For example, there might be an RGB image and a binary mask indicating the foreground of that image.

class Image:
    def save(self, filename: str) -> None:
        ...

class ColourImage(Image):
    ...

class BitmaskImage(Image):
    ...

@final
class ImageAnnotation(TypedDict):
    img: ColourImage
    foreground: BitmaskImage
    # Although these are all Images, we don't want to use a general Dict[str, Image] because we'd lose the specialisations.

def save_all(img_pack: ImageAnnotation, prefix: str) -> None:
    for k, v in img_pack.items():
        # Currently this fails because save is not defined for object. 
        v.save(f"{prefix}_{k}.png")
        # It would instead be necessary to say 
        # for k, v in cast(Mapping[str, Image], img_pack).items():

A particularly elegant construction that might be enabled by marking a TypedDict closed for extension is the ability to create a new one of the same kind with a comprehension. For example

ImageT = TypeVar("ImageT", bound=Image)

def resize(img: ImageT, scale: float) -> ImageT:
    ...

def resize_all(img_pack: ImageAnnotation, scale: float) -> ImageAnnotation:
    return {k: resize(v, scale) for k, v in img_pack.items()}
    # This wouldn't work even with a cast

[wmdrichards]

I've also just run into this issue -- use case is I have a number of TypedDicts that are all Mapping[str, T], but for each TypedDict I'd like to restrict the keys to a known set. Besides declaring @final, one other option that comes to mind is to make TypedDict a generic class, so that the following syntax would work:

class Point(TypedDict[float]):
    x: float
    y: float
    z: float

This would I think solve the use cases in this thread, without disallowing extension

[richardxia]

However, I'm not sure how useful this would be. I'd be interested in hearing if anybody has real-world use cases where this would be helpful.

I just hit a case where it would be useful to have a final TypedDict. I have a utility function that accepts a list[dict[str, str]] and writes it out to a CSV file using csv.DictWriter. I'd like to be able to pass arbitrary TypedDicts to the utility function where I know the type of all the keys in the TypedDict is str. Currently, it fails because the TypedDicts don't conform to dict[str, str], presumably because non-final TypedDicts don't preclude other keys with different values. Since I am constructing the TypedDicts all within Python, I can guarantee that there are no other keys present, but there's no way for me to communicate that to the type system.

I suppose I could use a dataclass, but one thing that is useful about using TypedDicts this way is that I can use keys that are not valid Python identifiers (e.g. containing spaces), which I can directly serialize as a header row of the CSV, making it easier for non-technical users to read the CSVs.

[intgr]

I think most use cases mentioned here would be better served by intersection types: python/typing#213

So if you want to express that all values of a dict are float and it also conforms to a TypedDict, you would write SomeTypedDict & dict[str, float] or Intersection[SomeTypedDict, dict[str, float]]. And you wouldn't need to artificially restrict subtyping of SomeTypedDict.

[smurzin]

This now also makes mypy diverge with pylance - microsoft/pyright#1899
They've decided to allow final for TypedDict in order to be able to provide type narrowing in Union[TypedDict] case.

Which is problematic for projects that use pylance for IDE/intellisense and mypy in CI.