Allow using TypedDict for more precise typing of **kwds

[ilevkivskyi]

There are some situations where a user wants to have more precisely typed **kwds. Current syntax only allows homogeneous **kwds:

def fun(x: int, *, **options: str) -> None:
    ...

However, in situations with many heterogeneous options listing all options in the signature could be verbose and will require rewriting some existing code. There is a vague idea to allow TypedDict for such situations. For example:

class Options(TypedDict):
    timeout: int
    alternative: str
    on_error: Callable[[int], None]
    on_timeout: Callable[[], None]
    ...

def fun(x: int, *, **options: Options) -> None:
    ...

Maybe for such cases the TypedDict used should be automatically understood as defined with total=False. Also it is worth mentioning that this feature will allow reusing the TypedDicts in modules where several functions have same (or similar) option sets.

[JukkaL]

The proposed syntax is problematic, as it could also mean a homogeneous **options where each value is an Options. We could work around this with some extra syntax, but it could look messy. Here's an idea:

from mypy_extensions import Expand
...
def fun(x: int, *, **options: Expand[Options]) -> None:
    ...

Expand[...] could also be used for variadic type variables in case we add support for them (for example, Callable[[int, Expand[X]], None] if X is a variadic type variable).

I don't like the idea of implicitly assuming total=False.

[ilevkivskyi]

Yes, the syntax would be ambiguous. Introducing Expand only for this purpose may be not worth it, but using it also for variadic type variables is a good idea (and it feels natural).

I don't like the idea of implicitly assuming total=False.

OK, let's then drop this. In any case it is easy to add it explicitly (I think it will be used often, since keyword arguments are often optional).

[ilevkivskyi]

Raising priority to normal, since this is a very common request (and we have agreed on using Expand[...]).

[rmorshea]

@ilevkivskyi @JukkaL I'm in pretty desperate need of this so I might be able to justify working on this to my employer. Do you have any ideas about how to implement this? If not, are there areas of the code-base that I should look at in order to understand how to approach the problem?

[ilevkivskyi]

Although this is not super tricky feature, it may be not the best choice for the first contribution, unless you like to live dangerously :-)

You will need to add a new special form Expand[...] to mypy_extensions (where TypedDict lives). This should be allowed only as annotation to **kwars, and must have only one type parameter which is a TypedDict. Then look at all things that contain words formal_to_actual and/or actual_to_formal, they might need to be updated.

Also it may be not the best time to do this, since we are in the middle of a big refactoring of semantic analyzer (one of the first steps in mypy logic, that binds names to symbols). It will take two-three more weeks to finish. Otherwise you would need to repeat some parts of logic twice: in mypy/semanal.py and in mypy/newsemanal/semanal.py.

[rmorshea]

@ilevkivskyi, hmm maybe I'll hold off then. Let me know if there's any work a new contributor could provide in order to help out.

[ilevkivskyi]

@rmorshea You can try one of these: good-first-issue (unless they are too easy/not interesting).

[untitaker]

I got around to hacking something together in #7051. It's obviously very rough but hopefully I can build something off of that.

BTW, I think Expand completely breaks the typesystem. What does this do?

x: Expand[dict[str, int]] = 42

You could argue that this should work because having a type that just works for **kwargs is ugly. You could also argue that this is nonsense and should not be allowed.

I think Expand cannot be called a type. IMO the proper solution would be to introduce a breaking change in the typesystem and make **kwargs: X expand X automatically. Code that looks like this right now:

def foo(**kwargs: int): ...

would have to be changed to:

def foo(**kwargs: Dict[str, int]): ...

[untitaker]

@ilevkivskyi I would like to hear your thoughts on changing the behavior on kwargs to expand automatically. The more I think about it the more I resent the Expand type.

[ilevkivskyi]

If you mean this:

IMO the proper solution would be to introduce a breaking change in the typesystem

then no. Just the scale of possible breakages is already an enough reason not to do this. In addition this will introduce an inconsistency with *args (or you want to change them too?). Finally, Expand[...] will need to be introduced later anyway as part of support for variadic generics.

[untitaker]

Yes, I want to change *args the same way. I don't really understand why Expand is the only solution for variadic generics. In any case I feel that introducing a type that does not behave like a type at all is extremely ugly semantics to preserve backwards compat and work around syntactic limitations.

I think the mypy project will have to think about a system for introducing breaking changes (such as versioning hints) because right now it digs itself into a rabbit hole by building on mistakes for the sake of preserving compat.

[ilevkivskyi]

Just to save time, there is very little chance you will convince all of us. By us I mean the whole static typing community, there is also Pyre, pytype, PyRight, PyCharm, etc. and type system and syntax is shared by all of us and is standardized by several PEPs. If you really want to try this, start a discussion at typing-sig@python.org.

[Nearoo]

I don't really feel authorative enough to post here, but I figured there's an off chance that you'll like my suggestion: What about **kwargs: **Options?

[griels]

I don't really feel authorative enough to post here, but I figured there's an off chance that you'll like my suggestion: What about **kwargs: **Options?

I'm happy with either Expand or **, but my current project would really benefit from this ASAP, so whatever gets the most agreement soonest is good for me. I see that the PEP 589 already mentions Expand as a proposition.

[Mattwmaster58]

This would be really useful to have in a project I'm working on ATM. Is there any chance someone authoritative could take a look at #4441? Or is there any other way I could help out? My vote is for **kwargs: **Options, though I suspect it may be a little bit harder to implement.

[rattrayalex]

According to microsoft/pyright#3002 (comment), this is now supported in Pylance:

from typing_extensions import Unpack

class MyKwargs(TypedDict):
  foo: str
  bar: int

def baz(**kwargs: Unpack[MyKwargs]) -> None:
  pass

baz(foo="str", bar=3) # Pylance will affirm these types.

I am not sure what the current behavior of this with mypy is.

[krystean]

According to microsoft/pyright#3002 (comment), this is now supported in Pylance:

from typing_extensions import Unpack

class MyKwargs(TypedDict):
  foo: str
  bar: int

def baz(**kwargs: Unpack[MyKwargs]) -> None:
  pass

baz(foo="str", bar=3) # Pylance will affirm these types.

I am not sure what the current behavior of this with mypy is.

What happens if we don't provide bar as kwarg? Is it an error? Or is Unpack[MyKwargs] like a constrained optional kwargs?

Does Unpack support Union or | like Unpack[MyKwargs | MyOtherKwargs]?

Thank you!

[erictraut]

If you don't provide bar as a kwarg, it is an error because bar is a required field in the TypedDict.

Unpack does not support Union.

[krystean]

If you don't provide bar as a kwarg, it is an error because bar is a required field in the TypedDict.

Ah but we can specify totality like:

class MyKwargs(TypedDict, total=False):
    foo: str
    bar: int 

So bar or any key is not required anymore. Does Unpack still report this as an error?

Unpack does not support Union.

I now realize it's not Unpack that should support Union, it should be TypedDict. And TypedDict already does. MyTypedDict | OtherTypedDict means either of the two, not combining them. Maybe what I meant was like Unpack[MyTypedDict, OtherTypedDict] where it unpacks multiple TypedDicts. But I guess that would complicate things.

Thank you!

[dgellow]

@krystean Just to add to what you said, a small thing I found out yesterday is that you can also use Required and NotRequired for TypedDict if you want to have more control on individual keys (total= is for all keys of the typed dict).

See PEP-655 (also available via typing_extensions).

[rattrayalex]

I am interested in sponsoring work on support for Unpack[T] in mypy if anyone is interested in putting up a PR for this and would find sponsorship helpful.

[RonnyPfannschmidt]

this should also consider partial views on a class

for example if i had a type like

@dataclass
class MyModel:
   id: uuid
   name: str
   hostname: str
   memory_size: int

and i want to declare helpers like

# bad names
Magic = SubsetTypedDict(MyModel, exclude=["id"], total=False)

def wait_for_update(model_or_id: Model|uuid, **kw: Magic) -> Model:
   ...

[devmessias]

If you don't provide bar as a kwarg, it is an error because bar is a required field in the TypedDict.

Unpack does not support Union.

This works for me

from typing import Callable, TypedDict, Union
from typing_extensions import NotRequired, TypedDict

from typing_extensions import Unpack


class MyFuncKwargs(TypedDict):
    a: NotRequired[str]
    b: NotRequired[int]


def pint(b: int) -> None:
    print(b)


def pstr(a: str) -> None:
    print(a)

def my_func(f:Union[Callable[[str], None], Callable[[int], None]], **kwargs: Unpack[MyFuncKwargs]) -> None:
    print(kwargs)
    return


my_func(pstr, a="a")
my_func(pint, b=2)
my_func(pint, c=2)
my_func(pint, a=2)

/home/devmessias/phd/pkgs/pyastrx-proj/pyastrx-my/pyright.py
  /home/devmessias/phd/pkgs/pyastrx-proj/pyastrx-my/pyright.py:26:15 - error: No parameter named "c" (reportGeneralTypeIssues)
  /home/devmessias/phd/pkgs/pyastrx-proj/pyastrx-my/pyright.py:27:17 - error: Argument of type "Literal[2]" cannot be assigned to parameter "a" of type "str" in function "my_func"
    "Literal[2]" is incompatible with "str" (reportGeneralTypeIssues)
2 errors, 0 warnings, 0 informations 
Completed in 0.877sec

Are there any prospects to bring Unpack to mypy?

[devmessias]

I am interested in sponsoring work on support for Unpack[T] in mypy if anyone is interested in putting up a PR for this and would find sponsorship helpful.

Would be great to have unpack available in mypy

[tony]

this should also consider partial views on a class

for example if i had a type like

@dataclass
class MyModel:
   id: uuid
   name: str
   hostname: str
   memory_size: int

and i want to declare helpers like

# bad names
Magic = SubsetTypedDict(MyModel, exclude=["id"], total=False)

def wait_for_update(model_or_id: Model|uuid, **kw: Magic) -> Model:
   ...

@RonnyPfannschmidt Could you make this into an independent issue for this so it gets attention?

1. dataclass can be unpacked to act as a TypedDict
2. exclude fields
3. Being able to pass via **kw: Magic (conformance with Allow using TypedDict for more precise typing of **kwds #4441

P.S. if I understand correctly, you want to be able to take dictionary data,, and hydrate models / dataclasses via **kw. Is that an accurate depiction of what you're getting at?

[RonnyPfannschmidt]

@tony i don't want to hydrate, i want to correctly type helper functions for querying, waiting and other actions that replicate all if not most fields of types

[adam-grant-hendry]

It seems Unpack from typing_extensions does more than what PEP 646 originally proposed (i.e. originally only for *args/TypeVarTuple, not also **kwargs/TypedDict). It now seems to also do what Expand was proposed for in PEP 589. This is perfectly fine by me (I love having Unpack), but I have some questions:

1. Is Expand in PEP 646 also going to be implemented?
2. Will PEP 646 be updated to indicate that Unpack may also be used for variadic TypeDicts?

I think ideally there should only be 1 name and it should work for both types. (It would be confusing, for instance, to have both Expand and Unpack).

ASIDE:

1. If it is agreed there should be one name, my vote is for Unpack, simply because it's already implemented and the terminology matches Python terminology (we speak of "packing"/"unpacking" variables, rather than "collapsing"/"expanding" variables).

2. PEP 646 appears to have implemented a grammar change to utilize starred expressions in indeces beginning in Python 3.11. Could this be expanded to also implement double-starred expressions? There've been discussions here that implementing a ** syntax would require a grammar change. I'm wondering if expanding the grammar change for * would make this less work? This would be a nice-to-have (much like how the grammar was updated to support built-ins in the syntax directly rather than requiring from typing import UppercaseBuiltin and pipes for Union).

[erictraut]

There is a new PEP 692 underway for this functionality. It proposes to add ** syntax and use Unpack for backward compatibility. The new PEP is under review, so if you want to comment on it, I'm sure the feedback would be welcome.

Pyright has provisional support for PEP 692 if you want to play with it.

A few additional notes...
PEP 589 didn't specify Expand. The only reference to Expand is in the rejected alternatives section. There are currently no plans to introduce Expand to the typing module.

You asked whether PEP 646 would be updated to cover new usage of Unpack. That's not how PEPs work. They are intended to be point-in-time proposals, not live documentation of new functionality, so it wouldn't be appropriate to update PEP 646 to cover new uses of Unpack. That will be covered in PEP 692.

[adam-grant-hendry]

There is a new python/peps#2620 underway for this functionality. It proposes to add ** syntax and use Unpack for backward compatibility. The new PEP is under review, so if you want to comment on it, I'm sure the feedback would be welcome.

Pyright has provisional support for PEP 692 if you want to play with it.

Hurray! 🎉

I'll be sure to add some comments (would love to see dual support of Unpack for TypeVarTuple and TypedDict).

PEP 589 didn't specify Expand. The only reference to Expand is in the rejected alternatives section. There are currently no plans to introduce Expand to the typing module.

Yes, thank you for clarifying it was rejected (good catch). Thank you for the clarification!

You asked whether PEP 646 would be updated to cover new usage of Unpack. That's not how PEPs work. They are intended to be point-in-time proposals, not live documentation of new functionality, so it wouldn't be appropriate to update PEP 646 to cover new uses of Unpack. That will be covered in PEP 692.

Yes, another good catch. I wasn't aware of PEP 692, which is why I was asking. However, I've seen "update" notes added to PEPs before. For example, PEP 484 has several:

If type hinting proves useful in general, a syntax for typing variables may be provided in a future Python version. (UPDATE: This syntax was added in Python 3.6 through PEP 526.)

Could an update note be added to PEP 484 if and when PEP 692 is approved? This would help readers and add traceability between the two. Something as simple as

UPDATE: Usage of Unpack also for **kwargs and TypedDict is proposed in PEP 692.

would be great.

[XChikuX]

Since kwargs values can have defaults. Unpacking with defaults would be nice to support.
Is it not possible to Unpack a pydantic(BaseModel) ?
This would make integrations with libraries like strawberry easy.

For example

class AllFilters(BaseModel):
    beg_age: int = 21
    end_age: int = 55
    beg_height: int = 120
    end_height: int = 240
    search_term: str = ' '
    gender: str = 'female'

@strawberry.type
class Query:
    @strawberry.field
    async def get_filtered_users(self, **kwargs: Unpack[AllFilters]) -> List[User]:
        '''This function gets all users that match the filter criteria
        Strong default values are set to prevent empty results 
        '''