Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mypy shows type error with json.dump(s) and kwargs #8772

Closed
pganssle opened this issue May 4, 2020 · 8 comments
Closed

mypy shows type error with json.dump(s) and kwargs #8772

pganssle opened this issue May 4, 2020 · 8 comments

Comments

@pganssle
Copy link
Member

pganssle commented May 4, 2020

I have noticed that in the seemingly common situation of passing **kwargs to json.dump or json.dumps, mypy fails the type checks unless the kwargs are explicitly annotated as Dict[str, Any]. A simple example:

import json

json_kwargs = dict(indent=2)

json.dumps({}, **json_kwargs)

Which results in:

$ mypy test.py 
test.py:6: error: Argument 2 to "dumps" has incompatible type "**Dict[str, int]"; expected "bool"
test.py:6: error: Argument 2 to "dumps" has incompatible type "**Dict[str, int]"; expected "Optional[Type[JSONEncoder]]"
test.py:6: error: Argument 2 to "dumps" has incompatible type "**Dict[str, int]"; expected "Optional[Tuple[str, str]]"
test.py:6: error: Argument 2 to "dumps" has incompatible type "**Dict[str, int]"; expected "Optional[Callable[[Any], Any]]"
Found 4 errors in 1 file (checked 1 source file)

If I change the example to this:

import json

json_kwargs = dict(indent=2, separators=(",", ";"))

json.dumps({}, **json_kwargs)

The errors are the same, except the inferred type changes to **Dict[str, object]. I can work around it like so:

import json
from typing import Any, Dict

json_kwargs : Dict[str, Any] = dict(indent=2, separators=(",", ";"))

json.dumps({}, **json_kwargs)

Using Python 3.8 and mypy == 0.770 on Arch Linux. The error is still present when using the current master (commit 2a3de7b).

I have also tested this with pytype and pytype does not have the same failure.

@lulivi
Copy link

lulivi commented May 7, 2020

Same happens when passing **kwargs to a class constructor in Python3.7 with same version of mypy:

from typing import Dict


class A:
    """Class A."""

    def __init__(self, a: str, b: str, c: str, d: int = 2) -> None:
        """Construct A object."""
        self.a: str = a
        self.b: str = b
        self.c: str = c
        self.d: int = d


def main() -> None:
    """Run main method."""
    argument_a: str = "argument_a"
    arguments_b_c: Dict[str, str] = {"b": "argument_b", "c": "argument_c"}

    object_a: A = A(a=argument_a, **arguments_b_c)

    print(object_a)


if __name__ == "__main__":
    main()

Gives the error:

> mypy a.py
a.py:21: error: Argument 2 to "A" has incompatible type "**Dict[str, str]"; expected "int"
Found 1 error in 1 file (checked 1 source file)

Edit: Added error.

@JelleZijlstra
Copy link
Member

I feel like there's two issues here (neither of which is directly related to json or class constructors, and neither of which is necessarily a bug):

  • mypy by default infers a type like Dict[str, object] for a heterogeneous dict. This is a reasonable inference, but sometimes a TypedDict may better. To get a TypedDict type you need to explicitly declare it.
  • mypy is very strict about the type of dicts passed as **kwargs to a function. This is type-safe in theory but means the type of a **kwargs dict in most practical cases has to be Dict[str, Any] or a TypedDict.

@rra
Copy link
Contributor

rra commented May 13, 2020

To elaborate on the last point (and to check that I understand the semantics here), I believe that mypy is enforcing that the type of the value of the expanded dict must match every possible keyword argument to the function called. Therefore, if the function's keyword arguments take incompatible types, only a dict whose values are typed as Any (or a TypedDict where mypy knows the types of the values of specific keys) is permitted.

I was instead expecting mypy to err on the side of generosity and instead assume that if the value type of the dict matches the type of any of the keyword arguments of the function, the caller probably knows what they're doing.

I'm now not sure which approach is better. The approach I was expecting is less safe, but mypy's current approach is going to result in a lot of false positives. I suppose the workaround of typing the dict with Any values isn't too bad. Setting up a TypedDict here is a bit tedious, although obviously the most correct.

@msullivan
Copy link
Collaborator

Yeah, my inclination here is to say not-a-bug. Though if somebody wants to open a bug to argue for looser interpretation of kwargs typing, that would be reasonable.

@rra
Copy link
Contributor

rra commented May 19, 2020

Would it be possible to mention this directly in the documentation under common issues? I've seen multiple people get confused by this behavior, and it can be quite difficult to understand what's happening because of the way the type mismatch is reported.

@JukkaL
Copy link
Collaborator

JukkaL commented May 22, 2020

Added #8874 about improving the generated error messages.

@dliu-fn
Copy link

dliu-fn commented Jan 6, 2021

I would argue to lean towards less safe, but assuming the caller probably knows what they're doing. Ask forgiveness, not permission.

@hauntsaninja
Copy link
Collaborator

I think aspects of this behaviour have been changed on master, so if you have complaints, please make sure they're up to date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants