mypy crashes with TypeError: Object of type AnyType is not JSON serializable

[kalekseev]

Bug report

mypy crashes at the end of successful check trying to write cache. Not sure if that's django-stubs problem and can't test if mypy works without django-stubs because of many type checks errors mypy doesn't write cache at the end.

What's wrong

  File "/opt/hostedtoolcache/Python/3.9.6/x64/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 87, in main
  File "mypy/main.py", line 165, in run_build
  File "mypy/build.py", line 179, in build
  File "mypy/build.py", line 254, in _build
  File "mypy/build.py", line 2697, in dispatch
  File "mypy/build.py", line 3021, in process_graph
  File "mypy/build.py", line 3138, in process_stale_scc
  File "mypy/build.py", line 2288, in write_cache
  File "mypy/build.py", line 1476, in write_cache
  File "mypy/build.py", line 1428, in json_dumps
  File "/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type AnyType is not JSON serializable

System information

• OS:
• python version: 3.9.6
• django version: 3.2.7
• mypy version: 0.910
• django-stubs version: 1.9.0

[sobolevn]

It can be an issue with custom TypeInfo metadata 😞

We use it in several places:

1. django-stubs/mypy_django_plugin/transformers/querysets.py

   Line 215 in cfd69c0

   field_types = model_type.type.metadata.get("annotated_field_types")

2. django-stubs/mypy_django_plugin/lib/helpers.py

   Line 51 in 8da8ab4

   return model_info.metadata.setdefault("django", {})

3. django-stubs/mypy_django_plugin/transformers/managers.py

   Line 71 in 8da8ab4

   base_manager_info.metadata["from_queryset_managers"][custom_manager_generated_fullname] = new_manager_info.fullname

Can you try to use --pdb option with mypy? https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-pdb

[kalekseev]

Any is inside metadata object inside WithAnnotations, let me know if you need full object

"WithAnnotations[Model, TypedDict({'today': Any, 'created_date': Any})]": {
    ".class": "SymbolTableNode",
    "kind": "Gdef",
    "plugin_generated": True,
    "node": {
        ...
        "metadata": {
            "annotated_field_types": OrderedDict(
                [("today", Any), ("created_date", Any)] 
            )
        },
    },
},

[kalekseev]

The cause is annotate with custom database func

class DatetimeToDate(Func):
    class DatetimeToDateArgJoiner:
        def join(self, args: tuple[str, str]) -> str:
            datetime, tzname = args
            return f" {datetime} at time zone {tzname} "

    arg_joiner = DatetimeToDateArgJoiner()  # type: ignore
    function = "date"

    def __init__(
        self,
        datetime: Combinable | str,
        tzname: Combinable | str,
    ):
        super().__init__(
            datetime,
            tzname,
            output_field=models.DateField(),
        )



Model.objects.annotate(
        today=DatetimeToDate(Now()),
        created_date=DatetimeToDate("created_at"),
)

[sobolevn]

CC @syastrov as the code owner

[christianbundy]

Confirming the bug here too. Any ideas for a workaround?

[sobolevn]

Nothing you can do on your side except --no-incremental: https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-no-incremental

[christianbundy]

Thanks! I think the bug is happening during cache write, not cache read, so I had to use --cache-dir=/dev/null instead.

[christianbundy]

@kalekseev That isn't triggering the error for me. Are there any other ways you've been able to reproduce this? I'd love to throw a minimal repro into a test so that we can try to fix it.

[kalekseev]

@christianbundy have no time to debug that right now. Basically we have two cases that triggers the error and it seems like custom db functions don't play role in it.

1. is trivial

def pure_function() -> None:
    MyModel.objects.annotate(today=TruncDate("created_at"))

Cached object for first case:

{
    ".class": "MypyFile",
    "_fullname": "django_stubs_ext",
    "names": {
        ".class": "SymbolTable",
        "__name__": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "module_public": False,
            "node": {
                ".class": "Var",
                "name": "__name__",
                "fullname": "django_stubs_ext.__name__",
                "type": "builtins.str",
                "flags": ["is_ready"],
            },
        },
        "__doc__": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "module_public": False,
            "node": {
                ".class": "Var",
                "name": "__doc__",
                "fullname": "django_stubs_ext.__doc__",
                "type": "builtins.str",
                "flags": ["is_ready"],
            },
        },
        "__file__": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "module_public": False,
            "node": {
                ".class": "Var",
                "name": "__file__",
                "fullname": "django_stubs_ext.__file__",
                "type": "builtins.str",
                "flags": ["is_ready"],
            },
        },
        "__package__": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "module_public": False,
            "node": {
                ".class": "Var",
                "name": "__package__",
                "fullname": "django_stubs_ext.__package__",
                "type": "builtins.str",
                "flags": ["is_ready"],
            },
        },
        "ValuesQuerySet": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "cross_ref": "django_stubs_ext.aliases.ValuesQuerySet",
        },
        "Annotations": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "cross_ref": "django_stubs_ext.annotations.Annotations",
        },
        "WithAnnotations": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "cross_ref": "django_stubs_ext.annotations.WithAnnotations",
        },
        "monkeypatch": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "cross_ref": "django_stubs_ext.patch.monkeypatch",
        },
        "__all__": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "module_public": False,
            "node": {
                ".class": "Var",
                "name": "__all__",
                "fullname": "django_stubs_ext.__all__",
                "type": {
                    ".class": "Instance",
                    "type_ref": "builtins.list",
                    "args": ["builtins.str"],
                },
                "flags": [],
            },
        },
        "WithAnnotations[project.core.models.MyModel, TypedDict({'created_date': Any})]": {
            ".class": "SymbolTableNode",
            "kind": "Gdef",
            "plugin_generated": True,
            "node": {
                ".class": "TypeInfo",
                "module_name": "django_stubs_ext",
                "fullname": "django_stubs_ext.WithAnnotations[project.core.models.MyModel, TypedDict({'created_date': Any})]",
                "names": {
                    ".class": "SymbolTable",
                    "created_date": {
                        ".class": "SymbolTableNode",
                        "kind": "Mdef",
                        "plugin_generated": True,
                        "node": {
                            ".class": "Var",
                            "name": "created_date",
                            "fullname": "django_stubs_ext.WithAnnotations[project.core.models.MyModel, TypedDict({'created_date': Any})].created_date",
                            "type": {
                                ".class": "AnyType",
                                "type_of_any": 8,
                                "source_any": None,
                                "missing_import_name": None,
                            },
                            "flags": ["is_ready"],
                        },
                    },
                },
                "defn": {
                    ".class": "ClassDef",
                    "name": "WithAnnotations[project.core.models.MyModel, TypedDict({'created_date': Any})]",
                    "fullname": "django_stubs_ext.WithAnnotations[project.core.models.MyModel, TypedDict({'created_date': Any})]",
                    "type_vars": [],
                },
                "abstract_attributes": [],
                "type_vars": [],
                "bases": ["project.core.models.MyModel"],
                "mro": [
                    "django_stubs_ext.WithAnnotations[project.core.models.MyModel, TypedDict({'created_date': Any})]",
                    "project.core.models.MyModel",
                    "project.core.models.CreatedFieldMixin",
                    "project.core.models.AppModel",
                    "project.core.models.BaseAppModel",
                    "django.db.models.base.Model",
                    "builtins.object",
                ],
                "_promote": None,
                "declared_metaclass": None,
                "metaclass_type": None,
                "tuple_type": None,
                "typeddict_type": None,
                "flags": ["is_protocol"],
                "metadata": {
                    "annotated_field_types": OrderedDict([("created_date", Any)])
                },
            },
        },
    },
    "is_stub": False,
    "path": "/python-3.9.4/lib/python3.9/site-packages/django_stubs_ext/__init__.py",
    "is_partial_stub_package": False,
}

2. is more complicated

class MyQueryset(QuerySet):
    def annotate_start_datetime(self) -> models.QuerySet[Model]:
        return self.annotate(
            start_datetime=DateToDatetime(
                date="date",
                time="start_time",
                tzname="path_to_field",
            )
        )

    def to_remind(
        self,
    ) -> QuerySet[Model]:
        return self.annotate_start_datetime().annotate(
            remind_after=DateTimeIntervalAddition(
                datetime="start_datetime",  <- depends on another annotate
                hours=-models.F(
                    "path_to_field"
                ),
            ),
        )

[feslima]

I'm having the same problem here. However, got no success, short of copying and pasting the entire project, when trying to set a minimal reproducible example.

Here, the error happens when there is an annotated query enabled. If I place an reveal_type with any kind of argument (e.g. reveal_type(True) anywhere in the same file where the annotation is defined, the error goes away (doesn't matter if the no_incremental param is set to true or false).

The code below is how part of my models are structured, however the type checking works fine when outside the project where the error is happening:

from typing import TypeVar
from uuid import uuid4

from django.db import models

_P = TypeVar("_P", bound="AbstractBase", covariant=True)


class AbstractBaseManager(models.Manager[_P]):
    def get_all_for_sale(self) -> models.QuerySet[_P]:
        return self.get_queryset().filter(enabled=True)


class AbstractBase(models.Model):
    class Meta:
        abstract = True

    id = models.UUIDField(primary_key=True, default=uuid4, editable=False)
    name = models.CharField("name", max_length=50)
    enabled = models.BooleanField("enabled", default=False)

    objects = AbstractBaseManager()

    def __str__(self):
        return self.name


class BaseManager(AbstractBaseManager["Base"]):
    def get_all_for_sale(self) -> models.QuerySet["Base"]:
        # removing the annotate or placing the reveal_type makes  
        # the error go away
        annotated = self.get_queryset().annotate(
            qtt=models.Sum("variants__inventories__quantity")
        )
        reveal_type(True) 
        return annotated.filter(qtt__gt=0, enabled=True).order_by("name")


class Base(AbstractBase):
    class Meta(AbstractBase.Meta):
        verbose_name = "base"
        verbose_name_plural = "bases"

    objects = BaseManager()


class Variant(AbstractBase):
    class Meta(AbstractBase.Meta):
        verbose_name = "base"
        verbose_name_plural = "bases"

    parent = models.ForeignKey(Base, on_delete=models.CASCADE, related_name="variants")


class Inventory(models.Model):
    class Meta:
        verbose_name = "inventory"
        verbose_name_plural = "inventories"

    variant = models.ForeignKey(
        Variant, on_delete=models.CASCADE, related_name="inventories"
    )
    quantity = models.PositiveSmallIntegerField("quantity", default=0)

The pyproject.toml section of mypy config is as follows:

[tool.mypy]
python_version = 3.8
no_incremental = false
show_traceback = true
raise_exceptions = true
plugins = ["mypy_django_plugin.main"]

[kalekseev]

Created PR #725 with more details about that problem

[yoav-orca]

I'm also hitting this issue, can I help to make progress on it?

[sobolevn]

@yoav-orca you can take a look at #725 and maybe move from there. Thanks! ✋

[felixmeziere]

Same issue here!

[8bitcurser]

If you have this issue using OS X, python3 and a virtual env I managed to workaround it by uninstalling version 1.9.0 of the library and setting it to 1.8.0 🤷

[playpauseandstop]

Hi there!

Earlier today, I've run into this error, while making an upgrade of my 25K LoC Django project from django-stubs==1.8.0 to 1.9.0 (and from django-stubs-ext==0.2.0 to 0.3.1).

My project does not have much annotates: only 13 to be precise, so I decided to try @kalekseev PR with fix: #725.

It helps and everything works well for the project (at Python 3.9.8 & mypy 0.910), so I'd like to provide my help here and am wondering what steps need to be done in #725 to have it merged into the master and have another release of django-stubs?

I've seen comments that while PR fixes the original issue, it might have other drawbacks, but I'm, at a moment, cannot find any and checked that custom annotations works as expected,

class TotalUsersDict(TypedDict):
    total_users: int

...

    def get_queryset(
        self, request: HttpRequest
    ) -> QuerySet[WithAnnotations[UserProfile, TotalUsersDict]]:
        return cast(
            "QuerySet[WithAnnotations[UserProfile, TotalUsersDict]]",
            super()
            .get_queryset(request)
            .annotate(total_users=Count("user")),
        )

    @admin.display(description=_("Total users"))
    def total_users_link(self, obj: WithAnnotations[UserProfile, TotalUsersDict]) -> str:
         reveal_type(obj.total_users)  # Revealed type is "builtins.int"
         ...

---

@sobolevn

I have also take a look at GitHub Actions failures and they are actually about pystache, which is required by pytest-mytest-plugins and is not being compatible with latest setuptools version, the error which has been already fixed via #734, so in reality the PR is green (if it will be forwarded to latest master).

---

With all of that in mind and considering my willingness to help here, how can I be useful? Which cases should I cover to ensure that #725 is ready to be merged into master?

Thanks!

[sobolevn]

With all of that in mind and considering my willingness to help here, how can I be useful?

There are a lot of things to do! You can start with fixing the CI, try updating pytest-mypy-plugins to the latest version.
Also, removing cache from github actions seems to work.

Then, we can try to polish @kalekseev's PR.
Feel free to ask any questions 🙂

Thanks a lot!

[flaeppe]

Could the cache-write issue be resolved by adding a no_serialize=True here:

django-stubs/mypy_django_plugin/transformers/models.py

Lines 531 to 536 in 515e382

	annotated_typeinfo = add_new_class_for_module(
	model_module_file,
	type_name,
	bases=[model_type] if fields_dict is not None else [model_type, annotated_model_type],
	fields=fields_dict.items if fields_dict is not None else None,
	)

Though it wouldn't resolve the value of any type_ref written to cache, which means that the cache-read lookup issue seen in #725 still has to be resolved.

[sebastian-philipp]

Using no_serialize=True fixes the issue for me. Though I'm still looking for a reproducer