stubtest crashes when testing django-types

[last-partizan]

Hello.

I want to veryfy django-types with stubtest. But it crashes with following message:

Traceback

Traceback (most recent call last):
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/bin/stubtest", line 8, in <module>
    sys.exit(main())
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 1260, in main
    return test_stubs(parse_options(sys.argv[1:]))
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 1153, in test_stubs
    for error in test_module(module):
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 180, in test_module
    yield from verify(stub, runtime, [module_name])
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 229, in verify_mypyfile
    yield from verify(
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/mypy/stubtest.py", line 698, in verify_var
    if isinstance(runtime, Missing):
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/django/utils/functional.py", line 240, in inner
    self._setup()
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/django/contrib/admin/sites.py", line 540, in _setup
    AdminSiteClass = import_string(apps.get_app_config('admin').default_site)
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/django/apps/registry.py", line 154, in get_app_config
    self.check_apps_ready()
  File "/home/serg/.cache/pypoetry/virtualenvs/django-types-BW7xHCiR-py3.10/lib/python3.10/site-packages/django/apps/registry.py", line 136, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

To Reproduce

Clone django-types, install django and mypy, run stubtest django from django-types directory.

Problem

Main problem in this case - is django refusing to work without explicit setup, and django-stubs fixes it with their mypy_django_plugin, but django-types doesn't use that plugin.

What do you think, how to fix this error?

I propose doing something like this:

From 513d6a3655634074c7894168efb92e689bf950f1 Mon Sep 17 00:00:00 2001
From: Serg Tereshchenko <serg.partizan@gmail.com>
Date: Thu, 2 Jun 2022 20:38:05 +0300
Subject: [PATCH] fix: Handle runtime errors during verification

---
 mypy/stubtest.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/mypy/stubtest.py b/mypy/stubtest.py
index a85e9335a..ad0a7e78d 100644
--- a/mypy/stubtest.py
+++ b/mypy/stubtest.py
@@ -187,7 +187,10 @@ def test_module(module_name: str) -> Iterator[Error]:

     with warnings.catch_warnings():
         warnings.simplefilter("ignore")
-        yield from verify(stub, runtime, [module_name])
+        try:
+            yield from verify(stub, runtime, [module_name])
+        except Exception as err:
+            yield Error([module_name], f"Runtime error: {err}", stub, runtime)


 @singledispatch
--
2.36.1

I could hide this behaviour behind a flag, if you don't think this should be default. But this is really helpful for testing django stubs, unless we can find other ideas.

[AlexWaygood]

Cc. @hauntsaninja