Recurrent but not reproducible crash when writing to cache

[aberres]

Crash Report

Recently we (different team members) are now and then running into a mypy crash which can only be resolved by deleting .mypy_cache. We have no idea yet how to reproduce it - it likely is triggered after changing branches. But we could so far not figure out the problematic code path.

Traceback

Traceback (most recent call last):
  File "/Users/abc/venv/weplan3.10/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/Users/abc/venv/weplan3.10/lib/python3.10/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 96, in main
  File "mypy/main.py", line 173, in run_build
  File "mypy/build.py", line 180, in build
  File "mypy/build.py", line 256, in _build
  File "mypy/build.py", line 2727, in dispatch
  File "mypy/build.py", line 3071, in process_graph
  File "mypy/build.py", line 3189, in process_stale_scc
  File "mypy/build.py", line 2311, in write_cache
  File "mypy/build.py", line 1488, in write_cache
  File "mypy/nodes.py", line 344, in serialize
  File "mypy/nodes.py", line 3364, in serialize
  File "mypy/nodes.py", line 3301, in serialize
  File "mypy/nodes.py", line 2827, in serialize
  File "mypy/nodes.py", line 3364, in serialize
  File "mypy/nodes.py", line 3301, in serialize
  File "mypy/nodes.py", line 953, in serialize
  File "mypy/types.py", line 1086, in serialize
  File "mypy/types.py", line 1482, in serialize
  File "mypy/types.py", line 611, in serialize
  File "mypy/types.py", line 1079, in serialize
  File "mypy/nodes.py", line 2676, in fullname
AttributeError: attribute '_fullname' of 'TypeInfo' undefined

To Reproduce

Well yeah, seems to be some constellation of stars and cache state.

Your Environment

• Mypy version used: mypy 0.941 (I think it happened with 0.931 as well)
• Mypy command-line flags: Just a bunch of packages passed with -p and a config file.
• Mypy configuration options from mypy.ini (and other config files):

[mypy]
ignore_missing_imports = True
color_output = True
pretty = True
show_error_codes = True
check_untyped_defs = True
warn_unreachable = True
error_summary = True
# https://github.com/python/mypy/issues/9091
no_implicit_optional = True

# pip install sqlalchemy2-stubs
plugins = sqlalchemy.ext.mypy.plugin

• Python version used: 3.10
• Operating system and version: macOS Big Sur

[pkanavos]

I encountered the same problem just yesterday, after installing mypy, pandas-stubs and sqlalchemy[mypy]. Deleting the cache worked

[ronakts]

We also encountered this issue a few times, but not able to reproduce it. For us it always fails when writing a cache for ddtrace/internal/atexit.py and we also use mypy cache generated from other branches.

Exception:

LOG:  Processing SCC singleton (ddtrace.internal.atexit) as inherently stale
LOG:  Writing ddtrace.internal.atexit /opt/conda/envs/********3/lib/python3.10/site-packages/ddtrace/internal/atexit.py ddtrace/internal/atexit.meta.json ddtrace/internal/atexit.data.json
LOG:  Build finished in 73.491 seconds with 8396 modules, and 0 errors
Traceback (most recent call last):
  File "/opt/conda/envs/********3/lib/python3.10/runpy.py", line 196, in _run_module_as_****
    return _run_code(code, ****_globals, None,
  File "/opt/conda/envs/********3/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/********3/lib/python3.10/site-packages/mypy/__****__.py", line 34, in <module>
    console_entry()
  File "/opt/conda/envs/********3/lib/python3.10/site-packages/mypy/__****__.py", line 12, in console_entry
    ****(None, sys.stdout, sys.stderr)
  File "mypy/****.py", line 96, in ****
  File "mypy/****.py", line 173, in run_build
  File "mypy/build.py", line 180, in build
  File "mypy/build.py", line 256, in _build
  File "mypy/build.py", line 2727, in dispatch
  File "mypy/build.py", line 3075, in process_graph
  File "mypy/build.py", line 3193, in process_stale_scc
  File "mypy/build.py", line 2311, in write_cache
  File "mypy/build.py", line 1488, in write_cache
  File "mypy/nodes.py", line 344, in serialize
  File "mypy/nodes.py", line 3367, in serialize
  File "mypy/nodes.py", line 3304, in serialize
  File "mypy/nodes.py", line 956, in serialize
  File "mypy/types.py", line 1482, in serialize
  File "mypy/types.py", line 611, in serialize
  File "mypy/types.py", line 1079, in serialize
  File "mypy/nodes.py", line 2679, in fullname
AttributeError: attribute '_fullname' of 'TypeInfo' undefined


Exited with code exit status 1

We saw this issue on 0.940, 0.941, and 0.942. If we run mypy without cache we don't face this issue.

[Dreamsorcerer]

I encountered this after upgrading sqlalchemy and sqlalchemy2-stubs. So, maybe changes after upgrading a package can cause this crash.

[AlexWaygood]

I'm raising the priority to "high" given that this appears to be causing problems for lots of users

[ethanzh]

My team has also been experiencing this issue. We also noticed it has something to do with the cache + switching branches but I haven't been able to get a reproducible set of steps. We're also using sqlalchemy stubs as @Dreamsorcerer mentioned so that very well might be part of it.

[AlexWaygood]

Unfortunately it is quite difficult to debug an issue without a reproducer (even a "maximal" reproducer is better than no reproducer at all). If anybody who's experienced this issue is able to put in time trying to get a reproducer, that would be very helpful.

[Dreamsorcerer]

I've not experienced the problem for many releases, so this is basically resolved to me. Maybe it might be caused by a specific change in sqlachemy2-stubs and upgrading from a specific version (after running mypy on your code) might trigger it..

[ethanzh]

I've bumped our Mypy version from 0.942 to 0.950 and I've also replaced types-sqlalchemy for sqlalchemy-stubs. After trying various situations to trigger this crash I haven't been able to. I'll loop back to this thread after a few days of use to report if the issue's been solved for us.

[pranavrajpal]

Is everyone here using the sqlalchemy mypy plugin? If so, there's a decent chance that's what is causing the crash, so it might be worth disabling it for a few days and seeing if that stops the crashes.

Looking at the error message in the traceback, the thing that seems weird is that I can't find any code paths where _fullname wouldn't be defined on a TypeInfo. We define it in both the constructor and when deserializing, and there isn't any place we delete _fullname in mypy.

Some other things I've noticed that may or may not be related:

• fix(mypy): Placeholder vars should have fullname set sqlalchemy/sqlalchemy#7348 was a PR to the sqlalchemy mypy plugin which made it into sqlalchemy 1.4.30 (released January 19, so about 2 months before this issue). The fact that it's making a change to _fullname makes me suspicious that it's related, but it's changing the _fullname of a Var, not a TypeInfo, so it might be unrelated.
• In this serialization part of the traceback, these are the components that we're serializing at each line number:

  File "mypy/nodes.py", line 344, in serialize (MypyFile)
  File "mypy/nodes.py", line 3364, in serialize (SymbolTable)
  File "mypy/nodes.py", line 3301, in serialize (SymbolTableNode)
  File "mypy/nodes.py", line 2827, in serialize (TypeInfo)
  File "mypy/nodes.py", line 3364, in serialize (SymbolTable)
  File "mypy/nodes.py", line 3301, in serialize (SymbolTableNode)
  File "mypy/nodes.py", line 953, in serialize (Var)
  File "mypy/types.py", line 1086, in serialize (Instance)
  File "mypy/types.py", line 1482, in serialize (CallableType)
  File "mypy/types.py", line 611, in serialize (ParamSpecType)
  File "mypy/types.py", line 1079, in serialize (Instance)
  File "mypy/nodes.py", line 2676, in fullname

There's a chance that the TypeInfo that we're trying to get the fullname of (at the bottom of the traceback) is the same TypeInfo that we're serializing earlier (near the top of that traceback), which would probably cause problems if that were possible. The Var being serialized in that traceback also might mean that this is related to the sqlalchemy PR I mentioned, but I don't really have an explanation for why either of those things would cause the _fullname attribute to disappear during deserialization.

One other strange thing I noticed is that the line we're crashing on in the ParamSpecType serialization is this:

mypy/mypy/types.py

Line 611 in 4e18c78

'upper_bound': self.upper_bound.serialize(),

The docstring says the upper bound is always just object, which seems to imply _fullname is missing on the TypeInfo for object, and I don't really understand how that would happen.

[mavidser]

Posting some of my research here - I'm reproduced this with and without the sqlalchemy.ext.mypy.plugin. In my case, the issue if because I have flask-limiter installed, which depends on the package limits.

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/backend/tmp/mypy/mypy/__main__.py", line 34, in <module>
    console_entry()
  File "/backend/tmp/mypy/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/backend/tmp/mypy/mypy/main.py", line 108, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/backend/tmp/mypy/mypy/main.py", line 176, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/backend/tmp/mypy/mypy/build.py", line 155, in build
    sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr, extra_plugins
  File "/backend/tmp/mypy/mypy/build.py", line 230, in _build
    graph = dispatch(sources, manager, stdout)
  File "/backend/tmp/mypy/mypy/build.py", line 2733, in dispatch
    process_graph(graph, manager)
  File "/backend/tmp/mypy/mypy/build.py", line 3101, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/backend/tmp/mypy/mypy/build.py", line 3224, in process_stale_scc
    graph[id].write_cache()
  File "/backend/tmp/mypy/mypy/build.py", line 2320, in write_cache
    self.manager)
  File "/backend/tmp/mypy/mypy/build.py", line 1470, in write_cache
    data = tree.serialize()
  File "/backend/tmp/mypy/mypy/nodes.py", line 352, in serialize
    'names': self.names.serialize(self._fullname),
  File "/backend/tmp/mypy/mypy/nodes.py", line 3439, in serialize
    raise e
  File "/backend/tmp/mypy/mypy/nodes.py", line 3437, in serialize
    data[key] = value.serialize(fullname, key)
  File "/backend/tmp/mypy/mypy/nodes.py", line 3373, in serialize
    data['node'] = self.node.serialize()
  File "/backend/tmp/mypy/mypy/nodes.py", line 2893, in serialize
    'names': self.names.serialize(self.fullname),
  File "/backend/tmp/mypy/mypy/nodes.py", line 3439, in serialize
    raise e
  File "/backend/tmp/mypy/mypy/nodes.py", line 3437, in serialize
    data[key] = value.serialize(fullname, key)
  File "/backend/tmp/mypy/mypy/nodes.py", line 3373, in serialize
    data['node'] = self.node.serialize()
  File "/backend/tmp/mypy/mypy/nodes.py", line 762, in serialize
    'type': None if self.type is None else self.type.serialize(),
  File "/backend/tmp/mypy/mypy/types.py", line 1800, in serialize
    'variables': [v.serialize() for v in self.variables],
  File "/backend/tmp/mypy/mypy/types.py", line 1800, in <listcomp>
    'variables': [v.serialize() for v in self.variables],
  File "/backend/tmp/mypy/mypy/types.py", line 646, in serialize
    'upper_bound': self.upper_bound.serialize(),
  File "/backend/tmp/mypy/mypy/types.py", line 1189, in serialize
    type_ref = self.type.fullname
  File "/backend/tmp/mypy/mypy/nodes.py", line 2986, in __getattribute__
    raise AssertionError(object.__getattribute__(self, 'msg'))
AssertionError: De-serialization failure: TypeInfo not fixed

I was getting the same error as OP initially, before installing mypy with pip -e.

Digging deeper, lines like these seem to be the problem, with the P ParamSpec is causing mypy to crash. No idea what the exact issue is.

Steps to reproduce (at least for me):

• I was using mypy --install-types . which was installing types-Deprecated and types-setuptools due to the limits package.
• Uninstall types-Deprecated and types-setuptools, and removing .mypy_cache folder made the crash reproducible every time.

Might be related to #8645?

[aberres]

@pranavrajpal As the crash persists until the cache is cleaned - is there anything we could debug while being in the broken state?

[aberres]

Hmm, now that I read about ParamSpec above - I started to add ParamSpecs to our code base some time ago. Cannot tell for sure, but I think the crashes might have started at the same time.

[NeilGirdhar]

I'm running into the same problem after adding ParamSpec to my codebase https://github.com/NeilGirdhar/tjax/.

[ethanhs]

Over in #13099 @copperfield42 was able to reduce things to a minimal reproducer:

I manage to reduce to a minimum necessary

#my_typing_simple.py
from typing import TypeVar, ParamSpec, Callable

T = TypeVar('T')
P = ParamSpec('P')

and in another module fail.py:

"failing module"
from my_typing_simple import Callable, P, T

def my_decorator(folder:str) -> Callable[[Callable[P, T]], Callable[P, T]]:
    def decorator(func:Callable[P,T]) -> Callable[P,T]:
        def wrapper(*args:P.args,**kwarg:P.kwargs) -> T:
            print("wrapper")
            return func(*args,**kwarg)
        return wrapper
    return decorator

this one is very weird because seemingly insignificant changes result in the error or not, if I delete the mypy cache and open a new console and test it pass, but if I just add an empty line between the doc string and the imports and test again I get the error

"failling module"

from my_typing_simple import Callable, P, T

def my_decorator(folder:str) -> Callable[[Callable[P, T]], Callable[P, T]]:
    def decorator(func:Callable[P,T]) -> Callable[P,T]:
        def wrapper(*args:P.args,**kwarg:P.kwargs) -> T:
            print("wrapper")
            return func(*args,**kwarg)
        return wrapper
    return decorator

Error

C:\Users\copperfield\Desktop\mypy_tests\paramspec bug>mypy fail.py
Traceback (most recent call last):
  File "C:\Python3.10.5\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python3.10.5\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Python3.10.5\Scripts\mypy.exe\__main__.py", line 7, in <module>
  File "C:\Python3.10.5\lib\site-packages\mypy\__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy\main.py", line 96, in main
  File "mypy\main.py", line 173, in run_build
  File "mypy\build.py", line 154, in build
  File "mypy\build.py", line 230, in _build
  File "mypy\build.py", line 2729, in dispatch
  File "mypy\build.py", line 3087, in process_graph
  File "mypy\build.py", line 3205, in process_stale_scc
  File "mypy\build.py", line 2313, in write_cache
  File "mypy\build.py", line 1470, in write_cache
  File "mypy\nodes.py", line 352, in serialize
  File "mypy\nodes.py", line 3435, in serialize
  File "mypy\nodes.py", line 3372, in serialize
  File "mypy\nodes.py", line 761, in serialize
  File "mypy\types.py", line 1797, in serialize
  File "mypy\types.py", line 1801, in serialize
  File "mypy\types.py", line 647, in serialize
  File "mypy\types.py", line 1190, in serialize
  File "mypy\nodes.py", line 2742, in fullname
AttributeError: attribute 'TypeInfo' of '_fullname' undefined

meanwhile doing it like this

from my_typing_simple import Callable, ParamSpec, T
P = ParamSpec("P")

def my_decorator(folder:str) -> Callable[[Callable[P, T]], Callable[P, T]]:
    def decorator(func:Callable[P,T]) -> Callable[P,T]:
        def wrapper(*args:P.args,**kwarg:P.kwargs) -> T:
            print("wrapper")
            return func(*args,**kwarg)
        return wrapper
    return decorator

everything is ok

[ilevkivskyi]

Note this may be fixed by #13381 (you can check when the PR is merged)

[A5rocks]

I tried reproducing and it does seem to work now on mypy 0.982. Are people still able to reproduce this?