Crash when deserialising ParamSpec in ClassDef

[Fogapod]

Bug Report

Mypy crashes with AssertionError after running it for the second time with namespace-packages option enabled. Looks like some typevar is improperly serialized in cache.
First time mypy runs without issues:

main.py:1: error: Cannot find implementation or library stub for module named "humanize"
main.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

Second time it crashes:
Traceback from dev version:

Traceback (most recent call last):
  File "/home/eugene/mypy_test/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/main.py", line 96, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/main.py", line 173, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/build.py", line 180, in build
    result = _build(
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/build.py", line 256, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/build.py", line 2727, in dispatch
    process_graph(graph, manager)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/build.py", line 3064, in process_graph
    process_fresh_modules(graph, prev_scc, manager)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/build.py", line 3139, in process_fresh_modules
    graph[id].load_tree()
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/build.py", line 1995, in load_tree
    self.tree = MypyFile.deserialize(data)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 356, in deserialize
    tree.names = SymbolTable.deserialize(data['names'])
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 3373, in deserialize
    st[key] = SymbolTableNode.deserialize(value)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 3314, in deserialize
    node = SymbolNode.deserialize(data['node'])
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 248, in deserialize
    return method(data)
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 2849, in deserialize
    defn = ClassDef.deserialize(data['defn'])
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 1036, in deserialize
    [mypy.types.TypeVarType.deserialize(v) for v in data['type_vars']],
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 1036, in <listcomp>
    [mypy.types.TypeVarType.deserialize(v) for v in data['type_vars']],
  File "/home/eugene/mypy_test/.venv/lib/python3.8/site-packages/mypy/types.py", line 524, in deserialize
    assert data['.class'] == 'TypeVarType'
AssertionError

I added print(data) above the assert check, here's a few last lines before crash, last line causes it:

{'.class': 'TypeVarType', 'fullname': 'discord.ext.commands.core.CogT', 'id': 1, 'name': 'CogT', 'upper_bound': 'discord.ext.commands.cog.Cog', 'values': [], 'variance': 0}
{'.class': 'TypeVarType', 'fullname': 'discord.ext.commands.core.T', 'id': 3, 'name': 'T', 'upper_bound': 'builtins.object', 'values': [], 'variance': 0}
{'.class': 'TypeVarType', 'fullname': 'discord.ext.commands.core.CogT', 'id': 1, 'name': 'CogT', 'upper_bound': 'discord.ext.commands.cog.Cog', 'values': [], 'variance': 0}
{'.class': 'ParamSpecType', 'flavor': 0, 'fullname': 'discord.ext.commands.core.P', 'id': 2, 'name': 'P', 'upper_bound': 'builtins.object'}
Traceback (most recent call last):
...

So mypy doesn't like P typevar, it is defined here: https://github.com/Rapptz/discord.py/blob/45d498c1b76deaf3b394d17ccf56112fa691d160/discord/ext/commands/core.py#L111
Note that this is typing_extensions.ParamSpec, not typing.ParamSpec. patching library to use typing.ParamSpec produced same crash.

I am unsure what exactly causes error, so I can't produce a smaller example of this crash. I will try to narrow it down in further comments.

To Reproduce

0. create and activate venv
1. install mypy
2. pip install discord.py @ git+https://github.com/Rapptz/discord.py@45d498c1b76deaf3b394d17ccf56112fa691d160
3. mypy main.py --namespace-packages
4. mypy main.py --namespace-packages (run it again)

main.py

# does not matter if it is installed or no, but assertion error does not happen if this import is not there for some reason
import humanize

from discord.ext import commands

Expected Behavior

mypy to not crash.

Actual Behavior

mypy crashes with AssertionError.

Your Environment

• Mypy version used: both 0.931 and latest master 0.940+dev.feca706d5f2540003ae8b24009d56dac7c067eba
• Mypy command-line flags: --namespace-packages
• Mypy configuration options from mypy.ini (and other config files):
• Python version used: 3.8.12 / 3.10.2
• Operating system and version: Arch Linux
• pip freeze (new venv, after installing problematic package):

aiohttp==3.7.4.post0
async-timeout==3.0.1
attrs==21.4.0
chardet==4.0.0
discord.py==2.0.0a3575+g45d498c1
idna==3.3
multidict==6.0.2
mypy @ git+https://github.com/python/mypy.git@feca706d5f2540003ae8b24009d56dac7c067eba
mypy-extensions==0.4.3
tomli==2.0.1
typing-extensions==4.1.1
yarl==1.7.2

[hauntsaninja]

Thanks, looks like a pretty straightforward bug in ParamSpec deserialisation.

mypy/mypy/nodes.py

Line 1036 in 14de8c6

[mypy.types.TypeVarType.deserialize(v) for v in data['type_vars']],

(Note that mypy doesn't yet have great support for classes that are generic over ParamSpec, but obviously it can do much better than this)

[A5rocks]

Spotted in #11847 but that will probably not be merged in in time for v0.940 (and so a seperate PR for the fix might be preferred).

[Fogapod]

I tried just selecting ParamSpec or TypeVar depending on class name but that doesn't seem to work. I am unsure how to fix this

    @classmethod
    def deserialize(self, data: JsonDict) -> 'ClassDef':
        assert data['.class'] == 'ClassDef'
        type_to_class = {
            'TypeVarType': mypy.types.TypeVarType,
            'ParamSpecType': mypy.types.ParamSpecType,
        }
        res = ClassDef(data['name'],
                       Block([]),
                       [type_to_class[v['.class']].deserialize(v) for v in data['type_vars']],
                       )
        res.fullname = data['fullname']
        return res

Edit: actually, this resolves issue in provided example, but second run of mypy causes a different error in original codebase. Unsure if this is the result of wrong fix or something unrelated.

version: 0.940+dev.feca706d5f2540003ae8b24009d56dac7c067eba
Traceback (most recent call last):
  File "/home/eugene/src/stella_bot/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/main.py", line 96, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/main.py", line 173, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/build.py", line 180, in build
    result = _build(
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/build.py", line 256, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/build.py", line 2727, in dispatch
    process_graph(graph, manager)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/build.py", line 3071, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/build.py", line 3178, in process_stale_scc
    if not graph[id].type_check_second_pass():
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/build.py", line 2205, in type_check_second_pass
    return self.type_checker().check_second_pass()
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 368, in check_second_pass
    self.check_partial(node)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 379, in check_partial
    self.accept(node)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 424, in accept
    stmt.accept(self)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 738, in accept
    return visitor.visit_func_def(self)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 774, in visit_func_def
    self._visit_func_def(defn)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 785, in _visit_func_def
    self.check_method_override(defn)
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 1486, in check_method_override
    if self.check_method_or_accessor_override_for_base(defn, base):
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 1514, in check_method_or_accessor_override_for_base
    if self.check_method_override_for_base_with_name(defn, name, base):
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 1600, in check_method_override_for_base_with_name
    self.check_override(typ,
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/checker.py", line 1762, in check_override
    self.msg.signature_incompatible_with_supertype(
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/messages.py", line 848, in signature_incompatible_with_supertype
    self.pretty_callable_or_overload(original, context, offset=ALIGN_OFFSET + 2 * OFFSET,
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/messages.py", line 870, in pretty_callable_or_overload
    self.note(pretty_callable(tp), context,
  File "/home/eugene/src/stella_bot/.venv/lib/python3.8/site-packages/mypy/messages.py", line 1954, in pretty_callable
    definition_args = [arg.variable.name for arg in tp.definition.arguments]
AttributeError: arguments

[cdce8p]

@Fogapod Take a look at the solution in #11847. That should would.
https://github.com/python/mypy/pull/11847/files#diff-d17e62a76795c1a5e75fb9bfe726570f0963e1a8e0430cb78fc6d069a7b46e68R1036-R1038

        res = ClassDef(data['name'],
                        Block([]),
                        # https://github.com/python/mypy/issues/12257
                        [cast(mypy.types.TypeVarLikeType, mypy.types.deserialize_type(v))
                         for v in data['type_vars']],
                        )

[Fogapod]

That branch produces same error for me. So must be a separate issue

[cdce8p]

In that case it's likely because ParamSpec variables in Generic classes aren't supported yet to begin with. #11847 will add support and already includes the fix. Let's just go with that then.