Crash involving dataclasses, generics, forward references

[NeilGirdhar]

Running master MyPy 0.960 on the latest efax 1.6.3 gives

❯ git clone git@github.com:NeilGirdhar/efax.git
❯ cd efax
❯ mypy efax --show-traceback
efax/_src/distributions/dirichlet.py:29: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.ee78afe325800f9266ccc0d143b49b8c0b29d8a3
Traceback (most recent call last):
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/main.py", line 96, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/main.py", line 173, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 180, in build
    result = _build(
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 256, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 2733, in dispatch
    process_graph(graph, manager)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 3081, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 3173, in process_stale_scc
    mypy.semanal_main.semantic_analysis_for_scc(graph, scc, manager.errors)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal_main.py", line 78, in semantic_analysis_for_scc
    process_top_levels(graph, scc, patches)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal_main.py", line 199, in process_top_levels
    deferred, incomplete, progress = semantic_analyze_target(next_id, state,
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal_main.py", line 326, in semantic_analyze_target
    analyzer.refresh_partial(refresh_node,
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 414, in refresh_partial
    self.refresh_top_level(node)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 425, in refresh_top_level
    self.accept(d)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 5340, in accept
    node.accept(self)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/nodes.py", line 1028, in accept
    return visitor.visit_class_def(self)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 1121, in visit_class_def
    self.analyze_class(defn)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 1201, in analyze_class
    self.analyze_class_body_common(defn)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 1210, in analyze_class_body_common
    self.apply_class_plugin_hooks(defn)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 1256, in apply_class_plugin_hooks
    hook(ClassDefContext(defn, decorator, self))
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/plugins/dataclasses.py", line 524, in dataclass_class_maker_callback
    transformer.transform()
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/plugins/dataclasses.py", line 120, in transform
    attributes = self.collect_attributes()
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/plugins/dataclasses.py", line 408, in collect_attributes
    attr.expand_typevar_from_subtype(ctx.cls.info)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/plugins/dataclasses.py", line 106, in expand_typevar_from_subtype
    self.type = map_type_from_supertype(self.type, sub_type, self.info)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/typeops.py", line 151, in map_type_from_supertype
    inst_type = map_instance_to_supertype(inst_type, super_info)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/maptype.py", line 24, in map_instance_to_supertype
    return map_instance_to_supertypes(instance, superclass)[0]
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/maptype.py", line 37, in map_instance_to_supertypes
    a.extend(map_instance_to_direct_supertypes(t, sup))
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/maptype.py", line 82, in map_instance_to_direct_supertypes
    t = expand_type(b, env)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/expandtype.py", line 18, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/types.py", line 1168, in accept
    return visitor.visit_instance(self)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/expandtype.py", line 90, in visit_instance
    args = self.expand_types(t.args)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/expandtype.py", line 215, in expand_types
    a.append(t.accept(self))
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/types.py", line 2513, in accept
    assert isinstance(visitor, SyntheticTypeVisitor)
AssertionError: 
efax/_src/distributions/dirichlet.py:29: : note: use --pdb to drop into pdb

[NeilGirdhar]

I briefly tried to make an MWE, but was unsuccessful.

[hauntsaninja]

Looks like this is a regression between 0.942 and 0.950

[hauntsaninja]

This bisects to #12656

[hauntsaninja]

Okay, took another look at this and got a small repro:

from dataclasses import dataclass
from typing import Generic, TypeVar

T = TypeVar("T")

@dataclass
class Parent(Generic[T]):
    key: str

@dataclass
class Child1(Parent["Child2"]): ...

@dataclass
class Child2(Parent["Child1"]): ...

[NYKevin]

You can also repro it with just one class, which is probably still tickling the same bug:

import dataclasses
import typing

T = typing.TypeVar('T', bound='Foo')

@dataclasses.dataclass
class Foo(typing.Generic[T]):
	data: int

If you remove either of the following elements, the problem won't be reproducible:

• The data field.
• The bound on T.

The bound need not be Foo itself, as long as it is a necessary forward reference - that is, a forward reference to a class defined below T (and therefore "necessary" because, if you removed the quotes, it would fail with a NameError). If you put e.g. 'object', or the name of a class which already exists, then it won't reproduce. But you can just put something like class Bar: pass below the declaration of T, and then bound='Bar' will reproduce the bug.

The traceback is also slightly different:

example.py:8: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.612074f05ba5d65e016c031dc94b13f962d9c6b4
Traceback (most recent call last):
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Kevin\poker\mypy-testing\Scripts\mypy.exe\__main__.py", line 7, in <module>
    sys.exit(console_entry())
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\main.py", line 96, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\main.py", line 173, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 181, in build
    result = _build(
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 257, in _build
    graph = dispatch(sources, manager, stdout)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 2752, in dispatch
    process_graph(graph, manager)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 3110, in process_graph
    process_stale_scc(graph, scc, manager)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 3202, in process_stale_scc
    mypy.semanal_main.semantic_analysis_for_scc(graph, scc, manager.errors)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal_main.py", line 79, in semantic_analysis_for_scc
    process_functions(graph, scc, patches)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal_main.py", line 233, in process_functions
    process_top_level_function(analyzer,
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal_main.py", line 274, in process_top_level_function
    deferred, incomplete, progress = semantic_analyze_target(target, state, node, active_type,
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal_main.py", line 326, in semantic_analyze_target
    analyzer.refresh_partial(refresh_node,
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 417, in refresh_partial
    self.accept(node)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 5344, in accept
    node.accept(self)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\nodes.py", line 747, in accept
    return visitor.visit_func_def(self)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 637, in visit_func_def
    self.analyze_func_def(defn)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 669, in analyze_func_def
    self.defer(defn)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 5044, in defer
    assert not self.final_iteration, 'Must not defer during final iteration'
AssertionError: Must not defer during final iteration
example.py:8: : note: use --pdb to drop into pdb

[hauntsaninja]

Thanks. I think that's actually a different issue (and one that is not a regression), see #12527

[JukkaL]

The root cause seems to be that we can't safely run the dataclass transform in the main pass of the semantic analysis, since various references can still be unbound, and we can encounter placeholder types, etc. I'm going to attempt to move the whole transform to a later pass where all references are bound.