Crash with "Should never get here in normal mode" with incremental=True

[lordmauve]

I was working in a file, adding types and replacing attrs with typing.NamedTuple in order to benefit from types, when I got this crash:

$ bin/mypy python/
Traceback (most recent call last):
  File "bin/mypy", line 33, in <module>
    sys.exit(console_entry())
  File "dist/lib/python3.6/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "dist/lib/python3.6/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "dist/lib/python3.6/mypy/main.py", line 97, in type_check_only
    options=options)
  File "dist/lib/python3.6/mypy/build.py", line 196, in build
    graph = dispatch(sources, manager)
  File "dist/lib/python3.6/mypy/build.py", line 1801, in dispatch
    process_graph(graph, manager)
  File "dist/lib/python3.6/mypy/build.py", line 2037, in process_graph
    process_fresh_scc(graph, prev_scc)
  File "dist/lib/python3.6/mypy/build.py", line 2106, in process_fresh_scc
    graph[id].fix_cross_refs()
  File "dist/lib/python3.6/mypy/build.py", line 1547, in fix_cross_refs
    self.manager.options.quick_and_dirty)
  File "dist/lib/python3.6/mypy/fixup.py", line 22, in fixup_module_pass_one
    node_fixer.visit_symbol_table(tree.names)
  File "dist/lib/python3.6/mypy/fixup.py", line 105, in visit_symbol_table
    self.visit_type_info(value.node)
  File "dist/lib/python3.6/mypy/fixup.py", line 62, in visit_type_info
    base.accept(self.type_fixer)
  File "dist/lib/python3.6/mypy/types.py", line 435, in accept
    return visitor.visit_instance(self)
  File "dist/lib/python3.6/mypy/fixup.py", line 176, in visit_instance
    assert self.quick_and_dirty, "Should never get here in normal mode"
AssertionError: Should never get here in normal mode

My mypi.ini contained

[mypy]
follow_imports = silent
incremental = True

However the error went away when I removed incremental = True.

[gvanrossum]

Thanks for the report -- if you can still repro this, is your source code public, and can you save a tarball of your .mypy_cache?

[lordmauve]

I'm afraid it's not public.

[mbforbes]

I'm also seeing this bug. (I also have another, possibly related bug, which I'll report in a separate comment below.)

Disclaimer: I'm pointing MYPYPATH to site-packages/ (many details in #3987), which I believe is unsupported.

I've made my code open source at mbforbes/rndjam1 @ 4714532. If you're trying to reproduce this and are having trouble running it, feel free to reply to me here, open an issue there, etc.

This error only occurs for me the second time I run mypy with incremental mode, which I believe means it has to build the cache first before this problem happens. I need to introduce a type error for it to happen. And if I comment out one of my local imports (dataio in my code), the error goes away.

For context:

$ rm -rf .mypy_cache/

$ mypy --version
mypy 0.530-dev-c332ea08b542728985fadcbee1db6651060c3876

$ cat mypy.ini 
[mypy]
mypy_path=/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/

Then:

$ mypy --ignore-missing-imports --follow-imports=silent --incremental regression.py 
regression.py:206: error: Argument 1 to "least_squares" has incompatible type "int"; expected "FloatTensor"

$ mypy --ignore-missing-imports --follow-imports=silent --incremental regression.py 
Traceback (most recent call last):
  File "/home/max/.pyenv/versions/rndj1/bin/mypy", line 11, in <module>
    load_entry_point('mypy===0.530-dev-c332ea08b542728985fadcbee1db6651060c3876', 'console_scripts', 'mypy')()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 198, in build
    graph = dispatch(sources, manager)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 1838, in dispatch
    process_graph(graph, manager)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 2081, in process_graph
    process_fresh_scc(graph, prev_scc)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 2150, in process_fresh_scc
    graph[id].fix_cross_refs()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 1576, in fix_cross_refs
    self.manager.options.quick_and_dirty)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/fixup.py", line 22, in fixup_module_pass_one
    node_fixer.visit_symbol_table(tree.names)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/fixup.py", line 105, in visit_symbol_table
    self.visit_type_info(value.node)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/fixup.py", line 62, in visit_type_info
    base.accept(self.type_fixer)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 476, in accept
    return visitor.visit_instance(self)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/fixup.py", line 176, in visit_instance
    assert self.quick_and_dirty, "Should never get here in normal mode"
AssertionError: Should never get here in normal mode

If I keep running the same command, I keep getting the error.

I've attached the result of tar -czvf mpc.tar.gz .mypy_cache/. mpc.tar.gz

(Edit: linked to specific commit in repository.)

[mbforbes]

With exactly the same setup as above, I get a different error if I run with --quick-and-dirty instead of --incremental:

$ mypy --ignore-missing-imports --follow-imports=silent --quick-and-dirty regression.py 
regression.py:206: error: Argument 1 to "least_squares" has incompatible type "int"; expected "FloatTensor"

$ mypy --ignore-missing-imports --follow-imports=silent --quick-and-dirty regression.py 
Traceback (most recent call last):
  File "/home/max/.pyenv/versions/rndj1/bin/mypy", line 11, in <module>
    load_entry_point('mypy===0.530-dev-c332ea08b542728985fadcbee1db6651060c3876', 'console_scripts', 'mypy')()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 198, in build
    graph = dispatch(sources, manager)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 1838, in dispatch
    process_graph(graph, manager)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 2081, in process_graph
    process_fresh_scc(graph, prev_scc)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 2148, in process_fresh_scc
    graph[id].load_tree()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/build.py", line 1570, in load_tree
    self.tree = MypyFile.deserialize(data)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 249, in deserialize
    tree.names = SymbolTable.deserialize(data['names'])
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 2435, in deserialize
    st[key] = SymbolTableNode.deserialize(value)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 2382, in deserialize
    node = SymbolNode.deserialize(data['node'])
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 182, in deserialize
    return method(data)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 674, in deserialize
    type = None if data['type'] is None else mypy.types.deserialize_type(data['type'])
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 30, in deserialize_type
    return method(data)
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 1375, in deserialize
    return TypeType.make_normalized(deserialize_type(data['item']))
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 1356, in make_normalized
    return TypeType(item, line=line, column=column)  # type: ignore
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 1344, in __init__
    if isinstance(item, CallableType) and item.is_type_obj():
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/types.py", line 721, in is_type_obj
    return t is not None and t.is_metaclass()
  File "/home/max/.pyenv/versions/rndj1/lib/python3.6/site-packages/mypy/nodes.py", line 2250, in __getattribute__
    raise AssertionError('De-serialization failure: TypeInfo not fixed')
AssertionError: De-serialization failure: TypeInfo not fixed

[ilevkivskyi]

@mbforbes Interesting, this points to some familiar places in code. I will take care of this.

[ilevkivskyi]

@gvanrossum I have a simple repro inspired by #4043 for the first crash here as well (the second is unrelated and will be fixed by #4038). Here is the recipe:

Initial setup:

# a.py
from b import x

# b.py
from c import C
x: C

# c.py
class C: pass

with any previous mypy caches removed. Steps to reproduce:

• run mypy a.py
• replace the content of c.py with def C(): pass
• run mypy -i a.py (it will show an obvious error)
• add random whitespace to a.py
• run mypy -i a.py again -- AssertionError: Should never get here in normal mode

[gvanrossum]

Interestingly, this one was not (directly) introduced by #2014. Bisecting points to #3304 (Tighten FakeInfo and fix crashes in --quick mode; rev d2638b1).

[gvanrossum]

the second is unrelated and will be fixed by #4038

This refers to the crash reporting AssertionError: De-serialization failure: TypeInfo not fixed, right?

[ilevkivskyi]

the second is unrelated and will be fixed by #4038

This refers to the crash reporting AssertionError: De-serialization failure: TypeInfo not fixed, right?

Yes.