File shadowing with --follow-imports=skip can cause crashes in incremental and daemon mode

[ilevkivskyi]

Imagine the following scenario: One uses --follow-imports=skip and then adds a file some_dir/some_file.py to the build. If there is a file in the some_dir called e.g. types.py that is not in the build, then this will cause actual typeshed types.pyi being silently kicked out of the build.

Although such behavior is IMO counterintuitive, it is technically correct (according to the current import resolution specs). The problem however is that this can cause crashes. I was not able to minimize the repros, but here are two tracebacks. One from the normal incremental mode:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "mypy/main.py", line 93, in main
  File "mypy/build.py", line 167, in build
  File "mypy/build.py", line 227, in _build
  File "mypy/build.py", line 2565, in dispatch
  File "mypy/build.py", line 2871, in process_graph
  File "mypy/build.py", line 2949, in process_fresh_modules
  File "mypy/build.py", line 1862, in fix_cross_refs
  File "mypy/fixup.py", line 25, in fixup_module
  File "mypy/fixup.py", line 90, in visit_symbol_table
  File "mypy/nodes.py", line 654, in accept__Node_glue
  File "mypy/nodes.py", line 655, in accept
  File "mypy/fixup.py", line 92, in visit_func_def__StatementVisitor_glue
  File "mypy/fixup.py", line 96, in visit_func_def
  File "mypy/types.py", line 959, in accept
  File "mypy/fixup.py", line 164, in visit_callable_type__TypeVisitor_glue
  File "mypy/fixup.py", line 170, in visit_callable_type
  File "mypy/types.py", line 650, in accept
  File "mypy/fixup.py", line 144, in visit_instance__TypeVisitor_glue
  File "mypy/fixup.py", line 157, in visit_instance
  File "mypy/types.py", line 1644, in accept
  File "mypy/fixup.py", line 226, in visit_union_type__TypeVisitor_glue
  File "mypy/fixup.py", line 229, in visit_union_type
  File "mypy/types.py", line 650, in accept
  File "mypy/fixup.py", line 144, in visit_instance__TypeVisitor_glue
  File "mypy/fixup.py", line 150, in visit_instance
  File "mypy/fixup.py", line 240, in lookup_qualified_typeinfo
  File "mypy/fixup.py", line 254, in lookup_qualified
  File "mypy/fixup.py", line 263, in lookup_qualified_stnode
  File "mypy/lookup.py", line 29, in lookup_fully_qualified
AssertionError: Cannot find module for types.TracebackType

and another from the fine-grained incremental mode

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "mypy/checkexpr.py", line 3263, in accept
  File "mypy/nodes.py", line 1470, in accept
  File "mypy/checkexpr.py", line 1752, in visit_member_expr__ExpressionVisitor_glue
  File "mypy/checkexpr.py", line 1755, in visit_member_expr
  File "mypy/checkexpr.py", line 1763, in analyze_ordinary_member_access
  File "mypy/checkexpr.py", line 177, in analyze_ref_expr
  File "mypy/checkmember.py", line 783, in type_object_type
ValueError: <TypeInfo logging.Formatter> is not in list

[gvanrossum]

I guess a workaround would be to add a __init__.py to some_dir. Is that acceptable?

[ilevkivskyi]

Yes, I think an __init__.py[i] should fix this. However, mypy shouldn't crash and instead try to give a reasonable error message. Also the crashes are hard to diagnose (especially the second one, since nothing points to what went wrong). Possible fix may be similar to #7459

[JukkaL]

One possible fix would be to define a set of "core" library modules that will always be looked up from typeshed. If any module outside typeshed has a name conflict with a core module, mypy would generate a blocking error and ignore the module. The error message should be accompanied with notes indicating common workarounds, such as renaming the module or adding a missing __init__.py file.

[emmatyping]

I think this is fixed by #13155? We should give a blocking error if this occurs.

[ilevkivskyi]

This may or may not be fixed, because this behavior is quite subtle. We will need to verify if it still happens (we don't have a simple repro unfortunately).