assert file not in self.flushed_files (without MYPY_PATH)

[chriselion]

• Are you reporting a bug, or opening a feature request?

Bug

• Please insert below the code you are checking with mypy, or a mock-up repro if the source is private. We would appreciate if you try to simplify your case to a minimal repro.

mypy_repro.tar.gz
This is taken from https://github.com/Unity-Technologies/ml-agents/tree/develop - there are a lot of other large files in there, so this is only the relevant python files.

• What is the actual behavior/output?

$ mypy --ignore-missing-imports --disallow-incomplete-defs --namespace-packages --show-traceback --cache-dir=/dev/null ml-agents/setup.py  ml-agents/mlagents/trainers/components/reward_signals/curiosity/model.py 
ml-agents/mlagents/trainers/components/reward_signals/gail/model.py:223: error: "LearningModel" has no attribute "selected_actions"
ml-agents/mlagents/trainers/components/reward_signals/gail/model.py:242: error: "LearningModel" has no attribute "selected_actions"
ml-agents/mlagents/trainers/components/bc/model.py:57: error: "LearningModel" has no attribute "output"
ml-agents/mlagents/trainers/components/bc/model.py:63: error: "LearningModel" has no attribute "all_log_probs"
ml-agents/mlagents/trainers/components/reward_signals/curiosity/model.py:125: error: "LearningModel" has no attribute "selected_actions"
ml-agents/mlagents/trainers/components/reward_signals/curiosity/model.py:142: error: "LearningModel" has no attribute "selected_actions"
ml-agents/mlagents/trainers/components/reward_signals/curiosity/model.py:159: error: "LearningModel" has no attribute "selected_actions"
ml-agents/mlagents/trainers/components/reward_signals/curiosity/model.py:125: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.730+dev.28423cd78062b81365ec9aaa591b4e823af1606d
Traceback (most recent call last):
  File "/Users/chris.elion/code/mlagents_mypy/ml-agents/venv/bin/mypy", line 10, in <module>
    sys.exit(console_entry())
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/__main__.py", line 8, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/main.py", line 88, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/build.py", line 164, in build
    result = _build(sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/build.py", line 229, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/build.py", line 2566, in dispatch
    process_graph(graph, manager)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/build.py", line 2875, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/build.py", line 2974, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/build.py", line 2070, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 289, in check_first_pass
    self.accept(d)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 396, in accept
    stmt.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 927, in accept
    return visitor.visit_class_def(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 1671, in visit_class_def
    self.accept(defn.defs)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 396, in accept
    stmt.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 992, in accept
    return visitor.visit_block(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 1867, in visit_block
    self.accept(s)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 396, in accept
    stmt.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 668, in accept
    return visitor.visit_func_def(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 716, in visit_func_def
    self._visit_func_def(defn)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 720, in _visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 782, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 953, in check_func_def
    self.accept(item.body)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 396, in accept
    stmt.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 992, in accept
    return visitor.visit_block(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 1867, in visit_block
    self.accept(s)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 396, in accept
    stmt.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 1183, in accept
    return visitor.visit_if_stmt(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 2987, in visit_if_stmt
    self.accept(b)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 396, in accept
    stmt.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 992, in accept
    return visitor.visit_block(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 1867, in visit_block
    self.accept(s)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 396, in accept
    stmt.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 1050, in accept
    return visitor.visit_assignment_stmt(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 1901, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checker.py", line 2036, in check_assignment
    rvalue_type = self.expr_checker.accept(rvalue)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 3575, in accept
    typ = node.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 1532, in accept
    return visitor.visit_call_expr(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 263, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 337, in visit_call_expr_inner
    object_type, member)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 705, in check_call_expr_with_callee_type
    object_type=object_type)[0]
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 768, in check_call
    return self.check_any_type_call(args, callee)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 1833, in check_any_type_call
    self.infer_arg_types_in_empty_context(args)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 919, in infer_arg_types_in_empty_context
    arg_type = self.accept(arg)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 3575, in accept
    typ = node.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 1532, in accept
    return visitor.visit_call_expr(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 263, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 337, in visit_call_expr_inner
    object_type, member)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 705, in check_call_expr_with_callee_type
    object_type=object_type)[0]
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 768, in check_call
    return self.check_any_type_call(args, callee)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 1833, in check_any_type_call
    self.infer_arg_types_in_empty_context(args)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 919, in infer_arg_types_in_empty_context
    arg_type = self.accept(arg)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 3575, in accept
    typ = node.accept(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/nodes.py", line 1484, in accept
    return visitor.visit_member_expr(self)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 1859, in visit_member_expr
    result = self.analyze_ordinary_member_access(e, is_lvalue)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkexpr.py", line 1874, in analyze_ordinary_member_access
    in_literal_context=self.is_literal_context())
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkmember.py", line 117, in analyze_member_access
    result = _analyze_member_access(name, typ, mx, override_info)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkmember.py", line 134, in _analyze_member_access
    return analyze_instance_member_access(name, typ, mx, override_info)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkmember.py", line 210, in analyze_instance_member_access
    return analyze_member_var_access(name, typ, info, mx)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/checkmember.py", line 405, in analyze_member_var_access
    return mx.msg.has_no_attr(mx.original_type, itype, name, mx.context)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/messages.py", line 265, in has_no_attr
    code=codes.ATTR_DEFINED)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/messages.py", line 147, in fail
    self.report(msg, context, 'error', code=code, file=file, origin=origin)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/messages.py", line 137, in report
    code=code)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/errors.py", line 310, in report
    self.add_error_info(info)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/errors.py", line 341, in add_error_info
    self._add_error_info(file, info)
  File "/Users/chris.elion/code/ml-agents/venv/lib/python3.7/site-packages/mypy/errors.py", line 313, in _add_error_info
    assert file not in self.flushed_files
AssertionError: 

• What is the behavior/output you expect?

No assert (I still expect errors from my code though).

• What are the versions of mypy and Python you are using?

$ python --version
Python 3.7.3
$ mypy --version
mypy 0.730+dev.28423cd78062b81365ec9aaa591b4e823af1606d

This also happens using mypy==0.7.2

• What are the mypy flags you are using? (For example --strict-optional)

--ignore-missing-imports --disallow-incomplete-defs --namespace-packages --show-traceback --cache-dir=/dev/null
--cache-dir was just to simplify the repro

• If mypy crashed with a traceback, please paste the full traceback below.

See above

This seems similar to some other issues where this assert was hit (e.g. #4881, #6717) but in those, MYPY_PATH was set and here it's not.

[JukkaL]

I was able to reproduce the crash. If I comment out the assert, everything seems to work okay.

[ilevkivskyi]

Isn't this a duplicate of #4881?

[JukkaL]

Isn't this a duplicate of #4881?

Based on a quick investigation it wasn't clear to me whether the same fix would work for both issues. They are closely related at least. In this issue the key seems to be --namespace-packages, whereas #4881 doesn't use it but uses mypy_path.

The root cause seems to be the same -- mypy assigns multiple module names to a file, based on different package root paths.

[chriselion]

Thanks, glad the repro worked.

Given that this assert is the cause of a few issues, is it worth removing it (or maybe converting to a warning instead)?

[JukkaL]

Given that this assert is the cause of a few issues, is it worth removing it (or maybe converting to a warning instead)?

Yeah, maybe it should be turned into an error message. When we hit it something has gone wrong, but the crash is really unhelpful.

[chriselion]

OK, are you going to do that, or should I try it out?

[JukkaL]

@chriselion I'm going to be busy with other things for the next week and a half, so if you'd like to try it out, please go ahead!

[chriselion]

Hi @JukkaL,
Sorry, finally trying to get back to this. First attempt is in #7567, although I could use some guidance on where to add tests.

[chriselion]

@JukkaL Were you ever able to look into this further? My PR to disable the assert got some pushback, and I'm still not sure how to diagnose and resolve what's causing a file to get processed multiple times.

I'm currently running mypy without --namespace-packages but this means it's missing out on flagging some legitimate problems that need to be fixed.