dmypy crashes from RecursionError for some complex expressions (fastparse)

[kcdodd]

Crash Report

I have been unable to use dmypy for a while due to recursion error, but finally took the time to figure out why. This seems to happen when processing a dependency, SymPy source file+line: https://github.com/sympy/sympy/blob/e676df8ded885babe62bd31bec5db8c96b7480e7/sympy/polys/numberfields/resolvent_lookup.py#L248.
It seems that the AST of the expression is just simply too deep to process before hitting recursion limit, estimating its a couple thousand nodes deep.

Honestly, I wouldn't consider this a bug, except that it crashes the daemon and prevents it from processing anything else. The ast module clearly warns "It is possible to crash the Python interpreter with a sufficiently large/complex string due to stack depth limitations in Python’s AST compiler.", so it might even be close to that limit as well.

But if there was a way of simply not processing such large nodes, or recovering, I think that would deserve a "fix".

Traceback

version: 1.12.0+dev.fe15ee69b9225f808f8ed735671b73c31ae1bed8
Daemon crashed!
Traceback (most recent call last):
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 236, in serve
    resp = self.run_command(command, data)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 285, in run_command
    ret = method(self, **data)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 353, in cmd_run
    return self.check(sources, export_types, is_tty, terminal_width)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 427, in check
    res = self.initialize_fine_grained(sources, is_tty, terminal_width)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/dmypy_server.py", line 466, in initialize_fine_grained
    result = mypy.build.build(sources=sources, options=self.options, fscache=self.fscache)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 193, in build
    result = _build(
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 268, in _build
    graph = dispatch(sources, manager, stdout)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 2897, in dispatch
    graph = load_graph(sources, manager)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 3083, in load_graph
    st = State(
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 2000, in __init__
    self.parse_file(temporary=temporary)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 2176, in parse_file
    self.tree = manager.parse_file(
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/build.py", line 841, in parse_file
    tree = parse(source, path, id, self.errors, options=options)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/parse.py", line 27, in parse
    tree = mypy.fastparse.parse(source, fnam=fnam, module=module, errors=errors, options=options)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 242, in parse
    ).visit(ast)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 876, in visit_Module
    body = self.fix_function_overloads(self.translate_stmt_list(mod.body, ismodule=True))
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 491, in translate_stmt_list
    node = self.visit(stmt)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 889, in visit_FunctionDef
    return self.do_func_def(n)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1023, in do_func_def
    body = self.as_required_block(n.body, can_strip=True, is_coroutine=is_coroutine)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 606, in as_required_block
    self.translate_stmt_list(stmts, can_strip=can_strip, is_coroutine=is_coroutine)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 491, in translate_stmt_list
    node = self.visit(stmt)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1443, in visit_Expr
    value = self.visit(n.value)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1497, in visit_BinOp
    e = OpExpr(op, self.visit(n.left), self.visit(n.right))
...
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1497, in visit_BinOp
    e = OpExpr(op, self.visit(n.left), self.visit(n.right))
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 417, in visit
    return visitor(node)
  File "/media/box/projects/moebius-n2212/v310/lib/python3.10/site-packages/mypy/fastparse.py", line 1492, in visit_BinOp
    op = self.from_operator(n.op)
RecursionError: maximum recursion depth exceeded

To Reproduce

Run dymy on https://github.com/sympy/sympy/blob/e676df8ded885babe62bd31bec5db8c96b7480e7/sympy/polys/numberfields/resolvent_lookup.py.

Your Environment

• Mypy version used: dmypy 1.12.0+dev.fe15ee69b9225f808f8ed735671b73c31ae1bed8
• Mypy command-line flags: dmypy run -- sympy/polys/numberfields/resolvent_lookup.py
• Python version used: 3.10

[kcdodd]

As a small update, I was wondering why the non-daemon mypy command did not crash even though it should be going through the same analysis. Apparently mypy.main sets the recursion limit to 2**14 (or 16384) but mypy.dmypy leaves the default limit (1000 in this case).

I have linked a pull request that makes these consistent, which might address this by itself. But it also includes a proposed method of handling such expressions, even though its probably up for debate what the proper handling of it should be.