Debugging a segmentation fault

[jace]

Bug Report

Some code I was working on last week is causing mypy to segfault. I tried retracing my steps to find the change that broke mypy, without luck.

Config:

• Platform: macOS 12.4 on M1
• Arch: arm64
• Python: 3.9.13
• mypy: 0.971
• mypy plugins: none (segfault happens with all plugins disabled and is not linked to any imported modules)

mypy --tb and mypy --pdb are segfaulting without any info. gdb is not available on arm64. I get a bit more info using faulthandler, but all it indicates is the file:

$ python -X faulthandler -m mypy -v .
…
LOG:  Processing SCC singleton (coaster.views.classview) as inherently stale with stale deps (__future__ asyncio builtins coaster coaster.auth coaster.sqlalchemy coaster.typing coaster.utils coaster.views.misc flask flask.blueprints flask.typing functools sqlalchemy.orm.attributes sqlalchemy.orm.descriptor_props sqlalchemy.orm.mapper sqlalchemy.orm.properties sqlalchemy.orm.query types typing typing_extensions werkzeug.local werkzeug.routing werkzeug.wrappers)
Fatal Python error: Segmentation fault

Current thread 0x0000000102f10580 (most recent call first):
  File "/Users/jace/.virtualenvs/hasgeek/lib/python3.9/site-packages/mypy/__main__.py", line 12 in console_entry
  File "/Users/jace/.virtualenvs/hasgeek/lib/python3.9/site-packages/mypy/__main__.py", line 34 in <module>
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87 in _run_code
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197 in _run_module_as_main
[1]    44130 segmentation fault  python -X faulthandler -m mypy -v .

The file in question is located at https://github.com/hasgeek/coaster/blob/369ce391f11850f10fc0ab42a52c196188436a50/coaster/views/classview.py

To Reproduce

1. git clone https://github.com/hasgeek/coaster.git
2. cd coaster
3. git checkout 369ce391f11850f10fc0ab42a52c196188436a50
4. mypy coaster/views/classview.py

[hauntsaninja]

Thanks, I needed to disable the "sqlmypy" plugin in order to be able to repro.

When running with uncompiled mypy (using latest master), I get the following stacktrace:

coaster/views/classview.py:193: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.980+dev.23ee1e7aff357e656e3102435ad0fe3b5074571e
Traceback (most recent call last):
  File "/Users/shantanu/.virtualenvs/mypy-qyes/bin/mypy", line 33, in <module>
    sys.exit(load_entry_point('mypy', 'console_scripts', 'mypy')())
  File "/Users/shantanu/dev/mypy/mypy/__main__.py", line 15, in console_entry
    main()
  File "/Users/shantanu/dev/mypy/mypy/main.py", line 95, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/Users/shantanu/dev/mypy/mypy/main.py", line 174, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 187, in build
    result = _build(
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 271, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2874, in dispatch
    process_graph(graph, manager)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3258, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3359, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2302, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 465, in check_first_pass
    self.accept(d)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1127, in accept
    return visitor.visit_class_def(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 2004, in visit_class_def
    self.accept(defn.defs)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1202, in accept
    return visitor.visit_block(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 2384, in visit_block
    self.accept(s)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 811, in accept
    return visitor.visit_func_def(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 935, in visit_func_def
    self._visit_func_def(defn)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 939, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 1005, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 1189, in check_func_def
    self.accept(item.body)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1202, in accept
    return visitor.visit_block(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 2384, in visit_block
    self.accept(s)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1386, in accept
    return visitor.visit_return_stmt(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 3861, in visit_return_stmt
    self.check_return_stmt(s)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 3895, in check_return_stmt
    self.expr_checker.accept(
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 4615, in accept
    typ = node.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1822, in accept
    return visitor.visit_call_expr(self)
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 385, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 505, in visit_call_expr_inner
    ret_type = self.check_call_expr_with_callee_type(
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 1155, in check_call_expr_with_callee_type
    ret_type, callee_type = self.check_call(
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 1290, in check_call
    return self.check_call(
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 1257, in check_call
    call_function = analyze_member_access(
  File "/Users/shantanu/dev/mypy/mypy/checkmember.py", line 187, in analyze_member_access
    result = _analyze_member_access(name, typ, mx, override_info)
  File "/Users/shantanu/dev/mypy/mypy/checkmember.py", line 206, in _analyze_member_access
    return analyze_instance_member_access(name, typ, mx, override_info)
  File "/Users/shantanu/dev/mypy/mypy/checkmember.py", line 304, in analyze_instance_member_access
    signature = freshen_function_type_vars(signature)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 101, in freshen_function_type_vars
    fresh = cast(CallableType, expand_type(callee, tvmap)).copy_modified(variables=tvs)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1743, in accept
    return visitor.visit_callable_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 254, in visit_callable_type
    arg_types=self.expand_types(t.arg_types),
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 345, in expand_types
    a.append(t.accept(self))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 2509, in accept
    return visitor.visit_union_type(self)
    
  ... 32000 lines elided ...

  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 325, in visit_union_type
    return make_simplified_union(self.expand_types(t.items), t.line, t.column)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 454, in make_simplified_union
    simplified_set: Sequence[Type] = _remove_redundant_union_items(items, keep_erased)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 517, in _remove_redundant_union_items
    if is_redundant_literal_instance(proper_item, proper_tj) and is_proper_subtype(
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 208, in is_proper_subtype
    return _is_subtype(left, right, subtype_context, proper_subtype=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 306, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1256, in accept
    return visitor.visit_instance(self)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 557, in visit_instance
    call = find_member("__call__", left, left, is_operator=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 996, in find_member
    return find_node_type(method, itype, subtype)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 1088, in find_node_type
    signature = bind_self(
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in bind_self
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in <listcomp>
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 318, in expand
    return expand_type(target, {id: to_apply[all_ids.index(id)] for id in ids})
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 2509, in accept
    return visitor.visit_union_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 325, in visit_union_type
    return make_simplified_union(self.expand_types(t.items), t.line, t.column)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 454, in make_simplified_union
    simplified_set: Sequence[Type] = _remove_redundant_union_items(items, keep_erased)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 517, in _remove_redundant_union_items
    if is_redundant_literal_instance(proper_item, proper_tj) and is_proper_subtype(
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 208, in is_proper_subtype
    return _is_subtype(left, right, subtype_context, proper_subtype=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 306, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1256, in accept
    return visitor.visit_instance(self)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 557, in visit_instance
    call = find_member("__call__", left, left, is_operator=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 996, in find_member
    return find_node_type(method, itype, subtype)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 1088, in find_node_type
    signature = bind_self(
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in bind_self
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in <listcomp>
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 318, in expand
    return expand_type(target, {id: to_apply[all_ids.index(id)] for id in ids})
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 2509, in accept
    return visitor.visit_union_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 325, in visit_union_type
    return make_simplified_union(self.expand_types(t.items), t.line, t.column)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 454, in make_simplified_union
    simplified_set: Sequence[Type] = _remove_redundant_union_items(items, keep_erased)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 517, in _remove_redundant_union_items
    if is_redundant_literal_instance(proper_item, proper_tj) and is_proper_subtype(
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 208, in is_proper_subtype
    return _is_subtype(left, right, subtype_context, proper_subtype=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 306, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1256, in accept
    return visitor.visit_instance(self)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 557, in visit_instance
    call = find_member("__call__", left, left, is_operator=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 996, in find_member
    return find_node_type(method, itype, subtype)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 1088, in find_node_type
    signature = bind_self(
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in bind_self
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in <listcomp>
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 318, in expand
    return expand_type(target, {id: to_apply[all_ids.index(id)] for id in ids})
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 2509, in accept
    return visitor.visit_union_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 325, in visit_union_type
    return make_simplified_union(self.expand_types(t.items), t.line, t.column)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 345, in expand_types
    a.append(t.accept(self))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1743, in accept
    return visitor.visit_callable_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 253, in visit_callable_type
    return t.copy_modified(
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1678, in copy_modified
    return CallableType(
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1617, in __init__
    super().__init__(line, column)
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1348, in __init__
    super().__init__(line, column)
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 207, in __init__
    super().__init__(line, column)
RecursionError: maximum recursion depth exceeded while calling a Python object
coaster/views/classview.py:193: : note: use --pdb to drop into pdb

[ilevkivskyi]

Probably another example of issue caused by expand_type() calling make_simplified_union(). I was recently thinking about this but still don't have a good solution.

[jace]

Is there a particular type declaration in my code that's causing this? Type checking is broken downstream for me because of this one file. I have to test my client apps with the import removed from env (using a pre-commit container), which sucks because there's no type checking against this library's exports.

[hauntsaninja]

You can avoid the crash by doing:

diff --git a/coaster/views/classview.py b/coaster/views/classview.py
index 34e7e47..13e4e5c 100644
--- a/coaster/views/classview.py
+++ b/coaster/views/classview.py
@@ -183,6 +183,7 @@ class ViewHandler(  # pylint: disable=too-many-instance-attributes
         self.requires_roles = requires_roles or {}
         self.endpoints = set()
 
+    @t.no_type_check
     def reroute(self: ViewHandlerType, f: t.Callable) -> ViewHandlerType:
         """Replace a view handler in a subclass while keeping its URL route rules."""
         # Use type(self) instead of ViewHandler so this works for (future) subclasses
@@ -206,6 +207,7 @@ class ViewHandler(  # pylint: disable=too-many-instance-attributes
         r.endpoints = set()
         return r
 
+    @t.no_type_check
     def __call__(
         self: ViewHandlerType, decorated: t.Union[t.Callable, ViewHandlerType]
     ) -> ViewHandlerType:

Didn't yet look into minimising it.

You can also get mypy to avoid processing a single module by doing something like the following in your mypy config:

[mypy-coaster.views.classview]
follow_imports = skip

[ilevkivskyi]

A simple repro test case is

[case testCallableUnionCB]
from typing import Union, Callable, TypeVar

TA = TypeVar("TA", bound=A)
class A:
    def __call__(self: TA, other: Union[Callable, TA]) -> TA: ...
a: A
a()

Actually, looking at this, the bug is that we never push to assumptions stack for callback subtyping, while we should. I am not sure what is the best way to do this without a visible performance impact, maybe we can just push them locally in few places where __call__ is special-cased for subtyping/inference (also there may be an additional complication that we may need to "anonimize" generic callables before hashing them, not sure if this is needed).

[erictraut]

I'm not able to repro a crash with the latest version of mypy (1.5). I suspect it was fixed somewhere along the way.

[hauntsaninja]

Fixed in #14178, but I think worth adding Ivan's test case