AttributeError: 'UnionType' object has no attribute 'type'

[zero323]

This looks like a regression (?) between Mypy 0.501 and 0.510.

Failure is reported at this line: https://github.com/zero323/pyspark-stubs/blob/d25da4d051f878d358bdef6a0e08a6597afbee4f/third_party/3/pyspark/sql/functions.pyi#L24 but it can be reproduced with simplified input:

# Stubs for pyspark.sql.functions (Python 3.5)

from typing import Union
from pyspark.sql.column import Column

ColumnOrName = Union[pyspark.sql.column.Column, str]

def last(col: ColumnOrName, ignorenulls: bool = ...) -> Column: ...

Python version:

python --version
Python 3.5.3 :: Anaconda custom (64-bit)

Mypy version: 0.510, 0.511

Traceback:

third_party/3/pyspark/sql/column.pyi:24: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.511
Traceback (most recent call last):
  File "/home/user/anaconda3/bin/mypy", line 6, in <module>
    main(__file__)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/main.py", line 93, in type_check_only
    options=options)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/build.py", line 1595, in dispatch
    process_graph(graph, manager)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/build.py", line 1838, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/build.py", line 1937, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/build.py", line 1510, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 177, in check_first_pass
    self.accept(d)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/nodes.py", line 750, in accept
    return visitor.visit_class_def(self)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 1080, in visit_class_def
    self.accept(defn.defs)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/nodes.py", line 811, in accept
    return visitor.visit_block(self)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 1173, in visit_block
    self.accept(s)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/nodes.py", line 565, in accept
    return visitor.visit_func_def(self)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 510, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 569, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 616, in check_func_def
    self.check_reverse_op_method(item, typ, name)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 806, in check_reverse_op_method
    defn)
  File "/home/user/anaconda3/lib/python3.5/site-packages/mypy/checker.py", line 876, in check_overlapping_op_methods
    forward_base.type.name(), forward_name, context)
AttributeError: 'UnionType' object has no attribute 'type'
third_party/3/pyspark/sql/column.pyi:24: note: use --pdb to drop into pdb

[gvanrossum]

Can you come up with a repro that doesn't still depend on the pyspark stubs? I get an error on the import line.

[zero323]

Let me try to dig deeper into this over the weekend. It doesn't appear in isolation, so it guess the problem can be somewhere completely else, and just manifests in an awkward way.

[ilevkivskyi]

Here is a simple repro for the crash:

class X:
    def __mod__(self, other: Union[X, str]) -> X: ...
    def __rmod__(self, other: Union[X, str]) -> X: ...

I think the problem is because of an unsafe cast here https://github.com/python/mypy/blob/master/mypy/checker.py#L805

[gvanrossum]

Ah, casts! They cause so much trouble and are so misunderstood. That one should have been an assert, but how to prevent crashing?

(PS Crash >= Bug so I removed the bug label.)

[ilevkivskyi]

but how to prevent crashing?

TBH I am not sure what to do here. It looks like this crash happens only if Any appears in the special method signature (like in str.__mod__). @JukkaL do you have any ideas?

[JukkaL]

I can have a look at this. The relevant code is really old and seems to have bit rotted (assuming that it ever worked).

[elazarg]

I think it is related to the issue I have described in #3227

[wilfreddenton]

I get the same error when I run mypy on fractions.pyi in the typeshed. Using mypy 0.550.

❯ mypy /Users/wilfred/python-envs/sailfish/lib/mypy/typeshed/stdlib/2and3/fractions.pyi --show-traceback
/Users/wilfred/python-envs/sailfish/lib/mypy/typeshed/stdlib/2and3/fractions.pyi:80: error: INTERNAL ERROR -- please report a bug at https://github.com/pyth
on/mypy/issues version: 0.550                                                                                                                              
Traceback (most recent call last):
  File "/Users/wilfred/python-envs/sailfish/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/main.py", line 66, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/main.py", line 119, in type_check_only
    options=options)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/build.py", line 214, in build
    graph = dispatch(sources, manager)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/build.py", line 1935, in dispatch
    process_graph(graph, manager)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/build.py", line 2199, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/build.py", line 2327, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/build.py", line 1836, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 196, in check_first_pass
    self.accept(d)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/nodes.py", line 690, in accept
    return visitor.visit_class_def(self)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 1185, in visit_class_def
    self.accept(defn.defs)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/nodes.py", line 750, in accept
    return visitor.visit_block(self)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 1305, in visit_block
    self.accept(s)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/nodes.py", line 496, in accept
    return visitor.visit_func_def(self)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 531, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 591, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 641, in check_func_def
    self.check_reverse_op_method(item, typ, name)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 859, in check_reverse_op_method
    defn)
  File "/Users/wilfred/python-envs/sailfish/lib/python3.5/site-packages/mypy/checker.py", line 929, in check_overlapping_op_methods
    forward_base.type.name(), forward_name, context)
AttributeError: 'UnionType' object has no attribute 'type'
/Users/wilfred/python-envs/sailfish/lib/mypy/typeshed/stdlib/2and3/fractions.pyi:80: : note: use --pdb to drop into pdb

However if I run the same command but within my home directory I do not have this issue. I made sure to delete all instances of .mypy_cache.

[elazarg]

I can't reproduce this, but there's a slim chance that #4017 will avoid the crash.

[elazarg]

@ilevkivskyi in your repro (with the additional from typing import Union), #4017 gives

tmp.py:5: error: Signatures of "__rmod__" of "X" and "__mod__" of "Union[X, str]" are unsafely overlapping

I'm not sure if that's correct, but at least it's not a crash.