Repository navigation
Implicit type aliases not recognised with PEP 604 + import cycles #11887
Description
This bug was originally surfaced by python/typeshed#6612. It is hard to tell if it is the only bug that was revealed by that PR, or if there are other PEP 604-related bugs in mypy.
Top-line summary
If a single line in typeshed/stdlib/_typeshed/__init__.pyi is changed from using Union to PEP 604-style syntax, a large number of errors are output by mypy:
--- a/stdlib/_typeshed/__init__.pyi
+++ b/stdlib/_typeshed/__init__.pyi
@@ -189,7 +189,7 @@ ReadOnlyBuffer = bytes # stable
# for it. Instead we have to list the most common stdlib buffer classes in a Union.
WriteableBuffer = Union[bytearray, memoryview, array.array[Any], mmap.mmap, ctypes._CData] # stable
# Same as _WriteableBuffer, but also includes read-only buffer types (like bytes).
-ReadableBuffer = Union[ReadOnlyBuffer, WriteableBuffer] # stable
+ReadableBuffer = ReadOnlyBuffer | WriteableBuffer # stableMypy errors
stdlib\io.pyi:44: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:44: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:54: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:54: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:61: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:61: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:74: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:74: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:104: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:104: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:52: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:52: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:53: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:53: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:55: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:55: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:64: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:64: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ctypes\__init__.pyi:89: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ctypes\__init__.pyi:89: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\builtins.pyi:672: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\builtins.pyi:672: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:9: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:9: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:10: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:10: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:11: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:11: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:23: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:23: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:24: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:24: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:25: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:25: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\lzma.pyi:90: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\lzma.pyi:90: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:19: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:19: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:30: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:30: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:31: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:31: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:37: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:37: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:42: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:42: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:12: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:12: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:16: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:16: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:19: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:19: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:20: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:20: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:21: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:21: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:22: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:22: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:23: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:23: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:24: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:24: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:25: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:25: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:49: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:49: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:56: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:56: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:60: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:60: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:70: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:70: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:72: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:72: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:89: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:89: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:92: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:92: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:93: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:93: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:94: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:94: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:14: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:14: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:560: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:560: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:561: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:561: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:563: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:563: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:565: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:565: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\socket.pyi:603: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\socket.pyi:603: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\gzip.pyi:136: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\gzip.pyi:136: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\bz2.pyi:127: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\bz2.pyi:127: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\bz2.pyi:128: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\bz2.pyi:128: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:343: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:343: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:344: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:344: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:346: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:346: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:348: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:348: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
Investigation
This bug is hard to pin down. I have not managed to reproduce it outside of the typeshed repo.
I have managed to reproduce the bug in a branch of typeshed in which the vast majority of the stdlib has been deleted. This indicates to me that the bug has something to do with an interaction between builtins, _typeshed and mmap. In this "slimmed-down" version of typeshed, I see the following errors from mypy:
stdlib\builtins.pyi:677: error: Name "ReadableBuffer" is not defined
stdlib\io.pyi:5: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
Notes on the bug
The bug can be reproduced if ReadableBuffer is defined like this in _typeshed:
import mmap
WriteableBuffer = mmap.mmap
ReadableBuffer = bytes | WriteableBufferHowever...
(1) It cannot be reproduced if ReadableBuffer is defined like this in typeshed:
import mmap
ReadableBuffer = bytes | mmap.mmap(2) Nor can it be reproduced if ReadableBuffer is defined like this in typeshed:
from mmap import mmap
WriteableBuffer = mmap
ReadableBuffer = bytes | WriteableBuffer(3) If the definition of ReadableBuffer is moved to builtins, the errors in builtins and io that you see in "slimmed-down typeshed" continue to be reported by mypy.
(4) The errors in the io module that you see in "slimmed-down typeshed" do not exist if io defines its own version of ReadableBuffer, instead of importing the definition from _typeshed or builtins.
Expected behaviour
No error should be reported; the semantics of typing.Union and PEP 604 union-type expressions should be identical.
mypy version
0.930. Reproduced on many Python versions by the typeshed CI; local testing done on Python 3.10.
To reproduce
- Either clone the typeshed master branch, and apply the above diff manually, or clone my "slimmed-down typeshed branch".
- cd into your clone of the typeshed repo.
- Run
mypy stdlib/abc.pyi --custom-typeshed-dir .(the precise file you run mypy on is irrelevant).
Activity
hauntsaninja commentedon Jan 3, 2022
I think you miiiiight be able to use PEP 613 as a workaround.
Also note, your slimmed down branch is a little broken, ReadableBuffer does seem actually undefined.
hauntsaninja commentedon Jan 3, 2022
Import cycles involving builtins can do some scary things, here's a bug I ran into: #11535
hauntsaninja commentedon Jan 3, 2022
I did some tracing of the code involved.
Looking at semanal's
can_be_type_alias(called bycheck_and_set_up_type_alias, called byvisit_assignment_stmt).mypy/mypy/semanal.py
Line 2190 in 1beed9a
If we encounter
ReadableBuffer = Union[...], we have rv is a IndexExpr, we go intois_type_refandtyping.Unionis included invalid_refs = type_constructorsso we return True and all is well. We don't even look at the contents of the Union.If we encounter
ReadableBuffer = ReadOnlyBuffer | WriteableBufferwe check to see if both sides of "|" can be a type alias.This check succeeds for the left side (
'NameExpr(ReadOnlyBuffer [_typeshed.ReadOnlyBuffer])'), but fails for the right side'NameExpr(WriteableBuffer)'.So mypy knows that
ReadOnlyBufferis a type alias and sois_type_refis okay with it. But all it has forWriteableBufferis a placeholder (likely because of the import cycles), and it is not okay with that --if n and isinstance(n.node, PlaceholderNode) and n.node.becomes_typeinfois False.So two things that could mechanically fix this:
WriteableBuffer = Union[...], so mypy could easily know at this point that it is a type alias, even if...involves horrendous import cycles. Although obviously we'd want our solution to still work ifWriteableBufferused PEP 604.WriteableBufferis indeed not going to become a type info, but it is going to become a type alias and that should count for the purposes ofis_type_ref. It looks like the placeholder'sbecomes_typeinfois basically just used for this kind of thing, so I think we could repurpose it asbecomes_type.Unfortunately, I'm not yet sure what the best way to ensure either of those.
The secret to both lies back in
visit_assignment_stmt:mypy/mypy/semanal.py
Line 2070 in 1beed9a
I don't fully understand the reference tagging thing, but tldr the import cycle causes us to skip calling
check_and_set_up_type_aliasforWriteableBufferfor now, so all mypy has is a placeholder node.Moreover, the placeholder's
becomes_typeinfois False, because we callmark_incompletewithout specifyingbecomes_typeinfo=Truemypy/mypy/semanal.py
Line 2075 in 1beed9a
I'm not sure how we could know
becomes_typeinfo=Trueat this point, although we could knowcould_become_type..... 🤷
So next step is probably to see the impact of changing the semantics of
becomes_typeinfotocould_become_type. Unless you see a better fix.Anyway, for typeshed purposes, maybe just use PEP 613 :-) (works because we don't need the result of
can_be_type_alias)AlexWaygood commentedon Jan 3, 2022
It's defined here in my branch, no? 🙂 note that my branch doesn't make any kind of sense from a typing perspective, I was just trying to delete as many modules as possible while still being able to reproduce the bug!!
AlexWaygood commentedon Jan 3, 2022
Thanks for taking the time to look into it!! This is a great rundown.
Unfortunately, I think PEP 613 may still be unuseable for typeshed purposes — I'm not sure if pytype supports it yet 😕 (google/pytype#787) but, worth a shot!
For this specific instance of the bug, it seems like it can be fixed just by doing
from mmap import mmapin_typeshed(but my worry is that quite a few things are imported intypeshed/stdlib/builtins.pyi, so there might be other instances of this bug — haven't investigated that yet...)[-]Very specific bug related to PEP 604, type aliases & typeshed[/-][+]Implicit type aliases not recognised with PEP 604 + import cycles[/+](minor) Short circuit call to `can_be_type_alias` when using PEP 613
(minor) Short circuit call to `can_be_type_alias` when using PEP 613 (#…
AlexWaygood commentedon Jan 5, 2022
Oh wait, yes, lol — I was experimenting to see if the
ioerrors are still there if it is imported iniofrom_typeshed, but isn't imported inbuiltinsfrom_typeshed. (Answer: yes, they are — which I think is because other things are still imported from_typeshedinbuiltins, meaning you still get cyclic references.)I left that in by mistake when I pushed to my remote branch — sorry for the confusion!!
AlexWaygood commentedon Jan 5, 2022
"smol typeshed" should now be fixed by AlexWaygood/typeshed@79f1d16 -- the only error from mypy is now:
AlexWaygood commentedon Jan 5, 2022
Okay, this is very strange. If I'm in the branch of typeshed with nearly all modules deleted, and
ReadableBufferonly imported inio, and I run typeshed'sstubtest_stdlib.pyscript, I get this output (the output I reported immediately above):But if I just run
mypy, I get no errors at all in the branch withReadableBufferonly imported inio:Any idea why mypy 0.930 is succeeding, but stubtest 0.930 is failing with "mypy build errors"??
Akuli commentedon Jan 5, 2022
Currently python/typeshed#6819 has the same problem.
AlexWaygood commentedon Jan 7, 2022
You should now be able to reproduce the bug in "smol typeshed" by simply running
mypyinstead of runningstubtestafter AlexWaygood/typeshed@01496f2 ...AlexWaygood commentedon Jan 12, 2022
Okay, I think I have managed to create a repro for this issue that is only 11 lines of code, spread over 6 files. SO, to reproduce:
mypy .Here's what the minimal repro looks like:
(minor) Short circuit call to `can_be_type_alias` when using PEP 613 (p…
hauntsaninja commentedon Jan 30, 2022
To clarify current state, this issue is patched for stubs, but remains for py files (where we can't blindly treat
x = y | zas always being a type alias). There's an xfail test case.If you run into this, your best option is likely using an explicit type alias (
x: TypeAlias = y | z).Fix new style union aliases with not ready items (python#22172)