stdlib: use PEP 604 in most places #5564

hauntsaninja · 2021-06-01T07:36:26Z

This uses PEP 604 in place of Optional and Union in most cases.

Script used to produce this (stuck with regexes and not lib2to3 or libcst or astor or whatever for fear of https://xkcd.com/1319):

import subprocess

commands = r"""
# some of these regexes are subsumed by later regexes, but doing things this way isolates the more
# complicated replacements to their own commits, where they can be inspected more thoroughly

# simple unions
fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+), ([\w\.]+)\]' '$1 | $2'
fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+), ([\w\.]+), ([\w\.]+)\]' '$1 | $2 | $3'
fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+), ([\w\.]+), ([\w\.]+), ([\w\.]+)\]' '$1 | $2 | $3 | $4'

# unions with single level generics
fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+)(\[[^\[\]]+\])?, ([\w\.]+)(\[[^\[\]]+\])?\]' '$1$2 | $3$4'
fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+)(\[[^\[\]]+\])?, ([\w\.]+)(\[[^\[\]]+\])?, ([\w\.]+)(\[[^\[\]]+\])?\]' '$1$2 | $3$4 | $5$6'

# simple optionals
fastmod -d stdlib --accept-all '\bOptional\[([\w\.\| ]+)\]' '$1 | None'
# optionals with single level generics
fastmod -d stdlib --accept-all '\bOptional\[([^\[\]]+)\[([^\[\]]+)\]\]' '$1[$2] | None'

# unused imports
for x in $(rg '\bOptional\b' -c | grep stdlib | pyp 'x.split(":")[0] if x.endswith(":1") else None'); do fastmod -g $x --accept-all '\s*from typing import Optional$' ''; done
for x in $(rg '\bOptional\b' -c | grep stdlib | pyp 'x.split(":")[0] if x.endswith(":1") else None'); do fastmod -g $x --accept-all '^\s+Optional,$' ''; done
for x in $(rg '\bOptional\b' -c | grep stdlib | pyp 'x.split(":")[0] if x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import )Optional, (.*)' '$1$2'; done
for x in $(rg '\bOptional\b' -c | grep stdlib | pyp 'x.split(":")[0] if x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import.*), Optional' '$1'; done
for x in $(rg '\bUnion\b' -c | grep stdlib | pyp 'x.split(":")[0] if x.endswith(":1") else None'); do fastmod -g $x --accept-all '\s*from typing import Union$' ''; done
for x in $(rg '\bUnion\b' -c | grep stdlib | pyp 'x.split(":")[0] if x.endswith(":1") else None'); do fastmod -g $x --accept-all '^\s+Union,$' ''; done
for x in $(rg '\bUnion\b' -c | grep stdlib | pyp 'x.split(":")[0] if x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import )Union, (.*)' '$1$2'; done
for x in $(rg '\bUnion\b' -c | grep stdlib | pyp 'x.split(":")[0] if x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import.*), Union' '$1'; done
"""

subprocess.check_call("git reset --hard origin/master", shell=True)
for cmd in commands.splitlines():
    print(cmd)
    subprocess.check_call(cmd, shell=True)
    try:
        subprocess.check_call("git diff --exit-code > /dev/null", shell=True)
    except:
        subprocess.check_call(["git", "commit", "--all", "-m", cmd])
subprocess.check_call("black stdlib; isort stdlib", shell=True)
subprocess.check_call(["git", "commit", "--all", "-m", "black and isort"])

…| $2'

…]+)\]' '$1 | $2 | $3'

…]+), ([\w\.]+)\]' '$1 | $2 | $3 | $4'

…\w\.]+)(\[[^\[\]]+\])?\]' '$1$2 | $3$4'

…\w\.]+)(\[[^\[\]]+\])?, ([\w\.]+)(\[[^\[\]]+\])?\]' '$1$2 | $3$4 | $5$6'

…]\]' '$1[$2] | None'

…if x.endswith(":1") else None'); do fastmod -g $x --accept-all '\s*from typing import Optional$' ''; done

…if x.endswith(":1") else None'); do fastmod -g $x --accept-all '^\s+Optional,$' ''; done

…if x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import )Optional, (.*)' '$1$2'; done

…if x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import.*), Optional' '$1'; done

…x.endswith(":1") else None'); do fastmod -g $x --accept-all '\s*from typing import Union$' ''; done

…x.endswith(":1") else None'); do fastmod -g $x --accept-all '^\s+Union,$' ''; done

…x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import )Union, (.*)' '$1$2'; done

…x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import.*), Union' '$1'; done

github-actions · 2021-06-01T08:03:00Z

Diff from mypy_primer, showing the effect of this PR on open source code:

black (https://github.com/psf/black.git)
+ src/blib2to3/pgen2/grammar.py:110:17: error: Argument "dir" to "NamedTemporaryFile" has incompatible type "str"; expected "None"
+ src/black/cache.py:79:46: error: Argument "dir" to "NamedTemporaryFile" has incompatible type "str"; expected "None"
+ src/black_primer/lib.py:54:28: error: "<nothing>" object is not iterable
+ src/black_primer/lib.py:68:49: error: Cannot determine type of "stdout"
+ src/black_primer/lib.py:68:64: error: Cannot determine type of "stderr"
+ src/black_primer/lib.py:71:13: error: Cannot determine type of "stdout"
+ src/black_primer/lib.py:71:21: error: Cannot determine type of "stderr"

pip (https://github.com/pypa/pip.git)
+ src/pip/_internal/utils/distutils_args.py:22: error: unused "type: ignore" comment
+ src/pip/_internal/utils/logging.py:221: error: Value of type _OptExcInfo? is not indexable
+ src/pip/_internal/utils/filesystem.py:88: error: Argument "dir" to "NamedTemporaryFile" has incompatible type "str"; expected "None"
+ src/pip/_internal/utils/subprocess.py:169: error: No overload variant of "Popen" matches argument types "List[str]", "int", "int", "int", "Optional[str]", "Dict[str, str]", "str"
+ src/pip/_internal/utils/subprocess.py:169: note:     <3 more non-matching overloads not shown>
+ src/pip/_internal/utils/subprocess.py:169: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: Literal[False] = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, text: Optional[Literal[False]] = ..., encoding: None = ..., errors: None = ...) -> Popen[bytes]
+ src/pip/_internal/utils/subprocess.py:169: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: bool = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, text: Optional[bool] = ..., encoding: Optional[str] = ..., errors: Optional[str] = ...) -> Popen[Any]
+ src/pip/_internal/utils/subprocess.py:169: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: bool = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, text: Optional[bool] = ..., encoding: Optional[str] = ..., errors: str) -> Popen[str]
+ src/pip/_internal/utils/subprocess.py:169: note: Possible overload variants:
+ src/pip/_internal/vcs/versioncontrol.py:96: error: Incompatible return value type (got "bytes", expected "Optional[str]")
+ src/pip/_internal/operations/install/wheel.py:255: error: Incompatible types in assignment (expression has type "bytes", variable has type "str")
+ src/pip/_internal/operations/install/wheel.py:793: error: Value of type _OptExcInfo? is not indexable
+ src/pip/_internal/operations/install/legacy.py:123: error: Argument 1 to "join" of "str" has incompatible type "List[bytes]"; expected "Iterable[str]"
+ src/pip/_internal/network/lazy_wheel.py:35: error: unused "type: ignore" comment
+ src/pip/_internal/network/lazy_wheel.py:177: error: unused "type: ignore" comment

manticore (https://github.com/trailofbits/manticore.git)
+ tests/native/test_integration_native.py:43: error: No overload variant of "Popen" matches argument types "List[str]", "TextIO"
+ tests/native/test_integration_native.py:43: note:     <3 more non-matching overloads not shown>
+ tests/native/test_integration_native.py:43: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: Literal[False] = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, encoding: None = ..., errors: None = ...) -> Popen[bytes]
+ tests/native/test_integration_native.py:43: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: bool = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, encoding: Optional[str] = ..., errors: Optional[str] = ...) -> Popen[Any]
+ tests/native/test_integration_native.py:43: note: Possible overload variants:

pandas (https://github.com/pandas-dev/pandas.git)
+ pandas/io/common.py:786: error: unused "type: ignore" comment
+ pandas/io/excel/_base.py:1092: error: unused "type: ignore" comment
+ pandas/tests/io/xml/test_xml.py:296: error: Argument 1 to "BytesIO" has incompatible type "str"; expected "bytes"  [arg-type]
+ pandas/tests/io/xml/test_xml.py:799: error: Argument 1 to "BytesIO" has incompatible type "str"; expected "bytes"  [arg-type]
+ pandas/tests/io/xml/test_xml.py:949: error: Argument 1 to "BytesIO" has incompatible type "str"; expected "bytes"  [arg-type]
+ pandas/tests/io/xml/test_to_xml.py:970: error: Argument 1 to "BytesIO" has incompatible type "str"; expected "bytes"  [arg-type]

aiortc (https://github.com/aiortc/aiortc)
+ src/aiortc/rtcdtlstransport.py:597: error: Need type annotation for "data"

poetry (https://github.com/python-poetry/poetry.git)
- poetry/locations.py:19: error: Argument 1 to "Path" has incompatible type "Optional[str]"; expected "Union[str, PathLike[str]]"
+ poetry/utils/helpers.py:132: error: Argument "dir" to "TemporaryFile" has incompatible type "str"; expected "None"
- poetry/inspection/info.py:268: error: Incompatible types in assignment (expression has type "Callable[[Union[str, bytes, PathLike[str], PathLike[bytes], None], str, Optional[IO[bytes]], int, DefaultNamedArg(Optional[int], 'format'), DefaultNamedArg(Optional[Type[TarInfo]], 'tarinfo'), DefaultNamedArg(Optional[bool], 'dereference'), DefaultNamedArg(Optional[bool], 'ignore_zeros'), DefaultNamedArg(Optional[str], 'encoding'), DefaultNamedArg(str, 'errors'), DefaultNamedArg(Optional[Mapping[str, str]], 'pax_headers'), DefaultNamedArg(Optional[int], 'debug'), DefaultNamedArg(Optional[int], 'errorlevel'), DefaultNamedArg(Optional[int], 'compresslevel')], TarFile]", variable has type "Type[ZipFile]")
+ poetry/inspection/info.py:268: error: Incompatible types in assignment (expression has type "Callable[[Optional[Union[str, bytes, PathLike[str], PathLike[bytes]]], str, Optional[IO[bytes]], int, DefaultNamedArg(Optional[int], 'format'), DefaultNamedArg(Optional[Type[TarInfo]], 'tarinfo'), DefaultNamedArg(Optional[bool], 'dereference'), DefaultNamedArg(Optional[bool], 'ignore_zeros'), DefaultNamedArg(Optional[str], 'encoding'), DefaultNamedArg(str, 'errors'), DefaultNamedArg(Optional[Mapping[str, str]], 'pax_headers'), DefaultNamedArg(Optional[int], 'debug'), DefaultNamedArg(Optional[int], 'errorlevel'), DefaultNamedArg(Optional[int], 'compresslevel')], TarFile]", variable has type "Type[ZipFile]")
- poetry/console/commands/init.py:496: note:     def [AnyStr in (str, bytes)] search(pattern: AnyStr, string: AnyStr, flags: Union[int, RegexFlag] = ...) -> Optional[Match[AnyStr]]
+ poetry/console/commands/init.py:496: note:     def [AnyStr in (str, bytes)] search(pattern: AnyStr, string: AnyStr, flags: _FlagsType? = ...) -> Optional[Match[AnyStr]]
- poetry/console/commands/init.py:496: note:     def [AnyStr in (str, bytes)] search(pattern: Pattern[AnyStr], string: AnyStr, flags: Union[int, RegexFlag] = ...) -> Optional[Match[AnyStr]]
+ poetry/console/commands/init.py:496: note:     def [AnyStr in (str, bytes)] search(pattern: Pattern[AnyStr], string: AnyStr, flags: _FlagsType? = ...) -> Optional[Match[AnyStr]]
- poetry/console/commands/self/update.py:60: error: Argument 1 to "Path" has incompatible type "Optional[str]"; expected "Union[str, PathLike[str]]"
- poetry/console/commands/plugin/remove.py:48: error: Argument 1 to "Path" has incompatible type "Union[str, None, Path]"; expected "Union[str, PathLike[str]]"
- poetry/console/commands/plugin/add.py:78: error: Argument 1 to "Path" has incompatible type "Union[str, None, Path]"; expected "Union[str, PathLike[str]]"

graphql-core (https://github.com/graphql-python/graphql-core.git)
+ tests/subscription/test_map_async_iterator.py:375: error: Value of type _OptExcInfo? is not indexable
+ src/graphql/error/graphql_error.py:137: error: Value of type _OptExcInfo? is not indexable
+ tests/execution/test_parallel.py:57: error: Need type annotation for "result"
+ tests/execution/test_parallel.py:87: error: Need type annotation for "result"
+ tests/execution/test_parallel.py:150: error: Need type annotation for "result"

sphinx (https://github.com/sphinx-doc/sphinx.git)
+ sphinx/util/console.py: note: In function "get_terminal_width":
+ sphinx/util/console.py:39:47: error: Argument 2 to "unpack" has incompatible type "int"; expected "Union[bytes, bytearray, memoryview, array[Any], mmap]"
+ sphinx/util/osutil.py: note: In function "relpath":
+ sphinx/util/osutil.py:140:16: error: Incompatible return value type (got "bytes", expected "str")
+ sphinx/util/__init__.py: note: In function "save_traceback":
+ sphinx/util/__init__.py:222:11: error: Value of type _OptExcInfo? is not indexable
+ sphinx/util/__init__.py: note: In function "format_exception_cut_frames":
+ sphinx/util/__init__.py:380:20: error: "_OptExcInfo?" object is not iterable
+ sphinx/util/__init__.py:383:33: error: Cannot determine type of "tb"
+ sphinx/util/__init__.py:385:44: error: Cannot determine type of "typ"
+ sphinx/util/__init__.py:385:49: error: Cannot determine type of "val"
+ sphinx/cmd/build.py: note: In function "handle_exception":
+ sphinx/cmd/build.py:41:25: error: Value of type _OptExcInfo? is not indexable

kopf (https://github.com/nolar/kopf.git)
+ kopf/_cogs/aiokits/aioenums.py:170: error: Need type annotation for "coro"
+ kopf/_cogs/clients/scanning.py:53: error: Need type annotation for "coro"
+ kopf/_cogs/clients/scanning.py:77: error: Need type annotation for "coro"
+ kopf/_cogs/clients/scanning.py:103: error: Need type annotation for "coro"
+ kopf/_core/reactor/queueing.py:290: error: Need type annotation for "raw_event"
+ kopf/_core/reactor/queueing.py:299: error: Need type annotation for "next_event"
+ kopf/_core/engines/posting.py:83: error: Need type annotation for "future"
+ kopf/_core/engines/posting.py:135: error: "_OptExcInfo?" object is not iterable

prefect (https://github.com/PrefectHQ/prefect.git)
+ src/prefect/utilities/executors.py:61: error: No overload variant of "Popen" matches argument types "List[str]", "Dict[str, str]", "int", "int"
+ src/prefect/utilities/executors.py:61: note:     <4 more non-matching overloads not shown>
+ src/prefect/utilities/executors.py:61: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: Literal[False] = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, text: Optional[Literal[False]] = ..., encoding: None = ..., errors: None = ...) -> Popen[bytes]
+ src/prefect/utilities/executors.py:61: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: bool = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, text: Optional[bool] = ..., encoding: Optional[str] = ..., errors: Optional[str] = ...) -> Popen[Any]
+ src/prefect/utilities/executors.py:61: note: Possible overload variants:
+ src/prefect/storage/docker.py:328: error: Argument "dir" to "TemporaryDirectory" has incompatible type "Optional[str]"; expected "None"
+ src/prefect/executors/dask.py:255: error: "None" has no attribute "write"
+ src/prefect/executors/dask.py:256: error: "None" has no attribute "read"
+ src/prefect/executors/dask.py:259: error: "None" has no attribute "read"

python-chess (https://github.com/niklasf/python-chess.git)
+ chess/gaviota.py:1952: error: List comprehension has incompatible type List[bytes]; expected List[c_char_p]
+ chess/engine.py:2697: error: Need type annotation for "future"
+ chess/engine.py:2698: error: Returning Any from function declared to return "MutableMapping[str, Option]"
+ chess/engine.py:2706: error: Need type annotation for "future"
+ chess/engine.py:2707: error: Returning Any from function declared to return "Mapping[str, str]"
+ chess/engine.py:2712: error: Need type annotation for "future"
+ chess/engine.py:2713: error: Returning Any from function declared to return "T"
+ chess/engine.py:2717: error: Need type annotation for "coro"
+ chess/engine.py:2718: error: Need type annotation for "future"
+ chess/engine.py:2719: error: Returning Any from function declared to return "None"
+ chess/engine.py:2723: error: Need type annotation for "coro"
+ chess/engine.py:2724: error: Need type annotation for "future"
+ chess/engine.py:2725: error: Returning Any from function declared to return "None"
+ chess/engine.py:2729: error: Need type annotation for "coro"
+ chess/engine.py:2732: error: Need type annotation for "future"
+ chess/engine.py:2733: error: Returning Any from function declared to return "PlayResult"
+ chess/engine.py:2743: error: Need type annotation for "coro"
+ chess/engine.py:2746: error: Need type annotation for "future"
+ chess/engine.py:2747: error: Returning Any from function declared to return "Union[InfoDict, List[InfoDict]]"
+ chess/engine.py:2751: error: Need type annotation for "coro"
+ chess/engine.py:2754: error: Need type annotation for "future"
+ chess/engine.py:2759: error: Need type annotation for "coro"
+ chess/engine.py:2760: error: Need type annotation for "future"
+ chess/engine.py:2761: error: Returning Any from function declared to return "None"
+ chess/engine.py:2836: error: Need type annotation for "future"
+ chess/engine.py:2837: error: Returning Any from function declared to return "InfoDict"
+ chess/engine.py:2845: error: Need type annotation for "future"
+ chess/engine.py:2846: error: Returning Any from function declared to return "List[InfoDict]"
+ chess/engine.py:2854: error: Need type annotation for "future"
+ chess/engine.py:2855: error: Returning Any from function declared to return "BestMove"
+ chess/engine.py:2862: error: Need type annotation for "future"
+ chess/engine.py:2863: error: Returning Any from function declared to return "bool"
+ chess/engine.py:2867: error: Need type annotation for "future"
+ chess/engine.py:2868: error: Returning Any from function declared to return "InfoDict"
+ chess/engine.py:2872: error: Need type annotation for "future"
+ chess/engine.py:2873: error: Returning Any from function declared to return "Optional[InfoDict]"
+ chess/engine.py:2883: error: Need type annotation for "future"
+ chess/engine.py:2884: error: Returning Any from function declared to return "InfoDict"

dragonchain (https://github.com/dragonchain/dragonchain.git)
+ dragonchain/lib/interfaces/local/disk.py:123:31: error: Argument 1 to "startswith" of "bytes" has incompatible type "str"; expected "Union[bytes, Tuple[bytes, ...]]"
+ dragonchain/lib/interfaces/local/disk.py:125:12: error: Incompatible return value type (got "List[bytes]", expected "List[str]")

isort (https://github.com/pycqa/isort.git)
+ isort/settings.py:516: error: Incompatible types in assignment (expression has type "str", variable has type "bytes")
+ isort/settings.py:547: error: Unsupported operand types for + ("str" and "bytes")
+ isort/settings.py:547: error: Value of type variable "AnyStr" of "fnmatch" cannot be "object"

mypy_primer (https://github.com/hauntsaninja/mypy_primer.git)
+ mypy_primer.py:263: error: Need type annotation for "new_mypy"
+ mypy_primer.py:263: error: Need type annotation for "old_mypy"
+ mypy_primer.py:269: error: Need type annotation for "new_version"
+ mypy_primer.py:269: error: Need type annotation for "old_version"
+ mypy_primer.py:394: error: Need type annotation for "new_result"
+ mypy_primer.py:394: error: Need type annotation for "old_result"
+ mypy_primer.py:673: error: Need type annotation for "paths"
+ mypy_primer.py:708: error: Need type annotation for "result_fut"

ignite (https://github.com/pytorch/ignite)
+ ignite/handlers/checkpoint.py:697: error: Argument "dir" to "NamedTemporaryFile" has incompatible type "str"; expected "None"  [arg-type]

paasta (https://github.com/yelp/paasta.git)
+ paasta_tools/cli/cmds/list_deploy_queue.py:89: error: Value of type _OptExcInfo? is not indexable

mypy (https://github.com/python/mypy.git)
+ mypyc/crash.py:19: error: Value of type _OptExcInfo? is not indexable  [index]
+ mypyc/crash.py:26: error: Value of type _OptExcInfo? is not indexable  [index]
+ mypy/metastore.py:135: error: Incompatible types in assignment (expression has type "bytes", variable has type "str")  [assignment]
+ mypy/fscache.py:120: error: Incompatible types in assignment (expression has type "bytes", variable has type "str")  [assignment]
+ mypy/errors.py:746: error: Value of type _OptExcInfo? is not indexable  [index]
+ mypy/errors.py:758: error: Value of type _OptExcInfo? is not indexable  [index]
+ mypy/modulefinder.py:495: error: Argument 1 to "replace" of "bytes" has incompatible type "str"; expected "bytes"  [arg-type]
+ mypy/modulefinder.py:495: error: Argument 2 to "replace" of "bytes" has incompatible type "str"; expected "bytes"  [arg-type]
+ mypy/modulefinder.py:497: error: Unsupported operand types for + ("bytes" and "str")  [operator]
+ mypy/modulefinder.py:498: error: Value of type variable "AnyStr" of "search" cannot be "object"  [type-var]
+ mypy/modulefinder.py:500: error: On Python 3 '{}'.format(b'abc') produces "b'abc'", not 'abc'; use '{!r}'.format(b'abc') if this is desired behavior  [str-bytes-safe]
+ mypy/report.py:468: error: Argument 1 to "should_skip_path" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/report.py:480: error: Argument 1 to "FileInfo" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/report.py:482: error: Argument 1 to "iterate_python_lines" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/report.py:493: error: Argument 1 to "pathname2url" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/report.py:529: error: Argument 1 to "pathname2url" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/report.py:601: error: Argument 1 to "FileInfo" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/report.py:649: error: Invalid index type "bytes" for "Dict[str, Any]"; expected type "str"  [index]
+ mypy/report.py:697: error: Argument 1 to "startswith" of "bytes" has incompatible type "str"; expected "Union[bytes, Tuple[bytes, ...]]"  [arg-type]
+ mypy/report.py:699: error: Unsupported operand types for + ("bytes" and "str")  [operator]
+ mypy/report.py:740: error: Argument 1 to "startswith" of "bytes" has incompatible type "str"; expected "Union[bytes, Tuple[bytes, ...]]"  [arg-type]
+ mypy/report.py:742: error: Unsupported operand types for + ("bytes" and "str")  [operator]
+ mypy/report.py:827: error: Argument 1 to "should_skip_path" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/report.py:837: error: Argument 1 to "FileInfo" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/report.py:838: error: Argument 1 to "iterate_python_lines" has incompatible type "bytes"; expected "str"  [arg-type]
+ mypy/build.py:291: error: Incompatible return value type (got "bytes", expected "str")  [return-value]
+ mypy/build.py:1182: error: Incompatible return value type (got "Tuple[bytes, bytes, None]", expected "Tuple[str, str, Optional[str]]")  [return-value]
+ mypy/build.py:2435: error: On Python 3 '%s' % b'abc' produces "b'abc'", not 'abc'; use '%r' % b'abc' if this is desired behavior  [str-bytes-safe]
+ mypy/main.py:1022: error: Incompatible types in assignment (expression has type "bytes", variable has type "str")  [assignment]
+ mypy/dmypy_server.py:258: error: Value of type _OptExcInfo? is not indexable  [index]
+ mypy/test/testmodulefinder.py:152: error: List item 0 has incompatible type "bytes"; expected "str"  [list-item]
+ mypy/test/testcmdline.py:60: error: No overload variant of "Popen" matches argument types "List[str]", "int", "int", "str", "Dict[str, str]"  [call-overload]
+ mypy/test/testcmdline.py:60: note:     <3 more non-matching overloads not shown>
+ mypy/test/testcmdline.py:60: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: Literal[False] = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, encoding: None = ..., errors: None = ...) -> Popen[bytes]
+ mypy/test/testcmdline.py:60: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: bool = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, encoding: Optional[str] = ..., errors: Optional[str] = ...) -> Popen[Any]
+ mypy/test/testcmdline.py:60: note: Possible overload variants:
+ mypyc/test/test_run.py:202: error: Incompatible types in assignment (expression has type "bytes", variable has type "str")  [assignment]
+ mypyc/test/test_run.py:294: error: No overload variant of "Popen" matches argument types "List[str]", "int", "int", "Dict[str, str]"  [call-overload]
+ mypyc/test/test_run.py:294: note:     <3 more non-matching overloads not shown>
+ mypyc/test/test_run.py:294: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: Literal[False] = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, encoding: None = ..., errors: None = ...) -> Popen[bytes]
+ mypyc/test/test_run.py:294: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: bool = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, encoding: Optional[str] = ..., errors: Optional[str] = ...) -> Popen[Any]
+ mypyc/test/test_run.py:294: note: Possible overload variants:

anyio (https://github.com/agronholm/anyio.git)
- src/anyio/_backends/_asyncio.py:191: error: Argument 1 to "getgeneratorstate" has incompatible type "Union[Generator[Optional[Future[object]], None, Any], Awaitable[Any]]"; expected "Generator[Any, Any, Any]"
+ src/anyio/_backends/_asyncio.py:191: error: Argument 1 to "getgeneratorstate" has incompatible type "Union[Generator[_TaskYieldType?, None, Any], Awaitable[Any]]"; expected "Generator[Any, Any, Any]"
- src/anyio/_backends/_asyncio.py:1803: error: Argument 4 to "TaskInfo" has incompatible type "Union[Generator[Optional[Future[object]], None, Any], Awaitable[Any]]"; expected "Coroutine[Any, Any, Any]"
+ src/anyio/_backends/_asyncio.py:1803: error: Argument 4 to "TaskInfo" has incompatible type "Union[Generator[_TaskYieldType?, None, Any], Awaitable[Any]]"; expected "Coroutine[Any, Any, Any]"

boostedblob (https://github.com/hauntsaninja/boostedblob.git)
+ boostedblob/path.py:491: error: Need type annotation for "fut"
+ boostedblob/syncing.py:51: error: Need type annotation for "dst_files"
+ boostedblob/syncing.py:51: error: Need type annotation for "src_files"

starlette (https://github.com/encode/starlette.git)
+ starlette/requests.py:255: error: Need type annotation for "message"

pytest (https://github.com/pytest-dev/pytest.git)
+ src/_pytest/_code/code.py:501: error: Value of type _OptExcInfo? is not indexable  [index]
+ src/_pytest/_code/code.py:502: error: Value of type _OptExcInfo? is not indexable  [index]
+ src/_pytest/_code/code.py:503: error: Value of type _OptExcInfo? is not indexable  [index]
+ src/_pytest/_code/code.py:504: error: Value of type _OptExcInfo? is not indexable  [index]
+ src/_pytest/fixtures.py:1140: error: Value of type _OptExcInfo? is not indexable  [index]
+ src/_pytest/pytester.py:1327: error: No overload variant of "Popen" matches argument types "Sequence[Union[str, PathLike[str]]]", "Union[int, TextIO]", "Union[int, TextIO]", "Dict[str, Any]"  [call-overload]
+ src/_pytest/pytester.py:1327: note:     <4 more similar overloads not shown, out of 6 total overloads>
+ src/_pytest/pytester.py:1327: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: bool = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, text: Optional[bool] = ..., encoding: Optional[str] = ..., errors: str) -> Popen[str]
+ src/_pytest/pytester.py:1327: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: bool = ..., startupinfo: Optional[Any] = ..., creationflags: int = ..., restore_signals: bool = ..., start_new_session: bool = ..., pass_fds: Any = ..., *, text: Optional[bool] = ..., encoding: str, errors: Optional[str] = ...) -> Popen[str]
+ src/_pytest/pytester.py:1327: note: Possible overload variants:
+ testing/test_runner.py:792: error: No overload variant of "Popen" matches argument types "List[str]", "int"  [call-overload]
+ testing/test_runner.py:792: note:     <4 more non-matching overloads not shown>
+ testing/test_runner.py:792: note:     def [AnyStr in (str, bytes)] __new__(cls, args: _CMD?, bufsize: int = ..., executable: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., stdin: None = ..., stdout: None = ..., stderr: None = ..., preexec_fn: Optional[Callable[[], Any]] = ..., close_fds: bool = ..., shell: bool = ..., cwd: Union[str, bytes, PathLike[str], PathLike[bytes], None] = ..., env: None = ..., universal_newlines: Literal[False] = ..., startupinfo: Optional[Any] = ..., ```

hauntsaninja · 2021-06-01T08:05:06Z

Hmm, looks like mypy and pyright both don't accept PEP 604 syntax in type aliases (at least, not without PEP 613). I guess I knew this about mypy python/mypy#9880 (comment)

hoefling · 2021-08-11T10:41:22Z

@hauntsaninja I can share a script I've used to update our internal stubs. Requires libcst though. Usage: paste the code in a run.py in the typeshed repo root and execute.

import functools
import logging
import libcst as cst
from pathlib import Path

rootdir = Path(__file__).parent / "stdlib"
none_node = cst.Name(value="None")


def main(where: Path = rootdir) -> None:
    try:
        from rich.logging import RichHandler

        fmt = "{message} ({funcName}, {filename}:{lineno})"
        handlers = [RichHandler()]
    except ImportError:
        fmt = "{asctime} [{levelname:5}] {message} ({funcName}, {filename}:{lineno})"
        handlers = None
    logging.basicConfig(
        level=logging.DEBUG,
        format=fmt,
        style="{",
        handlers=handlers,
    )
    logging.logThreads = 0
    logging.logProcesses = 0

    rewriter = PEP604Rewrite()
    for stub in rootdir.rglob("*.pyi"):
        logging.info("current file: %s", stub)
        stub_tree = cst.parse_module(stub.read_text())
        modified_stub_tree = stub_tree.visit(rewriter)
        stub.write_text(modified_stub_tree.code)


def bit_or(left: cst.BaseExpression, right: cst.BaseExpression) -> cst.BinaryOperation:
    return cst.BinaryOperation(
        operator=cst.BitOr(),
        left=left,
        right=right,
    )


def bit_or_none(node: cst.BaseExpression) -> cst.BinaryOperation:
    return cst.BinaryOperation(
        operator=cst.BitOr(),
        left=node,
        right=none_node,
    )


class PEP604Rewrite(cst.CSTTransformer):
    def transform(self, node: cst.Subscript) -> cst.CSTNode:
        if node.value.value == "Union":
            names = [element.slice.value for element in node.slice]
            new_node = functools.reduce(bit_or, names)
            logging.debug(
                "transformed Union: %r", cst.parse_module("").code_for_node(new_node)
            )
            return new_node

        if node.value.value == "Optional":
            new_node = bit_or_none(node.slice[0].slice.value)
            logging.debug(
                "transformed Optional: %r", cst.parse_module("").code_for_node(new_node)
            )
            return new_node
        return node

    def leave_ImportFrom(
        self, original_node: cst.ImportFrom, updated_node: cst.ImportFrom
    ) -> cst.CSTNode:
        if updated_node.module is not None and updated_node.module.value == "typing":
            # drop Optional & Union imports if present
            names = [
                alias
                for alias in updated_node.names
                if alias.name.value not in ("Union", "Optional")
            ]
            if names:
                # case `from typing import ..., Union`
                names[-1] = names[-1].with_changes(comma=cst.MaybeSentinel.DEFAULT)
                return updated_node.with_changes(names=names)
            else:
                # case `from typing import Union` etc
                return cst.RemoveFromParent()
        return updated_node

    def leave_Subscript(
        self, original_node: cst.Subscript, updated_node: cst.Subscript
    ) -> cst.CSTNode:
        logging.debug("%r", cst.parse_module("").code_for_node(updated_node))
        return self.transform(updated_node)


if __name__ == "__main__":
    import sys

    sys.exit(main())

srittau · 2021-08-11T11:28:45Z

I think @Akuli has now converted most of typeshed.

hoefling · 2021-08-11T12:33:24Z

@srittau then I'm too late with the script; pity! Just disregard the above comment then.

Akuli · 2021-08-11T14:10:42Z

I converted most stubs in #5872. If you want, you can run both scripts and compare the results. My script did weird things to some files, and I just did some files manually after running the script.

hoefling · 2021-08-11T15:40:18Z

@Akuli I executed the script over the current master and the only files touched are those from @python2, so I think there's nothing left to do. Good job!

srittau · 2021-08-11T17:25:44Z

I think we could convert the Python 2 files as well, couldn't we?

hoefling · 2021-08-11T17:31:28Z

@srittau this I can do :-)

Akuli · 2021-08-28T12:08:24Z

@hoefling Would you like to modify your script to instead check whether the stubs use the new syntax, and add it to CI? If you want, I can do that too.

hoefling · 2021-08-28T12:34:53Z

@Akuli no worries about pinging me! The script I pasted here is an overkill for the task anyway, since it uses libcst to actually modify the stubs. What @hauntsaninja proposes in #5903 is a much simpler solution.

hauntsaninja marked this pull request as draft June 1, 2021 07:39

hauntsaninja added 16 commits June 1, 2021 00:41

fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+), ([\w\.]+)\]' '$1 …

f719a0a

…| $2'

fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+), ([\w\.]+), ([\w\.…

2be1154

…]+)\]' '$1 | $2 | $3'

fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+), ([\w\.]+), ([\w\.…

882274b

…]+), ([\w\.]+)\]' '$1 | $2 | $3 | $4'

fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+)(\[[^\[\]]+\])?, ([…

ad937c8

…\w\.]+)(\[[^\[\]]+\])?\]' '$1$2 | $3$4'

fastmod -d stdlib --accept-all '\bUnion\[([\w\.]+)(\[[^\[\]]+\])?, ([…

73111c6

…\w\.]+)(\[[^\[\]]+\])?, ([\w\.]+)(\[[^\[\]]+\])?\]' '$1$2 | $3$4 | $5$6'

fastmod -d stdlib --accept-all '\bOptional\[([\w\.\| ]+)\]' '$1 | None'

c156d70

fastmod -d stdlib --accept-all '\bOptional\[([\w\.\| ]+)\[([^\[\]]+)\…

35fc69d

…]\]' '$1[$2] | None'

for x in $(rg '\bOptional\b' -c | grep stdlib | pyp 'x.split(":")[0] …

4d1ce15

…if x.endswith(":1") else None'); do fastmod -g $x --accept-all '\s*from typing import Optional$' ''; done

for x in $(rg '\bOptional\b' -c | grep stdlib | pyp 'x.split(":")[0] …

5a6c032

…if x.endswith(":1") else None'); do fastmod -g $x --accept-all '^\s+Optional,$' ''; done

for x in $(rg '\bOptional\b' -c | grep stdlib | pyp 'x.split(":")[0] …

e4c086b

…if x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import )Optional, (.*)' '$1$2'; done

for x in $(rg '\bOptional\b' -c | grep stdlib | pyp 'x.split(":")[0] …

3a8801f

…if x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import.*), Optional' '$1'; done

for x in $(rg '\bUnion\b' -c | grep stdlib | pyp 'x.split(":")[0] if …

7ae0a9e

…x.endswith(":1") else None'); do fastmod -g $x --accept-all '\s*from typing import Union$' ''; done

for x in $(rg '\bUnion\b' -c | grep stdlib | pyp 'x.split(":")[0] if …

67f8d44

…x.endswith(":1") else None'); do fastmod -g $x --accept-all '^\s+Union,$' ''; done

for x in $(rg '\bUnion\b' -c | grep stdlib | pyp 'x.split(":")[0] if …

5a90411

…x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import )Union, (.*)' '$1$2'; done

for x in $(rg '\bUnion\b' -c | grep stdlib | pyp 'x.split(":")[0] if …

7c43083

…x.endswith(":1") else None'); do fastmod -g $x --accept-all '(from typing import.*), Union' '$1'; done

black and isort

285129a

hauntsaninja force-pushed the pep604 branch from 9787f63 to 285129a Compare June 1, 2021 07:43

This comment has been minimized.

Sign in to view

hauntsaninja closed this Jun 1, 2021

hauntsaninja mentioned this pull request Jan 19, 2022

Enable TypeAlias check by default PyCQA/flake8-pyi#101

Merged

hoefling mentioned this pull request Nov 13, 2022

Setup flake8-pyi to lint stubs typeddjango/django-stubs#128

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stdlib: use PEP 604 in most places #5564

stdlib: use PEP 604 in most places #5564

hauntsaninja commented Jun 1, 2021

This comment has been minimized.

github-actions bot commented Jun 1, 2021

hauntsaninja commented Jun 1, 2021

hoefling commented Aug 11, 2021 •

edited

Loading

srittau commented Aug 11, 2021

hoefling commented Aug 11, 2021

Akuli commented Aug 11, 2021

hoefling commented Aug 11, 2021

srittau commented Aug 11, 2021

hoefling commented Aug 11, 2021

Akuli commented Aug 28, 2021

hoefling commented Aug 28, 2021

stdlib: use PEP 604 in most places #5564

stdlib: use PEP 604 in most places #5564

Conversation

hauntsaninja commented Jun 1, 2021

This comment has been minimized.

github-actions bot commented Jun 1, 2021

hauntsaninja commented Jun 1, 2021

hoefling commented Aug 11, 2021 • edited Loading

srittau commented Aug 11, 2021

hoefling commented Aug 11, 2021

Akuli commented Aug 11, 2021

hoefling commented Aug 11, 2021

srittau commented Aug 11, 2021

hoefling commented Aug 11, 2021

Akuli commented Aug 28, 2021

hoefling commented Aug 28, 2021

hoefling commented Aug 11, 2021 •

edited

Loading