Fix tuple subtyping #16073
Closed
Fix tuple subtyping #16073
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hauntsaninja
force-pushed
the
tuple-subtyping
branch
from
069e8e6
to
4031bdd
Compare
hauntsaninja
force-pushed
the
tuple-subtyping
branch
from
4031bdd
to
e5b617c
Compare
This comment has been minimized.
This comment has been minimized.
|
Wow, lots of primer, and potentially almost all good. New error cases are pandas, optuna, sockeye and maybe vision. |
|
Diff from mypy_primer, showing the effect of this PR on open source code: mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/coretypes/multidict.py:153: error: Unused "type: ignore" comment [unused-ignore]
+ mitmproxy/coretypes/multidict.py:167: error: Unused "type: ignore" comment [unused-ignore]
+ mitmproxy/tools/console/grideditor/base.py:170: error: Unused "type: ignore" comment [unused-ignore]
sockeye (https://github.com/awslabs/sockeye)
+ sockeye/data_io.py:695: error: Argument 1 to "combine_means" has incompatible type "tuple[Any, ...]"; expected "list[float | None]" [arg-type]
+ sockeye/data_io.py:704: error: Argument 2 to "combine_means" has incompatible type "tuple[Any, ...]"; expected "list[int]" [arg-type]
+ sockeye/data_io.py:705: error: Argument 3 to "combine_stds" has incompatible type "tuple[Any, ...]"; expected "list[int]" [arg-type]
- sockeye/training.py:86: error: Unused "type: ignore" comment [unused-ignore]
+ sockeye/translate.py:190: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", variable has type "list[str]") [assignment]
pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/apply.py:1577: error: Incompatible return value type (got "tuple[defaultdict[Any, Any], tuple[Any, ...], ndarray[Any, dtype[signedinteger[Any]]]]", expected "tuple[dict[Any, Any], list[str], ndarray[Any, dtype[signedinteger[Any]]]]") [return-value]
+ pandas/io/formats/format.py:944: error: List comprehension has incompatible type List[list[Any]]; expected List[tuple[Any, ...]] [misc]
+ pandas/io/formats/format.py:950: error: List comprehension has incompatible type List[list[str]]; expected List[tuple[Any, ...]] [misc]
+ pandas/io/formats/format.py:954: error: Incompatible return value type (got "list[tuple[Any, ...]]", expected "list[list[str]]") [return-value]
+ pandas/core/generic.py:7815: error: Incompatible types in assignment (expression has type "list[Never]", variable has type "tuple[Any, ...]") [assignment]
+ pandas/core/generic.py:7830: error: Incompatible types in assignment (expression has type "list[Never]", variable has type "tuple[Any, ...]") [assignment]
+ pandas/core/reshape/melt.py:211: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", variable has type "list[Any]") [assignment]
+ pandas/core/reshape/concat.py:866: error: Incompatible types in assignment (expression has type "Index", variable has type "tuple[Any, ...]") [assignment]
+ pandas/core/reshape/concat.py:866: error: Argument 1 to "ensure_index" has incompatible type "tuple[Any, ...]"; expected "ExtensionArray | ndarray[Any, Any] | Index | Series | list[Any] | range" [arg-type]
+ pandas/core/indexes/multi.py:3938: error: Argument 1 to "append" of "list" has incompatible type "list[Any]"; expected "tuple[Any, ...]" [arg-type]
+ pandas/core/indexes/multi.py:3945: error: Argument 1 to "append" of "list" has incompatible type "list[Any]"; expected "tuple[Any, ...]" [arg-type]
Expression (https://github.com/cognitedata/Expression)
- expression/collections/block.py:814: error: Overloaded function implementation cannot produce return type of signature 1 [misc]
- expression/collections/block.py:814: error: Overloaded function implementation cannot produce return type of signature 2 [misc]
- expression/collections/block.py:814: error: Overloaded function implementation cannot produce return type of signature 3 [misc]
- expression/extra/parser.py:219: error: Overloaded function implementation cannot produce return type of signature 1 [misc]
- expression/extra/parser.py:219: error: Overloaded function implementation cannot produce return type of signature 2 [misc]
steam.py (https://github.com/Gobot1234/steam.py)
- steam/_const.py:127: error: Incompatible default for argument "unpacker" (default has type "Callable[[Buffer, int], tuple[Any, ...]]", argument has type "Callable[[Buffer], tuple[int]]") [assignment]
- steam/trade.py:322: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", base class "Asset" defined the type as "tuple[str, str, str, str, str, str, str]") [assignment]
- steam/ext/csgo/backpack.py:260: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", base class "Asset" defined the type as "tuple[str, str, str, str, str, str, str]") [assignment]
- steam/ext/csgo/backpack.py:327: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", base class "Asset" defined the type as "tuple[str, str, str, str, str, str, str]") [assignment]
- steam/ext/csgo/backpack.py:327: error: Cannot determine type of "REPR_ATTRS" [has-type]
- steam/ext/csgo/state.py:34: error: Incompatible default for argument "_unpacker" (default has type "Callable[[Buffer, int], tuple[Any, ...]]", argument has type "Callable[[bytes], tuple[float]]") [assignment]
- steam/ext/tf2/backpack.py:80: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", base class "Asset" defined the type as "tuple[str, str, str, str, str, str, str]") [assignment]
psycopg (https://github.com/psycopg/psycopg)
+ tests/test_copy.py:418: error: Unused "type: ignore" comment [unused-ignore]
+ tests/test_copy_async.py:415: error: Unused "type: ignore" comment [unused-ignore]
+ tests/crdb/test_copy.py:123: error: Unused "type: ignore" comment [unused-ignore]
+ tests/crdb/test_copy_async.py:123: error: Unused "type: ignore" comment [unused-ignore]
pylint (https://github.com/pycqa/pylint)
+ pylint/__pkginfo__.py:40: error: Unused "type: ignore" comment [unused-ignore]
jinja (https://github.com/pallets/jinja)
+ src/jinja2/environment.py:1503: error: Unused "type: ignore" comment [unused-ignore]
mypy (https://github.com/python/mypy)
+ mypy/main.py:1362: error: Unused "type: ignore" comment [unused-ignore]
prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/server/utilities/database.py:631: error: Invalid self argument "URL" to attribute function "get_dialect" with type "Callable[[URL, bool], type[Dialect]]" [misc]
zetta_utils (https://github.com/ZettaAI/zetta_utils)
+ zetta_utils/layer/db_layer/layer.py:55: error: Unused "type: ignore" comment [unused-ignore]
vision (https://github.com/pytorch/vision)
+ torchvision/datasets/video_utils.py:139: error: Need type annotation for "video_fps" (hint: "video_fps: List[<type>] = ...") [var-annotated]
operator (https://github.com/canonical/operator)
- ops/storage.py:191: error: Incompatible types in "yield" (actual type "tuple[Any, ...]", expected type "tuple[str, str, str]") [misc]
optuna (https://github.com/optuna/optuna)
+ optuna/importance/_fanova/_tree.py:64: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", variable has type "list[Any]") [assignment]
jax (https://github.com/google/jax)
+ jax/_src/interpreters/mlir.py:693: error: Unused "type: ignore" comment [unused-ignore]
+ jax/_src/debugging.py:535: error: Unused "type: ignore" comment [unused-ignore]
+ jax/_src/scipy/special.py:293: error: Unused "type: ignore" comment [unused-ignore]
+ jax/experimental/sparse/bcoo.py:670: error: Unused "type: ignore" comment [unused-ignore]
+ jax/experimental/jax2tf/tests/model_harness.py:294: error: Unused "type: ignore" comment [unused-ignore]
streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/testing/element_tree.py: note: At top level:
+ lib/streamlit/testing/element_tree.py:339: error: Unused "type: ignore" comment [unused-ignore]
mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
+ pymongo/mongo_client.py:1685: error: Unused "type: ignore" comment [unused-ignore]
discord.py (https://github.com/Rapptz/discord.py)
- discord/app_commands/transformers.py:641: error: Incompatible types in assignment (expression has type "tuple[type[Any], ...]", variable has type "tuple[type[Any]]") [assignment]
- discord/ext/tasks/__init__.py:488: error: Incompatible types in assignment (expression has type "tuple[type, ...]", variable has type "tuple[type[OSError], type[GatewayNotFound], type[ConnectionClosed], type[ClientError], type[TimeoutError]]") [assignment]
- discord/ext/tasks/__init__.py:513: error: Incompatible types in assignment (expression has type "tuple[type, ...]", variable has type "tuple[type[OSError], type[GatewayNotFound], type[ConnectionClosed], type[ClientError], type[TimeoutError]]") [assignment]
ibis (https://github.com/ibis-project/ibis)
- ibis/util.py:126: error: Incompatible return value type (got "tuple[Any, ...]", expected "tuple[V]") [return-value]
- ibis/util.py:128: error: Incompatible return value type (got "tuple[V, ...]", expected "tuple[V]") [return-value]
- ibis/backends/oracle/__init__.py:51: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", base class "ExprTranslator" defined the type as "tuple[type[Lag], type[Lead], type[DenseRank], type[MinRank], type[FirstValue], type[LastValue], type[PercentRank], type[CumeDist], type[NTile]]") [assignment]
- ibis/backends/trino/compiler.py:23: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", base class "ExprTranslator" defined the type as "tuple[type[Lag], type[Lead], type[DenseRank], type[MinRank], type[FirstValue], type[LastValue], type[PercentRank], type[CumeDist], type[NTile]]") [assignment]
- ibis/backends/snowflake/__init__.py:58: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", base class "ExprTranslator" defined the type as "tuple[type[Lag], type[Lead], type[DenseRank], type[MinRank], type[FirstValue], type[LastValue], type[PercentRank], type[CumeDist], type[NTile]]") [assignment]
bokeh (https://github.com/bokeh/bokeh)
+ src/bokeh/server/tornado.py:449: error: Unused "type: ignore" comment [unused-ignore]
+ src/bokeh/server/tornado.py:452: error: Unused "type: ignore" comment [unused-ignore]
|
ilevkivskyi
reviewed
Sep 9, 2023
| if isinstance(right, TupleType): | ||
| return self._is_subtype(left, right.partial_fallback) and self._is_subtype( | ||
| left, mypy.typeops.tuple_fallback(right) | ||
| ) |
There was a problem hiding this comment.
Sorry, but this is wrong. IIUC this will make tuple[object, ...] a subtype of tuple[int, str]. While even tuple[int, ...] shoul not be a subtype of tuple[int, int]. Really only tuple[Any, ...] should be a subtype of say tuple[int, int]. This is the same as we consider *args: Any in callables very differently from *args: object or *args: int.
Note btw this question will appear even broader in context of my upcoming PR for variadic types, e.g. should we also special case Foo[int, *tuple[Any, ...], str] similarly in spirit to extending the special-casing of trivial arguments to trivial suffix in #15913. (Btw I would say yes, but it should be a separate PR after we gain more experience).
Also I bet most of the unused type ignores in mypy primer are from situations like:
v: tuple[int, ...]
if len(v) == 2:
x: tuple[int, int] = v(as e.g. for ignore in mypy itself). See also #1178. The proper way to solve such issues is by finally completing #10367. Unfortunately author is not responding anymore, so one of us may need to complete the PR, since it is an important one.
ilevkivskyi
reviewed
Sep 9, 2023
| # The reason for error in subtle: actual can be empty, formal cannot. | ||
| reveal_type(f(var_len_tuple)) # N: Revealed type is "Tuple[builtins.str, Unpack[builtins.tuple[builtins.int, ...]]]" \ | ||
| # E: Argument 1 to "f" has incompatible type "Tuple[int, ...]"; expected "Tuple[int, Unpack[Tuple[int, ...]]]" | ||
| reveal_type(f(var_len_tuple)) # N: Revealed type is "Tuple[builtins.str, Unpack[builtins.tuple[builtins.int, ...]]]" |
There was a problem hiding this comment.
The error is correct, because () is an element of tuple[int, ...] but it is not an element of tuple[int, *tuple[int, ...]] (this must have at least one integer in the tuple). So tuple[int, ...] is not a subtype of tuple[int, *tuple[int, ...]]
|
Thanks, this was silly of me! I guess action items are:
|
hauntsaninja
added a commit
to hauntsaninja/mypy
that referenced
this pull request
Sep 14, 2023
Follow up to python#16073 and python#16076 Fix needed for https://github.com/python/mypy/pull/16053/files#r1316481395 I add test cases that would have caught my previous incorrect PR. I add an explicit case for the new desirable behaviour we see with zip.
hauntsaninja
added a commit
that referenced
this pull request
Sep 14, 2023
Follow up to #16073 and #16076 Fix needed for https://github.com/python/mypy/pull/16053/files#r1316481395 I add test cases that would have caught my previous incorrect PR. I add an explicit case for the new desirable behaviour we see with zip.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix needed for https://github.com/python/mypy/pull/16053/files#r1316481395
Both test changes naively look like improvements, but I didn't quite follow the comment about empty actual.
The history here goes to bf9cb22