Add PEP 646 integration

The :class:`.Row` object now no longer makes use of an intermediary
``Tuple`` in order to represent its individual element types; instead,
the individual element types are present directly, via new :pep:`646`
integration, now available in more recent versions of Mypy.  Mypy
1.7 or greater is now required for statements, results and rows
to be correctly typed.   Pull request courtesy Yurii Karabas.

Fixes: sqlalchemy#10635
Closes: sqlalchemy#10634
Pull-request: sqlalchemy#10634
Pull-request-sha: 430785c

Change-Id: Ibd0ae31a98b4ea69dcb89f970e640920b2be6c48

doc/build/changelog/migration_21.rst

@@ -10,6 +10,76 @@ What's New in SQLAlchemy 2.1?
     version 2.1.
 
 
+.. _change_10635:
+
+``Row`` now represents individual column types directly without ``Tuple``
+--------------------------------------------------------------------------
+
+SQLAlchemy 2.0 implemented a broad array of :pep:`484` typing throughout
+all components, including a new ability for row-returning statements such
+as :func:`_sql.select` to maintain track of individual column types, which
+were then passed through the execution phase onto the :class:`_engine.Result`
+object and then to the individual :class:`_engine.Row` objects.   Described
+at :ref:`change_result_typing_20`, this approach solved several issues
+with statement / row typing, but some remained unsolvable.  In 2.1, one
+of those issues, that the individual column types needed to be packaged
+into a ``typing.Tuple``, is now resolved using new :pep:`646` integration,
+which allows for tuple-like types that are not actually typed as ``Tuple``.
+
+In SQLAlchemy 2.0, a statement such as::
+
+    stmt = select(column("x", Integer), column("y", String))
+
+Would be typed as::
+
+    Select[Tuple[int, str]]
+
+In 2.1, it's now typed as::
+
+    Select[int, str]
+
+When executing ``stmt``, the :class:`_engine.Result` and :class:`_engine.Row`
+objects will be typed as ``Result[int, str]`` and ``Row[int, str]``, respectively.
+The prior workaround using :attr:`_engine.Row._t` to type as a real ``Tuple``
+is no longer needed and projects can migrate off this pattern.
+
+Mypy users will need to make use of **Mypy 1.7 or greater** for pep-646
+integration to be available.
+
+Limitations
+^^^^^^^^^^^
+
+Not yet solved by pep-646 or any other pep is the ability for an arbitrary
+number of expressions within :class:`_sql.Select` and others to be mapped to
+row objects, without stating each argument position explicitly within typing
+annotations.   To work around this issue, SQLAlchemy makes use of automated
+"stub generation" tools to generate hardcoded mappings of different numbers of
+positional arguments to constructs like :func:`_sql.select` to resolve to
+individual ``Unpack[]`` expressions (in SQLAlchemy 2.0, this generation
+prodcued ``Tuple[]`` annotations instead).  This means that there are arbitrary
+limits on how many specific column expressions will be typed within the
+:class:`_engine.Row` object, without restoring to ``Any`` for remaining
+expressions; for :func:`_sql.select`, it's currently ten expressions, and
+for DML expresions like :func:`_dml.insert` that use :meth:`_dml.Insert.returning`,
+it's eight.    If and when a new pep that provides a ``Map`` operator
+to pep-646 is proposed, this limitation can be lifted. [1]_  Originally, it was
+mistakenly assumed that this limitation prevented pep-646 from being usable at all,
+however, the ``Unpack`` construct does in fact replace everything that
+was done using ``Tuple`` in 2.0.
+
+An additional limitation for which there is no proposed solution is that
+there's no way for the name-based attributes on :class:`_engine.Row` to be
+automatically typed, so these continue to be typed as ``Any`` (e.g. ``row.x``
+and ``row.y`` for the above example).   With current language features,
+this could only be fixed by having an explicit class-based construct that
+allows one to compose an explicit :class:`_engine.Row` with explicit fields
+up front, which would be verbose and not automatic.
+
+.. [1] https://github.com/python/typing/discussions/1001#discussioncomment-1897813
+
+:ticket:`10635`
+
+
 .. _change_10197:
 
 Asyncio "greenlet" dependency no longer installs by default

doc/build/changelog/unreleased_21/10296.rst

@@ -7,4 +7,4 @@
     be imported only when the asyncio extension is first imported.
     Alternatively, the ``greenlet`` library is still imported lazily on
     first use to support use case that don't make direct use of the
-    SQLAlchemy asyncio extension.
+    SQLAlchemy asyncio extension.

doc/build/changelog/unreleased_21/10635.rst

@@ -0,0 +1,14 @@
+.. change::
+    :tags: typing, feature
+    :tickets: 10635
+
+    The :class:`.Row` object now no longer makes use of an intermediary
+    ``Tuple`` in order to represent its individual element types; instead,
+    the individual element types are present directly, via new :pep:`646`
+    integration, now available in more recent versions of Mypy.  Mypy
+    1.7 or greater is now required for statements, results and rows
+    to be correctly typed.   Pull request courtesy Yurii Karabas.
+
+    .. seealso::
+
+        :ref:`change_10635`

doc/build/changelog/whatsnew_20.rst

@@ -75,6 +75,7 @@ result set.
    for the 2.0 series. Typing details are subject to change however
    significant backwards-incompatible changes are not planned.
 
+.. _change_result_typing_20:
 
 SQL Expression / Statement / Result Set Typing
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

lib/sqlalchemy/engine/_py_row.py

@@ -18,10 +18,11 @@
 from typing import Tuple
 from typing import Type
 
+from ..util.typing import TupleAny
+
 if typing.TYPE_CHECKING:
     from .result import _KeyType
     from .result import _ProcessorsType
-    from .result import _RawRowType
     from .result import _TupleGetterType
     from .result import ResultMetaData
 
@@ -33,14 +34,14 @@ class BaseRow:
 
     _parent: ResultMetaData
     _key_to_index: Mapping[_KeyType, int]
-    _data: _RawRowType
+    _data: TupleAny
 
     def __init__(
         self,
         parent: ResultMetaData,
         processors: Optional[_ProcessorsType],
         key_to_index: Mapping[_KeyType, int],
-        data: _RawRowType,
+        data: TupleAny,
     ):
         """Row objects are constructed by CursorResult objects."""
         object.__setattr__(self, "_parent", parent)

lib/sqlalchemy/engine/base.py

@@ -43,6 +43,9 @@
 from .. import util
 from ..sql import compiler
 from ..sql import util as sql_util
+from ..util.typing import TupleAny
+from ..util.typing import TypeVarTuple
+from ..util.typing import Unpack
 
 if typing.TYPE_CHECKING:
     from . import CursorResult
@@ -80,6 +83,7 @@
 
 
 _T = TypeVar("_T", bound=Any)
+_Ts = TypeVarTuple("_Ts")
 _EMPTY_EXECUTION_OPTS: _ExecuteOptions = util.EMPTY_DICT
 NO_OPTIONS: Mapping[str, Any] = util.EMPTY_DICT
 
@@ -1258,7 +1262,7 @@ def close(self) -> None:
     @overload
     def scalar(
         self,
-        statement: TypedReturnsRows[Tuple[_T]],
+        statement: TypedReturnsRows[_T],
         parameters: Optional[_CoreSingleExecuteParams] = None,
         *,
         execution_options: Optional[CoreExecuteOptionsParameter] = None,
@@ -1307,7 +1311,7 @@ def scalar(
     @overload
     def scalars(
         self,
-        statement: TypedReturnsRows[Tuple[_T]],
+        statement: TypedReturnsRows[_T],
         parameters: Optional[_CoreAnyExecuteParams] = None,
         *,
         execution_options: Optional[CoreExecuteOptionsParameter] = None,
@@ -1352,11 +1356,11 @@ def scalars(
     @overload
     def execute(
         self,
-        statement: TypedReturnsRows[_T],
+        statement: TypedReturnsRows[Unpack[_Ts]],
         parameters: Optional[_CoreAnyExecuteParams] = None,
         *,
         execution_options: Optional[CoreExecuteOptionsParameter] = None,
-    ) -> CursorResult[_T]:
+    ) -> CursorResult[Unpack[_Ts]]:
         ...
 
     @overload
@@ -1366,7 +1370,7 @@ def execute(
         parameters: Optional[_CoreAnyExecuteParams] = None,
         *,
         execution_options: Optional[CoreExecuteOptionsParameter] = None,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         ...
 
     def execute(
@@ -1375,7 +1379,7 @@ def execute(
         parameters: Optional[_CoreAnyExecuteParams] = None,
         *,
         execution_options: Optional[CoreExecuteOptionsParameter] = None,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         r"""Executes a SQL statement construct and returns a
         :class:`_engine.CursorResult`.
 
@@ -1424,7 +1428,7 @@ def _execute_function(
         func: FunctionElement[Any],
         distilled_parameters: _CoreMultiExecuteParams,
         execution_options: CoreExecuteOptionsParameter,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         """Execute a sql.FunctionElement object."""
 
         return self._execute_clauseelement(
@@ -1495,7 +1499,7 @@ def _execute_ddl(
         ddl: ExecutableDDLElement,
         distilled_parameters: _CoreMultiExecuteParams,
         execution_options: CoreExecuteOptionsParameter,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         """Execute a schema.DDL object."""
 
         exec_opts = ddl._execution_options.merge_with(
@@ -1590,7 +1594,7 @@ def _execute_clauseelement(
         elem: Executable,
         distilled_parameters: _CoreMultiExecuteParams,
         execution_options: CoreExecuteOptionsParameter,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         """Execute a sql.ClauseElement object."""
 
         execution_options = elem._execution_options.merge_with(
@@ -1663,7 +1667,7 @@ def _execute_compiled(
         compiled: Compiled,
         distilled_parameters: _CoreMultiExecuteParams,
         execution_options: CoreExecuteOptionsParameter = _EMPTY_EXECUTION_OPTS,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         """Execute a sql.Compiled object.
 
         TODO: why do we have this?   likely deprecate or remove
@@ -1713,7 +1717,7 @@ def exec_driver_sql(
         statement: str,
         parameters: Optional[_DBAPIAnyExecuteParams] = None,
         execution_options: Optional[CoreExecuteOptionsParameter] = None,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         r"""Executes a string SQL statement on the DBAPI cursor directly,
         without any SQL compilation steps.
 
@@ -1795,7 +1799,7 @@ def _execute_context(
         execution_options: _ExecuteOptions,
         *args: Any,
         **kw: Any,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         """Create an :class:`.ExecutionContext` and execute, returning
         a :class:`_engine.CursorResult`."""
 
@@ -1854,7 +1858,7 @@ def _exec_single_context(
         context: ExecutionContext,
         statement: Union[str, Compiled],
         parameters: Optional[_AnyMultiExecuteParams],
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         """continue the _execute_context() method for a single DBAPI
         cursor.execute() or cursor.executemany() call.
 
@@ -1994,7 +1998,7 @@ def _exec_insertmany_context(
         self,
         dialect: Dialect,
         context: ExecutionContext,
-    ) -> CursorResult[Any]:
+    ) -> CursorResult[Unpack[TupleAny]]:
         """continue the _execute_context() method for an "insertmanyvalues"
         operation, which will invoke DBAPI
         cursor.execute() one or more times with individual log and

lib/sqlalchemy/engine/cursor.py

@@ -28,7 +28,6 @@
 from typing import Sequence
 from typing import Tuple
 from typing import TYPE_CHECKING
-from typing import TypeVar
 from typing import Union
 
 from .result import IteratorResult
@@ -53,6 +52,9 @@
 from ..util import compat
 from ..util.typing import Literal
 from ..util.typing import Self
+from ..util.typing import TupleAny
+from ..util.typing import TypeVarTuple
+from ..util.typing import Unpack
 
 
 if typing.TYPE_CHECKING:
@@ -71,7 +73,7 @@
     from ..sql.type_api import _ResultProcessorType
 
 
-_T = TypeVar("_T", bound=Any)
+_Ts = TypeVarTuple("_Ts")
 
 
 # metadata entry tuple indexes.
@@ -344,7 +346,7 @@ def _adapt_to_context(self, context: ExecutionContext) -> ResultMetaData:
 
     def __init__(
         self,
-        parent: CursorResult[Any],
+        parent: CursorResult[Unpack[TupleAny]],
         cursor_description: _DBAPICursorDescription,
     ):
         context = parent.context
@@ -928,49 +930,53 @@ class ResultFetchStrategy:
     alternate_cursor_description: Optional[_DBAPICursorDescription] = None
 
     def soft_close(
-        self, result: CursorResult[Any], dbapi_cursor: Optional[DBAPICursor]
+        self,
+        result: CursorResult[Unpack[TupleAny]],
+        dbapi_cursor: Optional[DBAPICursor],
     ) -> None:
         raise NotImplementedError()
 
     def hard_close(
-        self, result: CursorResult[Any], dbapi_cursor: Optional[DBAPICursor]
+        self,
+        result: CursorResult[Unpack[TupleAny]],
+        dbapi_cursor: Optional[DBAPICursor],
     ) -> None:
         raise NotImplementedError()
 
     def yield_per(
         self,
-        result: CursorResult[Any],
+        result: CursorResult[Unpack[TupleAny]],
         dbapi_cursor: Optional[DBAPICursor],
         num: int,
     ) -> None:
         return
 
     def fetchone(
         self,
-        result: CursorResult[Any],
+        result: CursorResult[Unpack[TupleAny]],
         dbapi_cursor: DBAPICursor,
         hard_close: bool = False,
     ) -> Any:
         raise NotImplementedError()
 
     def fetchmany(
         self,
-        result: CursorResult[Any],
+        result: CursorResult[Unpack[TupleAny]],
         dbapi_cursor: DBAPICursor,
         size: Optional[int] = None,
     ) -> Any:
         raise NotImplementedError()
 
     def fetchall(
         self,
-        result: CursorResult[Any],
+        result: CursorResult[Unpack[TupleAny]],
         dbapi_cursor: DBAPICursor,
     ) -> Any:
         raise NotImplementedError()
 
     def handle_exception(
         self,
-        result: CursorResult[Any],
+        result: CursorResult[Unpack[TupleAny]],
         dbapi_cursor: Optional[DBAPICursor],
         err: BaseException,
     ) -> NoReturn:
@@ -1375,7 +1381,7 @@ def null_dml_result() -> IteratorResult[Any]:
     return it
 
 
-class CursorResult(Result[_T]):
+class CursorResult(Result[Unpack[_Ts]]):
     """A Result that is representing state from a DBAPI cursor.
 
     .. versionchanged:: 1.4  The :class:`.CursorResult``
@@ -2108,7 +2114,9 @@ def _fetchmany_impl(self, size=None):
     def _raw_row_iterator(self):
         return self._fetchiter_impl()
 
-    def merge(self, *others: Result[Any]) -> MergedResult[Any]:
+    def merge(
+        self, *others: Result[Unpack[TupleAny]]
+    ) -> MergedResult[Unpack[TupleAny]]:
         merged_result = super().merge(*others)
         setup_rowcounts = self.context._has_rowcount
         if setup_rowcounts: