Issue python#27243: Fix __aiter__ protocol

Author: 1st1

Doc/glossary.rst

@@ -76,13 +76,12 @@ Glossary
 
    asynchronous iterable
       An object, that can be used in an :keyword:`async for` statement.
-      Must return an :term:`awaitable` from its :meth:`__aiter__` method,
-      which should in turn be resolved in an :term:`asynchronous iterator`
-      object.  Introduced by :pep:`492`.
+      Must return an :term:`asyncronous iterator` from its
+      :meth:`__aiter__` method.  Introduced by :pep:`492`.
 
    asynchronous iterator
       An object that implements :meth:`__aiter__` and :meth:`__anext__`
-      methods, that must return :term:`awaitable` objects.
+      methods.  ``__anext__`` must return an :term:`awaitable` object.
       :keyword:`async for` resolves awaitable returned from asynchronous
       iterator's :meth:`__anext__` method until it raises
       :exc:`StopAsyncIteration` exception.  Introduced by :pep:`492`.

Doc/reference/compound_stmts.rst

@@ -726,7 +726,7 @@ The following code::
 Is semantically equivalent to::
 
     iter = (ITER)
-    iter = await type(iter).__aiter__(iter)
+    iter = type(iter).__aiter__(iter)
     running = True
     while running:
         try:

Doc/reference/datamodel.rst

@@ -2359,6 +2359,7 @@ generators, coroutines do not directly support iteration.
    Coroutine objects are automatically closed using the above process when
    they are about to be destroyed.
 
+.. _async-iterators:
 
 Asynchronous Iterators
 ----------------------
@@ -2371,7 +2372,7 @@ Asynchronous iterators can be used in an :keyword:`async for` statement.
 
 .. method:: object.__aiter__(self)
 
-   Must return an *awaitable* resulting in an *asynchronous iterator* object.
+   Must return an *asynchronous iterator* object.
 
 .. method:: object.__anext__(self)
 
@@ -2384,7 +2385,7 @@ An example of an asynchronous iterable object::
         async def readline(self):
             ...
 
-        async def __aiter__(self):
+        def __aiter__(self):
             return self
 
         async def __anext__(self):
@@ -2395,6 +2396,49 @@ An example of an asynchronous iterable object::
 
 .. versionadded:: 3.5
 
+.. note::
+
+   .. versionchanged:: 3.5.2
+      Starting with CPython 3.5.2, ``__aiter__`` can directly return
+      :term:`asynchronous iterators <asynchronous iterator>`.  Returning
+      an :term:`awaitable` object will result in a
+      :exc:`PendingDeprecationWarning`.
+
+      The recommended way of writing backwards compatible code in
+      CPython 3.5.x is to continue returning awaitables from
+      ``__aiter__``.  If you want to avoid the PendingDeprecationWarning
+      and keep the code backwards compatible, the following decorator
+      can be used::
+
+          import functools
+          import sys
+
+          if sys.version_info < (3, 5, 2):
+              def aiter_compat(func):
+                  @functools.wraps(func)
+                  async def wrapper(self):
+                      return func(self)
+                  return wrapper
+          else:
+              def aiter_compat(func):
+                  return func
+
+      Example::
+
+          class AsyncIterator:
+
+              @aiter_compat
+              def __aiter__(self):
+                  return self
+
+              async def __anext__(self):
+                  ...
+
+      Starting with CPython 3.6, the :exc:`PendingDeprecationWarning`
+      will be replaced with the :exc:`DeprecationWarning`.
+      In CPython 3.7, returning an awaitable from ``__aiter__`` will
+      result in a :exc:`RuntimeError`.
+
 
 Asynchronous Context Managers
 -----------------------------

Doc/whatsnew/3.5.rst

@@ -247,6 +247,19 @@ be used inside a coroutine function declared with :keyword:`async def`.
 Coroutine functions are intended to be run inside a compatible event loop,
 such as the :ref:`asyncio loop <asyncio-event-loop>`.
 
+
+.. note::
+
+   .. versionchanged:: 3.5.2
+      Starting with CPython 3.5.2, ``__aiter__`` can directly return
+      :term:`asynchronous iterators <asynchronous iterator>`.  Returning
+      an :term:`awaitable` object will result in a
+      :exc:`PendingDeprecationWarning`.
+
+      See more details in the :ref:`async-iterators` documentation
+      section.
+
+
 .. seealso::
 
    :pep:`492` -- Coroutines with async and await syntax

Include/genobject.h

@@ -54,6 +54,9 @@ typedef struct {
 PyAPI_DATA(PyTypeObject) PyCoro_Type;
 PyAPI_DATA(PyTypeObject) _PyCoroWrapper_Type;
 
+PyAPI_DATA(PyTypeObject) _PyAIterWrapper_Type;
+PyObject *_PyAIterWrapper_New(PyObject *aiter);
+
 #define PyCoro_CheckExact(op) (Py_TYPE(op) == &PyCoro_Type)
 PyObject *_PyCoro_GetAwaitableIter(PyObject *o);
 PyAPI_FUNC(PyObject *) PyCoro_New(struct _frame *,

Lib/_collections_abc.py

@@ -156,7 +156,7 @@ class AsyncIterable(metaclass=ABCMeta):
     __slots__ = ()
 
     @abstractmethod
-    async def __aiter__(self):
+    def __aiter__(self):
         return AsyncIterator()
 
     @classmethod
@@ -176,7 +176,7 @@ async def __anext__(self):
         """Return the next item or raise StopAsyncIteration when exhausted."""
         raise StopAsyncIteration
 
-    async def __aiter__(self):
+    def __aiter__(self):
         return self
 
     @classmethod

Lib/asyncio/compat.py

@@ -4,6 +4,7 @@
 
 PY34 = sys.version_info >= (3, 4)
 PY35 = sys.version_info >= (3, 5)
+PY352 = sys.version_info >= (3, 5, 2)
 
 
 def flatten_list_bytes(list_of_data):

Lib/asyncio/streams.py

@@ -689,3 +689,9 @@ def __anext__(self):
             if val == b'':
                 raise StopAsyncIteration
             return val
+
+    if compat.PY352:
+        # In Python 3.5.2 and greater, __aiter__ should return
+        # the asynchronous iterator directly.
+        def __aiter__(self):
+            return self

Lib/test/test_coroutines.py

@@ -1255,8 +1255,9 @@ async def __anext__(self):
 
         buffer = []
         async def test1():
-            async for i1, i2 in AsyncIter():
-                buffer.append(i1 + i2)
+            with self.assertWarnsRegex(PendingDeprecationWarning, "legacy"):
+                async for i1, i2 in AsyncIter():
+                    buffer.append(i1 + i2)
 
         yielded, _ = run_async(test1())
         # Make sure that __aiter__ was called only once
@@ -1268,12 +1269,13 @@ async def test1():
         buffer = []
         async def test2():
             nonlocal buffer
-            async for i in AsyncIter():
-                buffer.append(i[0])
-                if i[0] == 20:
-                    break
-            else:
-                buffer.append('what?')
+            with self.assertWarnsRegex(PendingDeprecationWarning, "legacy"):
+                async for i in AsyncIter():
+                    buffer.append(i[0])
+                    if i[0] == 20:
+                        break
+                else:
+                    buffer.append('what?')
             buffer.append('end')
 
         yielded, _ = run_async(test2())
@@ -1286,12 +1288,13 @@ async def test2():
         buffer = []
         async def test3():
             nonlocal buffer
-            async for i in AsyncIter():
-                if i[0] > 20:
-                    continue
-                buffer.append(i[0])
-            else:
-                buffer.append('what?')
+            with self.assertWarnsRegex(PendingDeprecationWarning, "legacy"):
+                async for i in AsyncIter():
+                    if i[0] > 20:
+                        continue
+                    buffer.append(i[0])
+                else:
+                    buffer.append('what?')
             buffer.append('end')
 
         yielded, _ = run_async(test3())
@@ -1338,7 +1341,7 @@ async def foo():
 
     def test_for_4(self):
         class I:
-            async def __aiter__(self):
+            def __aiter__(self):
                 return self
 
             def __anext__(self):
@@ -1368,8 +1371,9 @@ def __anext__(self):
                 return 123
 
         async def foo():
-            async for i in I():
-                print('never going to happen')
+            with self.assertWarnsRegex(PendingDeprecationWarning, "legacy"):
+                async for i in I():
+                    print('never going to happen')
 
         with self.assertRaisesRegex(
                 TypeError,
@@ -1393,7 +1397,7 @@ class Iterable:
             def __init__(self):
                 self.i = 0
 
-            async def __aiter__(self):
+            def __aiter__(self):
                 return self
 
             async def __anext__(self):
@@ -1417,7 +1421,11 @@ async def main():
                     I += 1
             I += 1000
 
-        run_async(main())
+        with warnings.catch_warnings():
+            warnings.simplefilter("error")
+            # Test that __aiter__ that returns an asyncronous iterator
+            # directly does not throw any warnings.
+            run_async(main())
         self.assertEqual(I, 111011)
 
         self.assertEqual(sys.getrefcount(manager), mrefs_before)
@@ -1470,15 +1478,65 @@ def test_for_7(self):
         class AI:
             async def __aiter__(self):
                 1/0
+        async def foo():
+            nonlocal CNT
+            with self.assertWarnsRegex(PendingDeprecationWarning, "legacy"):
+                async for i in AI():
+                    CNT += 1
+            CNT += 10
+        with self.assertRaises(ZeroDivisionError):
+            run_async(foo())
+        self.assertEqual(CNT, 0)
+
+    def test_for_8(self):
+        CNT = 0
+        class AI:
+            def __aiter__(self):
+                1/0
         async def foo():
             nonlocal CNT
             async for i in AI():
                 CNT += 1
             CNT += 10
         with self.assertRaises(ZeroDivisionError):
-            run_async(foo())
+            with warnings.catch_warnings():
+                warnings.simplefilter("error")
+                # Test that if __aiter__ raises an exception it propagates
+                # without any kind of warning.
+                run_async(foo())
         self.assertEqual(CNT, 0)
 
+    def test_for_9(self):
+        # Test that PendingDeprecationWarning can safely be converted into
+        # an exception (__aiter__ should not have a chance to raise
+        # a ZeroDivisionError.)
+        class AI:
+            async def __aiter__(self):
+                1/0
+        async def foo():
+            async for i in AI():
+                pass
+
+        with self.assertRaises(PendingDeprecationWarning):
+            with warnings.catch_warnings():
+                warnings.simplefilter("error")
+                run_async(foo())
+
+    def test_for_10(self):
+        # Test that PendingDeprecationWarning can safely be converted into
+        # an exception.
+        class AI:
+            async def __aiter__(self):
+                pass
+        async def foo():
+            async for i in AI():
+                pass
+
+        with self.assertRaises(PendingDeprecationWarning):
+            with warnings.catch_warnings():
+                warnings.simplefilter("error")
+                run_async(foo())
+
     def test_copy(self):
         async def func(): pass
         coro = func()

Lib/test/test_grammar.py

@@ -1076,7 +1076,7 @@ def test_async_for(self):
         class Done(Exception): pass
 
         class AIter:
-            async def __aiter__(self):
+            def __aiter__(self):
                 return self
             async def __anext__(self):
                 raise StopAsyncIteration

Misc/NEWS

@@ -130,6 +130,11 @@ Core and Builtins
 - Issue #25887: Raise a RuntimeError when a coroutine object is awaited
   more than once.
 
+- Issue #27243: Update the __aiter__ protocol: instead of returning
+  an awaitable that resolves to an asynchronous iterator, the asynchronous
+  iterator should be returned directly.  Doing the former will trigger a
+  PendingDeprecationWarning.
+
 
 Library
 -------