Skip to content

Commit

Permalink
Mention PEP 492 native coroutines in crawler chapter.
Browse files Browse the repository at this point in the history
  • Loading branch information
ajdavis committed Aug 4, 2015
1 parent a7845ec commit 77e15a1
Showing 1 changed file with 10 additions and 2 deletions.
12 changes: 10 additions & 2 deletions crawler/crawler.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,9 @@ It is also scalable. Compared to the 50k of memory per thread and the operating

The concept of a coroutine, dating to the elder days of computer science, is simple: it is a subroutine that can be paused and resumed. Whereas threads are preemptively multitasked by the operating system, coroutines multitask cooperatively: they choose when to pause, and which coroutine to run next.

There are many implementations of coroutines; even in Python there are several. The coroutines in the standard "asyncio" library are built upon Python generators, a Future class, and the "yield from" statement. We will engage in an exposition of generators and how they are used as coroutines in asyncio, and trust you will enjoy reading it as much as we enjoyed writing it. Once we have explained asyncio coroutines, we shall use them in our async web crawler.
There are many implementations of coroutines; even in Python there are several. The coroutines in the standard "asyncio" library in Python 3.4 are built upon generators, a Future class, and the "yield from" statement. Starting in Python 3.5, coroutines will be a native feature of the language itself[^17]; however, understanding coroutines as they were first implemented in Python 3.4, using pre-existing language facilities, is the foundation to tackle Python 3.5's native coroutines.

To explain Python 3.4's generator-based coroutines, we will engage in an exposition of generators and how they are used as coroutines in asyncio, and trust you will enjoy reading it as much as we enjoyed writing it. Once we have explained generator-based coroutines, we shall use them in our async web crawler.

## How Python Generators Work

Expand Down Expand Up @@ -1072,7 +1074,11 @@ If we squint so that the `yield from` statements blur, a coroutine looks like a

But when we open our eyes and focus on the `yield from` statements, we see they mark points when the coroutine cedes control and allows others to run. Unlike threads, coroutines display where our code can be interrupted and where it cannot. In his illuminating essay "Unyielding"[^4], Glyph Lefkowitz writes, "Threads make local reasoning difficult, and local reasoning is perhaps the most important thing in software development." Explicitly yielding, however, makes it possible to "understand the behavior (and thereby, the correctness) of a routine by examining the routine itself rather than examining the entire system."

Now that you know how asyncio coroutines work, you can largely forget the details. The machinery is tucked behind a dapper interface. But understanding the fundamentals empowers you to code correctly and efficiently in modern async environments.
This chapter was written during a renaissance in the history of Python and async. Generator-based coroutines, whose devising you have just learned, were released in the "asyncio" module with Python 3.4 in March 2014. In September 2015, Python 3.5 will be released with coroutines built in to the language itself. These native coroutines will be declared with the new syntax "async def", and instead of "yield from", they will use the new "await" keyword to delegate to a coroutine or wait for a Future.

Despite these advances, the core ideas remain. Python's new native coroutines will be syntactically distinct from generators but work very similarly; indeed, they will share an implementation within the Python interpreter. Task, Future, and the event loop will continue to play their roles in asyncio.

Now that you know how asyncio coroutines work, you can largely forget the details. The machinery is tucked behind a dapper interface. But your grasp of the fundamentals empowers you to code correctly and efficiently in modern async environments.

[^4]: [https://glyph.twistedmatrix.com/2014/02/unyielding.html](https://glyph.twistedmatrix.com/2014/02/unyielding.html)

Expand Down Expand Up @@ -1110,3 +1116,5 @@ Now that you know how asyncio coroutines work, you can largely forget the detail
<latex>
[^16]: Guido introduced the standard asyncio library, called "Tulip" then, at PyCon 2013.
</latex>

[^17]: Python 3.5's built-in coroutines are described in [PEP 492](https://www.python.org/dev/peps/pep-0492/), "Coroutines with async and await syntax." At the time of this writing, Python 3.5 was in beta, due for release in September 2015.

0 comments on commit 77e15a1

Please sign in to comment.