Generator fixup

[ddfisher]

This commit fixes up the handling of Generators in several ways:

• it removes YieldStmt and YieldFromStmt, because they unnecessarily
  duplicate ExpressionStmt of YieldExpr/YieldFromExpr
• it ensures functions are properly marked as is_generator when
  appropriate
• it fixes a parsing bug where empty yield expressions would cause a parse
  error
• it checks Generator functions for the appropriate type signature at
  the function level, rather than at the yield statement level
• it adds support for return statements in Generator functions

fixes #20, fixes #836, fixes #695, fixes #633, fixes #498

[ddfisher]

I wasn't sure how to write a unit test for this, but it should probably have one.

[gvanrossum]

About the Travis failures... It fails for Python 3.4 and 3.5 only, and is related to asyncio. Here is a repro that fails when run with your PR:

import asyncio
@asyncio.coroutine
def foo() -> asyncio.Future[None]:
        yield from asyncio.sleep(0.1)

And since Future isn't a base class of Generator, it fails your code, with this error:

asy.py: note: In function "factorial":
asy.py:3: error: The return type of a generator function should be "Generator" or one of its supertypes

The tests containing this pattern are in mypy/test/data/pythoneval-asyncio.test, and are skipped in 3.2 and 3.3, where asyncio doesn't exist (the test framework only runs pythoneval-asyncio.test in 3.4 and higher -- see mypy/test/testpythoneval.py:31).

So what to do? I can make this error go away by adding a check for 'asyncio.futures.Future' to is_generator_return_type(). That seems pretty unprincipled though -- and what if some other async library were to introduce a different Future type?

It is also a lie. The function foo() above does not return a Future! It really does return a generator. It doesn't have any of Future's methods, like cancel(), done() or add_done_callback().

Everywhere a test defines a coroutine with return type Future[T] it should really use Generator[Future[Any], None, T]. But we might as well say Generator[Any, None, T] since nobody except the event loop cares about the Future that is technically yielded whenever a coroutine yields. And send() is only called by the event loop, with a None parameter (in asyncio/tasks.py).

That's still unfortunately verbose though. Also, the stubs for asyncio need to be changed -- they currently define several coroutines that claim to return Futures, which are all lies: either those APIs aren't coroutines, or they don't return Futures -- asyncio is sometimes a bit vague about whether something is a coroutine or a Future, and callers aren't supposed to care much -- they are expected to call asyncio.ensure_future() if they need a Future (e.g. to add a callback) and they're not expected to depend on specific behavior of generators/coroutines -- only yield from them.

Apparently so far all this was put off till the future (!) by the code you are fixing that deals with yield-from -- that code was quietly accepting the lies.

Note that there's a further wrinkle -- Awaitable. This only exists in 3.5 and later (both in collections.abc and in typing). However, I think that we can put off Awaitable (maybe file an issue about it). There are no tests involving it yet, so it won't distract us further here.

[ddfisher]

Thanks for the detailed explanation! I definitely agree that we shouldn't be satisfied with lies here.

For the verbosity, it would be nice if we could define a type alias Coroutine[T] = Generator[Any, None, T], but it looks to me like type aliases can't currently expose type parameters? If that's the case, I'm not sure there's much better option than accepting the verbosity for now. (We could instead make Coroutine subclass of Generator, but then we'd need to do add some silly special casing, because we don't want to allow all Generator subclasses...) If we just write Generator[Any, None, T] (as you suggested) it's not even that much longer.

I also agree about punting on Awaitable support for now -- that seems like the sort of thing we should implement after we have the rest of the language working and fast. I filed an issue (#1139) for it.

[JukkaL]

It would be great if somebody would think through the whole asynchronous topic and how mypy should behave ideally in light of Python 3.5 and other recent developments. Most of the existing code related to yield from etc. is by external contributors and I haven't been very careful about how principled the code is, with the philosophy that it's easier to improve something that's a little wrong than to start implementing it from scratch.

[gvanrossum]

So my recommendation to unblock this PR is to just change the asyncio tests and stubs to use Generator[Any, None, X] instead of Future[X] (where X is usually None or some simple concrete type like int). In the future we can figure out a more compact spelling of that (maybe Coroutine[X], but Coroutine is only defined in 3.5 and later; so let's postpone solving that).

[gvanrossum]

Please get rid of this blank line (it looks weird in context).

[ddfisher]

Done.