asyncio.iscoroutinefunction returns false for Cython async function objects

[danigosa]

Referring to #2092

@scoder the implementation in asyncio is ridiculous

def iscoroutinefunction(func):
    """Return True if func is a decorated coroutine function."""
    return (inspect.iscoroutinefunction(func) or
            getattr(func, '_is_coroutine', None) is _is_coroutine)

So once we agreed inspect returns False, it happens to be that CPython just add a simple attribute to the function _is_coroutine, I think if Cython does this adding up this attribute it will make everything work

Here: https://github.com/python/cpython/blob/789e359f51d2b27bea01b8c6c3bf090aaedf8839/Lib/asyncio/coroutines.py#L152

And here:
https://github.com/python/cpython/blob/789e359f51d2b27bea01b8c6c3bf090aaedf8839/Lib/asyncio/coroutines.py#L160

So the solution as it can't be another it results in this, from my test in iPython:

In [46]: fut = init_connection()
In [47]: asyncio.iscoroutine(fut)
Out[47]: True

In [49]: init_connection.__class__
Out[49]: cython_function_or_method

In [50]: asyncio.iscoroutinefunction(init_connection)
Out[50]: False

In [56]: from asyncio.coroutines import _is_coroutine

In [57]: init_connection._is_coroutine = _is_coroutine

In [58]: asyncio.iscoroutinefunction(init_connection)
Out[58]: True

Dirty but easy. Don't know if this is a 5 seconds issue or a hara-kiri, theoretically it should be easy to plug the attribute runtime from from asyncio.coroutines import _is_coroutine, or not.

This is a super annoying problem (I don't blame anyone with the word bug) that make cythonized code to be very dramatic in several asyncio frameworks. asyncio.iscoroutine work well with cythonized funcs for the silliest reason I think.

Following the cyfunctions above, create_token is NOT a coroutine, while init_connection is an async function, this workaround works in both so it's ok if you can schedule the functions whether are coros or not, no way to easily fix this with asyncio.iscoroutinefunction or at least easier than that:

In [59]: async def test():
    ...:     await create_token({})
    ...:
In [61]: asyncio.get_event_loop().run_until_complete(test())
TypeError: object str can't be used in 'await' expression

In [73]: async def test():
    ...:     return await init_connection()
Out[74]: <asyncpg.pool.Pool at 0x7f78e04752c8>

Here comes the magic:

In [69]: async def test():
    ...:     return await asyncio.coroutine(create_token)({})
In [70]: asyncio.get_event_loop().run_until_complete(test())
Out[70]: 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJleHAiOjE1MjU1NTk0NTB9.v8wRZGIke6RYizIpJhl4oaypKyvuVARKaiq0KdpJg6XJ0qTB0o76BuTLera6kSQ_5qXnAb7_9DQSadwdRqgPmw'

In [71]: async def test():
    ...:     return await asyncio.coroutine(init_connection)()
In [72]: asyncio.get_event_loop().run_until_complete(test())
Out[72]: <asyncpg.pool.Pool at 0x7f78df48af48>

With that the workaround:

def nasty_iscoroutinefunction(func):
      is_coro = asyncio. iscoroutinefunction(func)
      if func.__class__.__name__ == 'cython_function_or_method':
            # It's cythonized, danger!
            is_coro = True  # We'll make it possible
            func = asyncio.coroutine(func)
      return is_coro

This works!

[danigosa]

Found an interesting, better and working workaround:

https://github.com/timothycrosley/hug/blob/develop/hug/introspect.py#L33

def is_coroutinefunction(function):
    """Returns True if the passed in function is a coroutine"""
    return function.__code__.co_flags & 0x0080 or getattr(function, '_is_coroutine', False)

[pohmelie]

Is there plans to fix this?
@danigosa, AFAIR you suggest to replace asyncio.iscoroutinefunction in asyncio module, but I think this should be fixed on cython side instead, so the original asyncio method will work as expected.

[pohmelie]

Hm, it looks like tests with code similar to inspect.py are disabled.

[scoder]

The problem is that if we make Cython coroutines look too much like Python coroutines at the C level (e,g, by setting the CO_COROUTINE flag on them), then CPython might start doing things with them that it shouldn't. Even if we find that it doesn't do that right now, it might change at any time, and then people have their Cython modules out there that break. So this is risky business.

Similarly, unconditionally patching inspect might have unfortunate side effects as well, even if requires an opt-in. It's impossible to guess what mixture of code end users will have on their systems.

[pohmelie]

@scoder
I'm half way to make PR to add CO_COROUTINE to coroutine wrappers. But if it will not be accepted anyway, I will stop.

AFAIR, right now c wrapper looks like:

def c_wrapper(...):
    return coroutine(...)

It looks like this can be changed to

@types.coroutine
def c_wrapper(...):
    return (yield from coroutine(...))

In this case all stuff is on python shoulders.

[pohmelie]

@scoder
I have made some dirty hack and it looks like it works. Any thoughts about such solution?
pohmelie@5a84869

import asyncio


async def foo():
    pass


print("foo", foo.__code__.co_flags)
print("foo", asyncio.iscoroutinefunction(foo))
print(__cython_coroutine_wrapper)

compiled code output:

('foo', 319)
('foo', True)
<function coroutine at 0x7f30b9a0e620>

[pohmelie]

Linked code is not working in all cases, it is just quick proof of concept.

[pohmelie]

By the way, I can't find any problem with add CO_COROUTINE flag, since

async def afoo():
    ...

is same as

def foo():
    return afoo()

you can await foo() and it will be absolutely same as await afoo().

[scoder]

There is at least this code in CPython, which I'd rather not want to trigger for Cython coroutines:
https://github.com/python/cpython/blob/caba55b3b735405b280273f7d99866a046c18281/Python/ceval.c#L3906-L3949
Can't say right now if it's possible to do that from Python code, probably via eval&friends.

Also, what we are discussing here actually applies to these three functions in inspect.py: isgeneratorfunction(), iscoroutinefunction() and isasyncgenfunction().

https://github.com/python/cpython/blob/41254ebd5e4f40a2e095d8aaea60bf3973de4647/Lib/inspect.py#L171-L194

[pohmelie]

But why old-style yield from coroutines with asyncio.coroutine works fine? Why we can't just wrap new-style coroutines with same decorator on building ast step?

[keakon]

@scoder Can you consider adding new flags like CO_CYTHON_COROUTINE?

I'm using Starlette, it requires asyncio.iscoroutinefunction() to distinguish coroutine functions. I can't fix it because Cython hides everything about coroutine functions.

[scoder]

Can't starlette do a type-check against the Coroutine ABC instead?

[keakon]

How can I know a function will return a coroutine without call it?

[scoder]

How can I know a function will return a coroutine without call it?

You cannot. There is no way to do that. It's called the "halting problem".
https://en.wikipedia.org/wiki/Halting_problem

[keakon]

But CPython can:

>>> async def f():
...   pass
...
>>> import inspect
>>> inspect.iscoroutinefunction(f)
True
>>> import asyncio
>>> asyncio.iscoroutinefunction(f)
True