Isolated usage of greenlets

[c-p-b]

Thanks for the library!

Prefect has been doing some experimenting with greenback here and ran into this limitation because SQLAlchemy is also used:

Unfortunately using greenback alongside sqlalchemy is not possible at this time. If the ephemeral application is running in the same thread as one in which we need to use greenback, then greenback and sqlalchemy will accidentally "mix" messages and crash each other.

The workarounds for this such as running in a separate thread have their own drawbacks

Is this kind of isolation something that could be handled at the greenback level? Or is this more of a limitation of the way greenlets and/or SQLAlchemy works?

[oremanj]

Are you able to give a reproducer of the conflict that doesn't have dependencies outside of greenback, sqlalchemy, and some async library? I'm not familiar with Prefect at all, and only a little bit familiar with sqlalchemy's use of greenlets.

[zanieb]

Hey @oremanj — I don't work at Prefect anymore but I was the one looking into this originally and am quite curious if there's a solution.

I'll see if I can find or create an example. It's possible @jakekaplan could provide one as well.

[zanieb]

Okay it's kind of late and I don't remember all the details so forgive me if these are not correct, but I can create a few symptoms as follows:

import greenback
import sqlalchemy
import asyncio


def fails():
    greenback.await_(asyncio.sleep(0))  # ERROR
    # TypeError: object NoneType can't be used in 'await' expression


async def main():
    await greenback.ensure_portal()
    greenback.await_(asyncio.sleep(0))  # OK
    await sqlalchemy._util.concurrency.greenlet_spawn(fails)


if __name__ == "__main__":
    asyncio.run(main())

import greenback
import sqlalchemy
import asyncio


def fails():
    sqlalchemy._util.concurrency.await_only(asyncio.sleep(0))  # OK
    greenback.await_(asyncio.sleep(0))  # ERROR
    # TypeError: object Value can't be used in 'await' expression


async def main():
    await greenback.ensure_portal()
    greenback.await_(asyncio.sleep(0))  # OK
    await sqlalchemy._util.concurrency.greenlet_spawn(fails)


if __name__ == "__main__":
    asyncio.run(main())

with

greenback==1.1.1
sqlalchemy==2.0.20

[zanieb]

@oremanj just wondering if you had a chance to look into this and if I can do anything else to help. Thanks!

[oremanj]

Hi, sorry for the delay here. It looks like sqlalchemy stores a link to the parent/driver greenlet explicitly in its child greenlet, instead of using greenlet.getcurrent().parent. I'm guessing that would fix the issue here. I'd be happy to review a PR if you like; otherwise I'll try to get a fix up, but I'm pretty busy at the moment.

[zanieb]

Thanks for looking into it. I tried to make some changes in that vein and must admit I'm still a bit confused. If you can provide more guidance I'm happy to play with it some more. Otherwise, I understand that you're busy.