Creating `anyio.CancelScope` in async fixture raises `Attempted to exit cancel scope in a different task than it was entered in`

[provinzkraut]

import anyio
import pytest


@pytest.fixture
async def scope():
    with anyio.CancelScope() as scope:
        yield scope


async def test(scope: anyio.CancelScope):
    pass

This will raise a RuntimeError: Attempted to exit cancel scope in a different task than it was entered in.

After looking at the code for fixture cleanup, I don't see an easy way to work around this, but it's quite a severe issue as it hinders working with cancel scopes / code that uses cancel scopes.

[seifertm]

Based on the error message, I assume you use pytest-asyncio in auto mode and the test function is executed using pytest-asyncio. At least I can reproduce your error with pytest-asyncio v1.1.0 and anyio v4.10.0 when running pytest --asyncio-mode=auto.

Can you explain why your use case makes it necessary to use anyio.CancelScope, but run the test suite with pytest-asyncio as opposed to running the tests with anyio?

[provinzkraut]

At least I can reproduce your error with pytest-asyncio v1.1.0 and anyio v4.10.0 when running pytest --asyncio-mode=auto.

The code with explicit usage of pytest-asyncio and --asyncio-mode=strict gives the same result.

Can you explain why your use case makes it necessary to use anyio.CancelScope, but run the test suite with pytest-asyncio as opposed to running the tests with anyio?

I'm working on the testing utilities of the Litestar framework, which among other things provides to its users an async test client they may use in their test suites. This test client makes use of anyio and in particular anyio.CancelScope.

While it might be possible for Litestar's own test suite to drop pytest-asyncio in favour of anyio, this would also mean that all its downstream users would have to do the same, which does not sound great; A user should be able to use whatever async test runner they want.

[seifertm]

pytest-asyncio synchronizes async fixtures so that they can be executed by pytest. For async genenerator fixtures, this is performed by _wrap_asyncgen_fixture. The function advances the async generator, in order to run the fixture function up to and including the yield statement. This gives us the fixture value. Before returning the value pytest-asyncio registers a pytest fixture finalizer that advances the async generator again to trigger the cleanup. Pytest-asyncio uses asyncio.Runner.run each time it advances the async generator.

The underlying issue seems to be that the asyncio.Runner.run creates a task for each coroutine. This results in the error messages you're seeing, which pretty much describes what is happening.

Making this work means rewriting _wrap_asyncgen_fixture. From the top of my head, I don't know how this could be achieved. That doesn't mean it's impossible, of course :)

[provinzkraut]

Making this work means rewriting _wrap_asyncgen_fixture. From the top of my head, I don't know how this could be achieved. That doesn't mean it's impossible, of course :)

Perhaps using a single "worker task" that coroutines can be sent into would work?

[provinzkraut]

Now that I think about it, there's a possibility this issue also occurs when sharing a CancelScope across the fixture / test boundary.

[provinzkraut]

Making this work means rewriting _wrap_asyncgen_fixture. From the top of my head, I don't know how this could be achieved. That doesn't mean it's impossible, of course :)

Perhaps using a single "worker task" that coroutines can be sent into would work?

AnyIO seems to do just that: https://github.com/agronholm/anyio/blob/b8e91a5cfe35c3b28f91ea8251674a871a7f4e26/src/anyio/_backends/_asyncio.py#L2222-L2239

[provinzkraut]

I've made a first attempt: #1193. Let me know what you think