pytest 5.4+ doesn't handle IsolatedAsyncioTestCase from CPython core.

[cjw296]

So, #5734 causes pytest to incorrectly skip some async tests, specifically those based on https://docs.python.org/3/library/unittest.html#unittest.IsolatedAsyncioTestCase or code like it

See #5734 (comment) for real world failure, the mock backport has some of its own code in this area, to cater for Python 3.6 and 3.7:
https://github.com/testing-cabal/mock/blob/b5ce0a5c4d372b77deff46fec8edf974d7d1f875/mock/backports.py#L38-L82

For now, I'll just pin to pytest<5.4, but would be interested to know in how this will be fixed.

[cjw296]

Here's a minimal reproducer for Python 3.8 that works on pytest 5.3, but not on 5.4:

from unittest import IsolatedAsyncioTestCase


class AsyncArguments(IsolatedAsyncioTestCase):

    async def test_something_async(self):

        async def addition(x, y):
            return x+y

        self.assertEqual(await addition(2, 2), 4)

[graingert]

Here's a minimal reproducer for Python 3.8 that works on pytest 5.3, but not on 5.4:

from unittest import IsolatedAsyncioTestCase


class AsyncArguments(IsolatedAsyncioTestCase):

    async def test_something_async(self):

        async def addition(x, y):
            return x+y

        self.assertEqual(await addition(2, 2), 4)

What happens if you run a failing test? Eg: assert await addition (2, 2) == 3?

[cjw296]

$ pytest mock/tests/test_reproducer.py 
========================================================= test session starts =========================================================
platform darwin -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/chris/vcs/git/mock, inifile: setup.cfg
plugins: cov-2.8.1
collected 1 item                                                                                                                      

mock/tests/test_reproducer.py F                                                                                                 [100%]

============================================================== FAILURES ===============================================================
_________________________________________________ AsyncArguments.test_something_async _________________________________________________

self = <mock.tests.test_reproducer.AsyncArguments testMethod=test_something_async>

    async def test_something_async(self):
    
        async def addition(x, y):
            return x+y
    
>       self.assertEqual(await addition(2, 2), 3)
E       AssertionError: 4 != 3

mock/tests/test_reproducer.py:11: AssertionError

IsolatedAsyncioTestCase's run method does all the async stuff, in the same way an async plugin would:

        def run(self, result=None):
            self._setupAsyncioLoop()
            try:
                return super().run(result)
            finally:
                self._tearDownAsyncioLoop()

Remember: this is a unittest.TestCase subclass, so it can do lots in that run method...

[RonnyPfannschmidt]

thanks for the details, i believe we should ignore async for unittest subclasses, i will try to fix this afternoon

[cjw296]

I think it might also make sense to have a setting to turn this off...

[RonnyPfannschmidt]

@cjw296 there is another issue at play, i made a opt-out specifically for unit-test and get

unittest/test_unittest_asyncio.py::AsyncArguments::test_something_async
  /home/ronny/Projects/pytest-dev/pytest/.env/lib/python3.8/site-packages/pluggy/callers.py:187: RuntimeWarning: coroutine 'AsyncArguments.test_something_async' was never awaited
    res = hook_impl.function(*args)

unittest/test_unittest_asyncio.py::AsyncArguments::test_something_async_fails
  /home/ronny/Projects/pytest-dev/pytest/.env/lib/python3.8/site-packages/pluggy/callers.py:187: RuntimeWarning: coroutine 'AsyncArguments.test_something_async_fails' was never awaited
    res = hook_impl.function(*args)

[RonnyPfannschmidt]

#5996 is the real origin of the issue, a fix will need some more time to understand

[cjw296]

Okay, if it helps, there’s no massive rush on this, pinning to the older version is fine for the foreseeable future...

[grigi]

I have the same issue with using https://github.com/Martiusweb/asynctest with CPython 3.7 in my project.

[nicoddemus]

#5996 is the real origin of the issue, a fix will need some more time to understand

Indeed, that's a very complicated (and emotional) thread.

Would dropping support for --pdb for TestCase-based tests be an option? It would greatly simplify the code.

[RonnyPfannschmidt]

I have a fix in mind but caught a cold in between, I'll take a look later