Future <Future pending> attached to a different loop error if hypothesis and @pytest.mark.asyncio are used

[VladimirWork]

Hello, could you please help me with this issue? I'm trying to send values from hypothesis in async test with await calls inside and getting subject error (I've tried trick from #38 for event_loop but it still doesn't work, also it looks like your hypothesis tests have no awaits inside but asserts only so could you point me to any real example of hypothesis+pytest-asyncio usage?):

platform linux -- Python 3.5.2, pytest-4.3.1, py-1.8.0, pluggy-0.7.1
plugins: hypothesis-4.14.0, testinfra-2.0.0, asyncio-0.10.0

conftest.py:

@pytest.fixture(scope="session")
def event_loop():
loop = asyncio.get_event_loop()
loop.run_until_complete(pool.set_protocol_version(2))
yield loop
loop.close()

test.py:

@settings(verbosity=Verbosity.debug)
@given(test_string=st.characters(whitelist_categories=('N', 'L')))
@pytest.mark.asyncio
async def test_misc_hypothesis(event_loop, test_string):
pool_handle, _ = await pool_helper()
wallet_handle, _, _ = await wallet_helper()
trustee_did, _ = await default_trustee(wallet_handle)
res = await schema_helper(pool_handle, wallet_handle, trustee_did, test_string, '1.0',
json.dumps(["age", "sex", "height", "name"]))
print(res)

stacktrace:

=================================== FAILURES ===================================
_____________________________ test_misc_hypothesis _____________________________

event_loop = <_UnixSelectorEventLoop running=False closed=False debug=False>

@settings(verbosity=Verbosity.debug)

@given(test_string=st.characters(whitelist_categories=('N', 'L')))
@pytest.mark.asyncio
async def test_misc_hypothesis(event_loop, test_string):

test_misc.py:623:

---

/usr/lib/python3.5/contextlib.py:77: in exit
self.gen.throw(type, value, traceback)
/usr/local/lib/python3.5/dist-packages/hypothesis/_settings.py:402: in local_settings
yield s
/usr/lib/python3.5/contextlib.py:77: in exit
self.gen.throw(type, value, traceback)
/usr/local/lib/python3.5/dist-packages/hypothesis/utils/dynamicvariables.py:42: in with_value
yield
/usr/local/lib/python3.5/dist-packages/hypothesis/_settings.py:402: in local_settings
yield s
/usr/local/lib/python3.5/dist-packages/hypothesis/core.py:568: in execute
result = self.test_runner(data, run)
/usr/local/lib/python3.5/dist-packages/hypothesis/executors.py:56: in default_new_style_executor
return function(data)
/usr/local/lib/python3.5/dist-packages/hypothesis/core.py:566: in run
return test(*args, **kwargs)
/usr/lib/python3.5/contextlib.py:77: in exit
self.gen.throw(type, value, traceback)
/usr/local/lib/python3.5/dist-packages/hypothesis/_settings.py:402: in local_settings
yield s
/usr/lib/python3.5/contextlib.py:77: in exit
self.gen.throw(type, value, traceback)
/usr/local/lib/python3.5/dist-packages/hypothesis/utils/dynamicvariables.py:42: in with_value
yield
/usr/local/lib/python3.5/dist-packages/hypothesis/_settings.py:402: in local_settings
yield s
/usr/local/lib/python3.5/dist-packages/hypothesis/core.py:566: in run
return test(*args, **kwargs)
/usr/local/lib/python3.5/dist-packages/hypothesis/control.py:73: in exit
self.assign_variable.exit(exc_type, exc_value, tb)
/usr/lib/python3.5/contextlib.py:77: in exit
self.gen.throw(type, value, traceback)
/usr/local/lib/python3.5/dist-packages/hypothesis/utils/dynamicvariables.py:42: in with_value
yield
/usr/local/lib/python3.5/dist-packages/hypothesis/core.py:566: in run
return test(*args, **kwargs)
/usr/lib/python3.5/contextlib.py:77: in exit
self.gen.throw(type, value, traceback)
/usr/local/lib/python3.5/dist-packages/hypothesis/internal/entropy.py:104: in deterministic_PRNG
yield
/usr/local/lib/python3.5/dist-packages/hypothesis/core.py:566: in run
return test(*args, **kwargs)
/usr/local/lib/python3.5/dist-packages/pytest_asyncio/plugin.py:166: in test_misc_hypothesis
def inner(**kwargs):
/usr/local/lib/python3.5/dist-packages/hypothesis/core.py:529: in test
result = self.test(*args, **kwargs)
/usr/local/lib/python3.5/dist-packages/pytest_asyncio/plugin.py:172: in inner
loop.run_until_complete(future)
/usr/lib/python3.5/asyncio/base_events.py:387: in run_until_complete
return future.result()
/usr/lib/python3.5/asyncio/futures.py:274: in result
raise self._exception
/usr/lib/python3.5/asyncio/tasks.py:241: in _step
result = coro.throw(exc)
test_misc.py:626: in test_misc_hypothesis
pool_handle, _ = await pool_helper()
utils.py:42: in pool_helper
await pool.create_pool_ledger_config(pool_name, pool_config)
/usr/local/lib/python3.5/dist-packages/indy/pool.py:39: in create_pool_ledger_config
create_pool_ledger_config.cb)

---

self =

def __iter__(self):
    if not self.done():
        self._blocking = True

      yield self  # This tells Task to wait for completion.

E RuntimeError: Task <Task pending coro=<test_misc_hypothesis() running at /home/indy/indy-test-automation/system/test_misc.py:626> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> got Future attached to a different loop

/usr/lib/python3.5/asyncio/futures.py:361: RuntimeError

[seifertm]

As far as I understand, test cases using Hypothesis run in their own event loop. That loop is completely separate from the loop returned by the event_loop fixture.

The function responsible is wrap_in_sync which is used to wrap hypothesis tests:

pytest-asyncio/pytest_asyncio/plugin.py

Lines 162 to 176 in f35d06b

	def wrap_in_sync(func):
	"""Return a sync wrapper around an async function."""
	@functools.wraps(func)
	def inner(**kwargs):
	loop = asyncio.get_event_loop_policy().new_event_loop()
	try:
	coro = func(**kwargs)
	if coro is not None:
	future = asyncio.ensure_future(coro, loop=loop)
	loop.run_until_complete(future)
	finally:
	loop.close()
	return inner

I think the same problem has been plaguing me for some time in the context of aiohttp fixtures. Can anybody explain why Hypothesis tests require their own event loop?

[skhoroshavin]

Can anybody explain why Hypothesis tests require their own event loop?

I suspect this could be done to make tests more reproducible - using clean event loop per generated example should minimize potential interference between examples. It can be worthwhile since hypothesis runs hundreds examples per test, and can be especially sensitive to flakiness during shrinking. However this is just a guess.

It would be great if someone could point how to address problem correctly. I suspect there can be one of two solutions:

• pytest-asyncio needs to be modified to support nondefault event loop
• Hypothesis needs to handle pytest.mark.asyncio itself

[Zac-HD]

Hi everyone, Hypothesis maintainer here - I actually wrote this code, so:

Can anybody explain why Hypothesis tests require their own event loop?

I'm pretty sure it's possible to hook it all togther so that it works, I just didn't understand pytest-asyncio well enough to do so! #102 was the pull request, and has some related commentary on this topic. In fact I still don't, but I'd be happy to review any proposals or pull requests for this - just @-mention me so I get notified.

(The key invariant is that each test case must be fully independent of any other, so state must not leak between calls, and I don't understand the event_loop behaviour well enough to be rely on that. It's also made harder by HypothesisWorks/hypothesis#377, that function-scoped fixtures are run once each for the outer wrapper rather than for each test case.)

But the important point is that Hypothesis tests don't require their own event loop, just that each test be fully isolated.

[seifertm]

Thank you for bringing this to attention again! I had actually worked on some changes, which have fallen by the wayside… I know neither pytest-asyncio nor hypothesis well, so I am happy for any input on the upcoming PR :)

[seifertm]

@VladimirWork is this issue resolved?

[VladimirWork]

@seifertm It looks like yes - I've got no errors using hypothesis @given together with @pytest.mark.asyncio and asynchronous fixtures in my test against asyncio-0.11.0.dev0 from master branch and hypothesis goes over values as expected. Thank you!