User application testing facility

[saaj]

Does aiohttp provide functionality to test application written with its HTTP client? It is in the sense of packages like httpretty, responses and this SO question. Basically it may be some mock connector to say here's the next response your should return on any request. Or it may be more elaborate with separate queues for HTTP methods, base URLs, et cetera. It's useful for integration and system testing. Workarounds are possible, but require to construct complete server to imitate the target and to patch the application's URLs, which can take some substantial effort.

Even though it's not in my direct interest, the same thing may also apply to HTTP server. Some sort of base test case class that provides means to start user's app, do requests and make assertions about them with ease. Here is how CherryPy does it.

I've looked at the test suite and aiohttp.test_utils but there seems to be nothing what I'm looking for exactly. If I have not overlooked it and there's no such functionality at the moment, this can be treated as a feature request.

[asvetlov]

I believe all mentioned projects are exist because it's hard to write test server in classical blocking way.
You should execute it in separate thread and synchronize testing code with this server.
However in asyncio world co-existing of client and server API's in the same test case is pretty easy.
We do use it in test suite for aiohttp, sure.
Please take a look on our typical test file: https://github.com/KeepSafe/aiohttp/blob/master/tests/test_client_functional_newstyle.py

[saaj]

I don't think that internal complexity really matters for an end user. She has certain use cases and there are common approaches to solve them without too much of boilerplate. Your answer, in other words, is DIY, which is actually an answer. Anyway, I want to point to some things.

Client testing

Specifically it is a white-box testing. A system written with aiohttp has endpoints (URLs) to which it performs requests during its normal operation. Unless you patch all the endpoints to localhost or if the design allows application-wide configuration, you can't test it. And ease is of composition isn't relevant here. Fake connector, which you can pass a number of pre-fabricated responses, is.

Server testing

The threaded server, CherryPy, didn't convince you. Here's my next attempt: http://tornado.readthedocs.org/en/latest/testing.html. You can see the same unittest API no mater if the things is blocking IO or not.

[asvetlov]

Well, I see your point.

For client testing fake connector that doesn't resolve host but connects to specific local port would be helpful.

Also we can introduce a TestCase good for aiohttp usage.

Currently we duplicate setUp/tearDown procedure in every our test file as well as find_unused_port and graceful server shutdown procedure.

Personally I don't feel the strong request for those changes but I'm open for Pull Request.

[samuelcolvin]

massive +1 for docs and/or helper methods for unit testing a server app.

In my few days starting to use aiohttp this has been the biggest stubbing block and the closest I've come to saying "aiohttp is not ready, lets go back to django/flask".

It's really important there's an easy and powerful way to make fake http requests, where one can:

• interrogate the response
• get proper stacktraces from server errors
• mock bits of the server on a per test basis.
• setup and teardown databases on the server between tests.

I've made a start on doing this myself by copying bits of aiohttp's own tests but it's not obvious and there ends up being quite a lot of code dedicated to setting up the tests which I would expect to normally like in the test framework. (eg. most of conftest.py).

If the expected approach at present is to use aiohttp's own tests as an example some more comments particularly in conftest.py would be great, eg. docstrings in methods like pytest_pycollect_makeitem the purpose of which is not immediately clear.

[samuelcolvin]

is it possible to use pytest-asyncio to replace some of the setup in conftest.py?

[saaj]

Saying aiohttp is not ready, lets go back to django/flask sounds like you only look in the spotlight, without really caring about practical qualities of the libraries/frameworks. Non-blocking IO doesn't come for free. It's good for one things, it's bad for others. If you're really looking for a non-blocking library/framework it's quite more reasonable to go back Tornado. It's the "now". aiohttp is more the "future".

[asvetlov]

aiohttp tests was started from classic unittest style.
We used it for about 2 years but I really don't very happy with this approach.
Then we tried to make pytest.
Looks much better. But before making our test suite tools public (e.g. document it) we need to collect better experience in the area.
Perhaps instead of trying to publish code is better to add documentation chapter about writing tests in both unittest and pytest ways.
pytest-asyncio is interesting project but we have a bit different views on how globallness of the loop. Again I don't want to contribute into pytest-asyncio until get satisfactory common solution for aiohttp.

Anyway, reading aiohttp test suite code is not so hard, isn't it?

[sloria]

We at Center for Open Science developed a "port" of httpretty for aiohttp: https://github.com/CenterForOpenScience/aiohttpretty . It's in its early days, but it's certainly come in handy for aiohttp client testing.

[dgelvin]

I have created a testing pattern that seems to work (based on very limited testing). Not sure if this is a very sane way to do it, but using this technique the tests and event loop are all running in the same main thread, so it should allow you to properly mock things.

EDIT Removed the example so I don't send people down the wrong track...

[asvetlov]

@dgelvin it may look surprising but you really don't need a separate thread for running client requests: they may share the same event loop.

[dgelvin]

@asvetlov I believe using my technique the client requests are performed on the same event loop. The thread is just used to schedule the request on the event loop, block for a response, then stop the event loop.

[asvetlov]

I feel your pain, guys.

The main challenge for me is introducing the test framework, not particular classes/functions but the complete solution.
Moreover, we switched (well, partially) to py.test instead of standard unittest.
Thus end users need two frameworks ideally.

Test framework means it should provide carefully calibrated tools for end users, covering not only aiohttp test needs but acceptable for application developers.
In other words I need to think about testing API as strong as about aiohttp itself.

We have plenty tests for aiohttp itself and aio-libs libraries. I should just get used techniques together and document the approach -- documenting is crucial part, you may copy-paste fixtures from tests/conftest.py for example but need to figure out what they does first.

I hope to introduce something next month -- perhaps for py.test first.

[asvetlov]

@dgelvin but you really don't need a thread at all -- please take a look on https://github.com/KeepSafe/aiohttp/blob/master/tests/test_urldispatch.py tests

[dgelvin]

@asvetlov Thanks, very helpful. I see that simply using run_until_complete with the client request works great.

[samuelcolvin]

I have some fixtures for pytest and aiohttp. They're based on the ones from aiohttp itself as well as pytest asyncio but are significantly simplified as they concentrate on testing the application not the framework.

I might try taking them out of the private repo they're currently in and create them as a gist or PR.

Obviously the test fixtures should end up as a reusable library rather than just available to be copy & pasted. If we're using pytest (which I really like) this should obviously be done as a plugin.

@asvetlov in an ideal world do you think this plugin should be part of aiohttp or a separate package?

[samuelcolvin]

In fact, just found I'm using them in a public repo: https://github.com/em-2/em2-core/blob/master/tests/http/conftest.py demonstrates a very simple conftest.py.

The tests are only really a sub so far, but they demonstrate the idea:

async def test_missing_conversation_valid_html(client):
    headers = {'Authorization': 'Token 321', 'Actor': 'test@example.com'}
    data = '{"key": "foobar"}'
    r = await client.post('/123/messages/add/', data=data, headers=headers)
    assert r.status == 400
    content = await r.read()
    assert content == b'ConversationNotFound: conversation 123 not found\n'

[asvetlov]

Fixed by #902 and #910

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.