Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asyncio drain fix #522

Merged
merged 5 commits into from
Jan 28, 2020
Merged

Asyncio drain fix #522

merged 5 commits into from
Jan 28, 2020

Conversation

Askaholic
Copy link
Collaborator

The server logs are plagued with the following error:

Traceback (most recent call last):
  File "/code/server/__init__.py", line 148, in do_report_dirties
    await asyncio.gather(*tasks)
  File "/code/server/servercontext.py", line 56, in broadcast_raw
    await asyncio.gather(*tasks)
  File "/code/server/protocol/qdatastreamprotocol.py", line 150, in send_raw
    await self.writer.drain()
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 339, in drain
    yield from self._protocol._drain_helper()
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 214, in _drain_helper
    assert waiter is None or waiter.cancelled()
AssertionError

It appears that this is caused by drain being awaited by multiple coroutines. As a result this is non-trivial to reproduce on the test server as it requires a lot of messages to be sent at the same time. As far as I can tell this behavior is considered a bug in asyncio, as it is not mentioned in the documentation. A workaround, which is employed by the websockets project, is to serialize the calls to drain with a mutex.

I don't expect the addition of a lock to have a noticeable effect on performance as typically the calls to drain were intended to serve as a way of serializing messages per connection anyways (applying backpressure), the only place where this wouldn't naturally occur was in the broadcast functions, which are the reason for occasionally getting concurrent calls to drain.

Copy link

@Wesmania Wesmania left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me

@Askaholic Askaholic merged commit 0ea040f into FAForever:develop Jan 28, 2020
@Askaholic Askaholic deleted the asyncio-drain-fix branch January 28, 2020 18:50
Askaholic added a commit that referenced this pull request Jan 30, 2020
* Add test for gameconnection error

* Add lock around protocol calls to drain

* Add tests for asyncio drain bug

* Propagate connection errors to server context

* Reset user online guage when the server restarts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants