Description
- Python version:
Darwin, python 3.7.4
# pipenv install pytest uvloop pytest-aiohttp
# pipenv run pip freeze
aiohttp==3.6.2
async-timeout==3.0.1
attrs==19.3.0
chardet==3.0.4
idna==2.8
importlib-metadata==1.5.0
more-itertools==8.2.0
multidict==4.7.4
packaging==20.1
pluggy==0.13.1
py==1.8.1
pyparsing==2.4.6
pytest==5.3.5
pytest-aiohttp==0.3.0
six==1.14.0
uvloop==0.14.0
wcwidth==0.1.8
yarl==1.4.2
zipp==2.2.0
Versions
This problem was already detected on my prod on linux.
platform darwin -- Python 3.7.4, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
-
Platform:
linux/darwin -
Can you reproduce the bug with
PYTHONASYNCIODEBUGin env?:
yes -
Does uvloop behave differently from vanilla asyncio? How?:
yes, test do not hang with pyloop.
When writing large amount of data in the stdin of a subprocess, we observe sometimes that the subprocess will hang trying to read from our app.
Trying to reproduce with a very small test, I find that, on macos, a limit of 8KiB.
Here is my test
# Standard Library
import asyncio.subprocess
import pytest
@pytest.mark.parametrize("num_lines", range(8188, 8194,1))
async def test_uvloopwrite(loop, num_lines):
proc = await asyncio.create_subprocess_exec("python", __file__,
stdout=asyncio.subprocess.PIPE,
stdin=asyncio.subprocess.PIPE)
data = b"\n" * num_lines + b"END\n"
print(num_lines, len(data))
print("writing", proc.stdin.transport.get_write_buffer_limits())
proc.stdin.write(data)
print("drain")
await proc.stdin.drain()
# proc.stdin.close()
print("waiting")
await proc.wait()
out = await proc.stdout.read()
assert int(out) == num_lines
# this small program is called by the test
if __name__ == "__main__":
import sys
n = 0
while True:
line = sys.stdin.readline()
if n > 8186:
print(n, repr(line), file=sys.stderr)
if line == "END\n":
break
n+=1
print(n)I run it with :
pipenv install pytest uvloop pytest-aiohttp
pipenv run pytest -s --aiohttp-loop=uvloop test_uvloop.py
When printing on the subprocess, I can see that the first 8191 bytes are seen by the subprocess, but the last one are never seen.
If I close the stdin (line commented in the test), it appears that it forces a buffer flush, and the test finishes.