create_subprocess_exec stdin writer does not write everything

[tardyp]

• Python version:
  Darwin, python 3.7.4

# pipenv install pytest uvloop pytest-aiohttp
# pipenv run pip freeze
aiohttp==3.6.2
async-timeout==3.0.1
attrs==19.3.0
chardet==3.0.4
idna==2.8
importlib-metadata==1.5.0
more-itertools==8.2.0
multidict==4.7.4
packaging==20.1
pluggy==0.13.1
py==1.8.1
pyparsing==2.4.6
pytest==5.3.5
pytest-aiohttp==0.3.0
six==1.14.0
uvloop==0.14.0
wcwidth==0.1.8
yarl==1.4.2
zipp==2.2.0

Versions
This problem was already detected on my prod on linux.

platform darwin -- Python 3.7.4, pytest-5.3.5, py-1.8.1, pluggy-0.13.1

• Platform:
  linux/darwin

• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?:
  yes

• Does uvloop behave differently from vanilla asyncio? How?:

yes, test do not hang with pyloop.

When writing large amount of data in the stdin of a subprocess, we observe sometimes that the subprocess will hang trying to read from our app.

Trying to reproduce with a very small test, I find that, on macos, a limit of 8KiB.

Here is my test

# Standard Library
import asyncio.subprocess
import pytest

@pytest.mark.parametrize("num_lines", range(8188, 8194,1))
async def test_uvloopwrite(loop, num_lines):
 proc = await asyncio.create_subprocess_exec("python", __file__,
 stdout=asyncio.subprocess.PIPE,
 stdin=asyncio.subprocess.PIPE)
 data = b"\n" * num_lines + b"END\n"
 print(num_lines, len(data))
 print("writing", proc.stdin.transport.get_write_buffer_limits())
 proc.stdin.write(data)
 print("drain")
 await proc.stdin.drain()
 # proc.stdin.close()
 print("waiting")
 await proc.wait()
 out = await proc.stdout.read()
 assert int(out) == num_lines

# this small program is called by the test
if __name__ == "__main__":
 import sys
 n = 0
 while True:
 line = sys.stdin.readline()
 if n > 8186:
 print(n, repr(line), file=sys.stderr)
 if line == "END\n":
 break
 n+=1
 print(n)

I run it with :

pipenv install pytest uvloop pytest-aiohttp
pipenv run pytest -s --aiohttp-loop=uvloop test_uvloop.py

When printing on the subprocess, I can see that the first 8191 bytes are seen by the subprocess, but the last one are never seen.

If I close the stdin (line commented in the test), it appears that it forces a buffer flush, and the test finishes.

[1st1]

Would appreciate you submitting a PR with a functional test (marked as x-fail) to demonstrate the problem.

[tardyp]

done. actually I re-tested under linux, and found out a weird threshold of 219264 bytes