Cannot use `sys.stdout` in `create_subprocess_exec`.

[ojii]

• uvloop version: 0.9.1
• Python version: 3.6.5
• Platform: macOS 10.13.3
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: Same behavior irregardless of PYTHONASYNCIODEBUG.

Code for reproduction:

import asyncio
import sys

import uvloop


async def run_subprocess():
 proc = await asyncio.create_subprocess_exec(
 sys.executable, '-c', 'print(1)',
 stdout=sys.stdout,
 )
 await proc.communicate()


def run(loop_factory):
 loop = loop_factory()
 asyncio.set_event_loop(loop)
 try:
 loop.run_until_complete(run_subprocess())
 finally:
 loop.close()


def main():
 print('asyncio')
 run(asyncio.new_event_loop)
 print('uvloop')
 run(uvloop.new_event_loop)


if __name__ == '__main__':
 main()

Output:

asyncio
1
uvloop
Traceback (most recent call last):
 File "hoge.py", line 32, in <module>
 main()
 File "hoge.py", line 28, in main
 run(uvloop.new_event_loop)
 File "hoge.py", line 19, in run
 loop.run_until_complete(run_subprocess())
 File "uvloop/loop.pyx", line 1364, in uvloop.loop.Loop.run_until_complete
 File "hoge.py", line 10, in run_subprocess
 stdout=sys.stdout,
 File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
 return self.gen.send(None)
 File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/subprocess.py", line 225, in create_subprocess_exec
 stderr=stderr, **kwds)
 File "uvloop/loop.pyx", line 2368, in __subprocess_run
 File "uvloop/handles/process.pyx", line 564, in uvloop.loop.UVProcessTransport.new
 File "uvloop/handles/process.pyx", line 49, in uvloop.loop.UVProcess._init
 File "uvloop/handles/process.pyx", line 38, in uvloop.loop.UVProcess._init
 File "uvloop/handles/process.pyx", line 254, in uvloop.loop.UVProcess._init_options
 File "uvloop/handles/process.pyx", line 456, in uvloop.loop.UVProcessTransport._init_files
ValueError: invalid stdout argument value 1

[achimnol]

I'm facing the same/similar error in my Travis CI environment.
My uvloop version is 0.8.1 and pytest version is 3.5.0, and I'm passing stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL.

 proc = await asyncio.create_subprocess_exec(*[
 'python', '-m', 'ai.backend.agent.stats',
 f'tcp://127.0.0.1:{stats_port}',
 cid,
 '--type', collection_type,
> ], **output_opts)
tests/test_stats.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.6/asyncio/subprocess.py:225: in create_subprocess_exec
 stderr=stderr, **kwds)
uvloop/loop.pyx:2164: in __subprocess_run (uvloop/loop.c:39419)
 ???
uvloop/handles/process.pyx:549: in uvloop.loop.UVProcessTransport.new (uvloop/loop.c:93810)
 ???
uvloop/handles/process.pyx:49: in uvloop.loop.UVProcess._init (uvloop/loop.c:84790)
 ???
uvloop/handles/process.pyx:38: in uvloop.loop.UVProcess._init (uvloop/loop.c:84510)
 ???
uvloop/handles/process.pyx:239: in uvloop.loop.UVProcess._init_options (uvloop/loop.c:88007)
 ???
uvloop/handles/process.pyx:415: in uvloop.loop.UVProcessTransport._init_files (uvloop/loop.c:91046)
 ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_pytest.capture.DontReadFromInput object at 0x7f4717c82860>
 def fileno(self):
> raise UnsupportedOperation("redirected stdin is pseudofile, "
 "has no fileno()")
E io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()
../../../virtualenv/agent/lib/python3.6/site-packages/_pytest/capture.py:583: UnsupportedOperation

[1st1]

Fixed in master. Please test!

[spumer]

In 0.14.0 still present

platform darwin -- Python 3.6.10, pytest-5.4.2, py-1.8.1, pluggy-0.13.1

Code failed with same issue:

 def _parse_list(output, files_patt):
 patt = r'(\d+)\s+.{16}\s+(%s)' % files_patt
 return re.findall(patt, output)

 async def list_files(cls, rar_file: str, files_patt):
 process = await asyncio.create_subprocess_exec(
 'unrar', 'l', rar_file,
 stdout=asyncio.subprocess.PIPE,
 )
 output, _ = await process.communicate()
 if process.returncode != os.EX_OK:
 raise RuntimeError(
 f'unrar failed: return_code={process.returncode!r}')

 return [(int(size), name) for size, name in _parse_list(output.decode(), files_patt)]

csv_in_rar_count = await list_files(rar_path, '(?i).*CSV')

[spumer]

@1st1

 process = await asyncio.create_subprocess_exec(*cmd)
../../../../../.pyenv/versions/3.7.8/lib/python3.7/asyncio/subprocess.py:217: in create_subprocess_exec
 stderr=stderr, **kwds)
uvloop/loop.pyx:2749: in subprocess_exec
 ???
uvloop/loop.pyx:2707: in __subprocess_run
 ???
uvloop/handles/process.pyx:596: in uvloop.loop.UVProcessTransport.new
 ???
uvloop/handles/process.pyx:60: in uvloop.loop.UVProcess._init
 ???
uvloop/handles/process.pyx:49: in uvloop.loop.UVProcess._init
 ???
uvloop/handles/process.pyx:274: in uvloop.loop.UVProcess._init_options
 ???
uvloop/handles/process.pyx:458: in uvloop.loop.UVProcessTransport._init_files
 ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.capture.DontReadFromInput object at 0x114fc6590>

 def fileno(self) -> int:
> raise UnsupportedOperation("redirected stdin is pseudofile, has no fileno()")
E io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()

../../.venv/lib/python3.7/site-packages/_pytest/capture.py:230: UnsupportedOperation

[blakewatters]

Whatever the underlying problem is here, it definitely has not been closed out. I have found that it can be worked around in tests where it pops up by using the default event loop policy via a pytest mark at the top of my test files that hit the issue. Hopefully, this helps until we can nail down the underlying issue:

# NOTE: These tests are brittle when run under uvloop. We run these under the default
# asyncio event loop policy to avoid exceptions relating to pytest output capture.
# The exception is: `io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()`
pytestmark = [pytest.mark.asyncio, pytest.mark.event_loop_policy("default")]

[fantix]

redirected stdin is pseudofile, has no fileno()

This error message is from pytest. If you try to read from stdin, it also says:

pytest: reading from stdin while output is captured! Consider using -s.

I'm checking why asyncio doesn't fail this test.