Subprocess timeout causes output to be returned as bytes in text mode

[macdjord]

BPO	43431
Nosy	@giampaolo, @eryksun
Files	test_subprocess.py: Demonstration of issue

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-08.05:41:46.323>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'Subprocess timeout causes output to be returned as bytes in text mode'
updated_at = <Date 2021-03-30.18:57:45.244>
user = 'https://bugs.python.org/macdjord'

bugs.python.org fields:

activity = <Date 2021-03-30.18:57:45.244>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-03-08.05:41:46.323>
creator = 'macdjord'
dependencies = []
files = ['49856']
hgrepos = []
issue_num = 43431
keywords = []
message_count = 3.0
messages = ['388257', '388265', '388272']
nosy_count = 3.0
nosy_names = ['giampaolo.rodola', 'eryksun', 'macdjord']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43431'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[macdjord]

Passing the argument text=True to subprocess.run() is supposed to mean that any captured output of the called process is automatically decoded and retuned to the user as test instead of bytes.

However, if you give a timeout and that timeout expires, the raised subprocess.TimeoutExpired exception will have the captured output as as bytes even if text mode is enabled.

Test output:
bash-5.0$ python3 test_subprocess.py
Version and interpreter information: namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-37', hexversion=50792432, name='cpython', version=sys.version_info(major=3, minor=7, micro=7, releaselevel='final', serial=0))
Completed STDOUT Type: <class 'str'>
Completed STDOUT Content: 'Start\nDone\n'
Timeout STDOUT Type: <class 'bytes'>
Timeout STDOUT Content: b'Start\n'

[eryksun]

communicate() is incomplete, so decoding the output may fail. For example, say the encoding is UTF-8, and the last multibyte character sequence (2-4 bytes) is incomplete. Maybe communicate() should always set stdout_bytes and stderr_bytes attributes on the timeout exception, and, in text mode, try to decode the output as stdout and/or stderr. If decoding fails, set the decoded value to None.

In Windows, run() tries to complete communication, which is dysfunctional in cases. I created bpo-43346 to propose changing the design in Windows, in order to address 3 cases that can cause subprocess.run() to ignore the given timeout. The proposed change also sets an incomplete read of stdout and stderr as bytes objects, regardless of text mode, because I was simply matching what POSIX does in this case.

[macdjord]

Eryk Sun: Well, I think step 1 should be to update the documentation for Python 3.7 through 3.10 on subprocess.run() and subprocess.TimeoutExpired to clearly state that TimeoutExpired.stdout and TimeoutExpired.stderr will be in bytes format even if text mode is set.

If we went with the model of having stdout_bytes and attempting to decode into stdout, we'd want an option to ignore a trailing decoding error.