[Windows] Hang on startup if stdin refers to a pipe with an outstanding concurrent operation on Windows

[izbyshev]

BPO	34780
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @izbyshev
PRs	bpo-34780: Fix potential hang during stdin initialization on Windows #9520

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-09-23.20:00:35.593>
labels = ['type-bug', '3.8', '3.9', 'expert-IO', '3.10', 'OS-windows']
title = '[Windows] Hang on startup if stdin refers to a pipe with an outstanding concurrent operation on Windows'
updated_at = <Date 2021-03-12.21:40:42.549>
user = 'https://github.com/izbyshev'

bugs.python.org fields:

activity = <Date 2021-03-12.21:40:42.549>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows', 'IO']
creation = <Date 2018-09-23.20:00:35.593>
creator = 'izbyshev'
dependencies = []
files = []
hgrepos = []
issue_num = 34780
keywords = ['patch']
message_count = 5.0
messages = ['326175', '326201', '327802', '388548', '388549']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'izbyshev']
pr_nums = ['9520']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34780'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[izbyshev]

In the following code inspired by a production issue I had to debug recently subprocess.call() won't return:

import os
import subprocess
import sys
import time

r, w = os.pipe()
p1 = subprocess.Popen([sys.executable, '-c',
'import sys; sys.stdin.read()'],
stdin=r)

time.sleep(1)
subprocess.call([sys.executable, '-c', ''], stdin=r)

os.close(w)
p1.wait()

The underlying reason is the same as in bpo-22976. Python performs certain operations on stdin during it's initialization (different in 2.7 and 3.x), which block because there is an outstanding ReadFile() on the pipe end stdin refers to. Assuming that subprocess.call() runs some app that doesn't use stdin at all, if a developer doesn't control how the app is run (which was my case), I don't see any way to workaround this in pure Python. (An obvious workaround is to make a wrapper which closes stdin or redirects it to something else, but this wrapper can't be run with CPython).

I propose to fix this in CPython. The details are slightly different for 2.7 and 3.x.

2.7 calls fstat(stdin) in dircheck() (Objects/fileobject.c). This hangs because msvcrt calls PeekNamedPipe() if stdin refers to a pipe. Ironically, this fstat() call is completely useless on Windows because msvcrt never sets S_IFDIR in st_mode (it can't distinguish between a file and a directory because it uses GetFileType() and doesn't perform extra checks). I've implemented a PR that skips dircheck() on Windows. (If we do want to add a proper dircheck() to 2.7, it should do something similar to 3.x).

3.x performs the dir check without relying on fstat(), but it also calls lseek() (in _buffered_init() (Modules/_io/bufferedio.c), if removed, there is another one in _io_TextIOWrapper___init___impl (Modules/_io/textio.c). mscvrt calls SetFilePointerEx(), which hangs too, which is somewhat surprising because its docs [1] say:

You cannot use the SetFilePointerEx function with a handle to a nonseeking device such as a pipe or a communications device.

The wording is unclear though -- it doesn't say what happens if I try. lseek() docs [2] contain the following:

On devices incapable of seeking (such as terminals and printers), the return value is undefined.

In practice, lseek() succeeds on pipes on Windows, but is nearly useless:

Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
>>> import os
>>> r, w = os.pipe()
>>> os.write(w, b'xyz')
3
>>> os.lseek(r, 0, os.SEEK_CUR)
0
>>> os.lseek(r, 0, os.SEEK_END)
3
>>> os.lseek(r, 2, os.SEEK_SET)
2
>>> os.read(r, 1)
b'x'
>>> os.lseek(r, 0, os.SEEK_CUR)
2
>>> os.read(r, 1)
b'y'
>>> os.lseek(r, 0, os.SEEK_CUR)
2
>>> os.lseek(r, 0, os.SEEK_END)
1

So lseek() can be used to check the current pipe buffer size, and that seems about it. Given the above, I suggest two solutions for the hang on Windows:

1. Make lseek() fail on pipes on Windows, as it does on Unix. A number of projects have already done that:

https://referencesource.microsoft.com/#mscorlib/system/io/filestream.cs,1029
https://go.googlesource.com/go/+/ce58a39fca067a19c505220c0c907ccf32793427/src/syscall/syscall_windows.go#374
https://trac.ffmpeg.org/ticket/986 (workaround: https://lists.ffmpeg.org/pipermail/ffmpeg-cvslog/2012-June/051590.html)
https://github.com/erikd/libsndfile/blob/123cb9f9a5a356b951a23e9e2ab8527f967425cc/src/file_io.c#L266

2. Delay lseek() until it's really needed. In both cases (BufferedIO and TextIO), lseek() is used to set some cached fields, so ISTM it's not necessary to do it during initialization. This would also be an optimization (skip lseek() syscall until a user really wants to tell()/seek()). This can be done as a sole fix or can be combined with the above (as an optimization).

I'd like to hear other people's opinions before doing anything for Python 3.

[1] https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-setfilepointerex
[2] https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/lseek-lseeki64

[eryksun]

lseek() succeeds on pipes on Windows, but is nearly useless

lseek isn't meaningful for pipe and character files (i.e. FILE_TYPE_PIPE and FILE_TYPE_CHAR). While SEEK_SET and SEEK_CUR operations trivially succeed in these cases, the underlying device simply ignores the current file position. I think it would be reasonable to fail these cases instead of succeeding misleadingly.

When a file is opened for synchronous access, its FilePositionInformation is managed by the I/O manager, not the device or file system. All the I/O manager does is get or set the CurrentByteOffset value in the File object 1. It doesn't matter whether the device actually uses this information.

Regarding the observed SEEK_END behavior, the named-pipe file system (NPFS) supports querying the FileStandardInformation of a pipe, in which it sets the EndOfFile value as the number of bytes available to be read from the pipe's inbound (server-side) queue. So SEEK_END (or WinAPI FILE_END) does provide some information to us, but it's misleading because the seek itself is meaningless.

[izbyshev]

Ping!

Thanks to @eryksun for providing feedback here and for the patch review.

[eryksun]

Console input handles pose the same risk of hanging indefinitely when io.FileIO is used in legacy standard I/O mode (i.e. PYTHONLEGACYWINDOWSSTDIO).

Seeking could simply be disallowed on all files that aren't FILE_TYPE_DISK. For example, change portable_lseek() in Modules/_io/fileio.c to check the file type:

 if (GetFileType((HANDLE)_get_osfhandle(fd)) != FILE_TYPE_DISK) {
 errno = ESPIPE;
 res = -1;
 } else {
 res = _lseeki64(fd, pos, whence);
 }

[eryksun]

If non-disk files are made non-seekable in Windows, this will also resolve bpo-42602.