use os.posix_spawn in subprocess

[nanjekyejoannah]

BPO	35537
Nosy	@gpshead, @vstinner, @serhiy-storchaka, @koobs, @izbyshev, @pablogsal, @miss-islington, @nanjekyejoannah, @kevans91
PRs	bpo-35537: use os.posix_spawn in subprocess #11242 bpo-35537: subprocess uses os.posix_spawn in some cases #11452 bpo-35537: subprocess can use posix_spawn with pipes #11575 bpo-35537: subprocess can now use os.posix_spawnp #11579 Revert "bpo-35537: subprocess can now use os.posix_spawnp (GH-11579)" #11582 bpo-35537: Add setsid parameter to os.posix_spawn() and os.posix_spawnp(). #11608 bpo-35537: Document posix_spawn() change in subprocess #11668 bpo-35537: Fix docstring typos in _use_posix_spawn() #11684 subprocess: close pipes/fds by using ExitStack #11686 bpo-35537: Skip test_start_new_session() of posix_spawn #11718 [WIP] bpo-35537: Fix function name in os.posix_spawnp() errors #11719 bpo-35537: Rewrite setsid test for os.posix_spawn #11721 bpo-35537: Change subprocess to use posix_spawnp instead of posix_spawn #11917 [3.8] bpo-35537: Rewrite setsid test for os.posix_spawn (GH-11721) #14093 bpo-35537: Add a comment to _Py_RestoreSignals() #18792
Files	subprocess_bench.py subprocess_bench_stdout.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2019-06-05.08:27:39.280>
created_at = <Date 2018-12-19.16:47:59.385>
labels = ['3.8', 'type-feature', 'library']
title = 'use os.posix_spawn in subprocess'
updated_at = <Date 2020-03-05.16:41:09.872>
user = 'https://github.com/nanjekyejoannah'

bugs.python.org fields:

activity = <Date 2020-03-05.16:41:09.872>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2019-06-05.08:27:39.280>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2018-12-19.16:47:59.385>
creator = 'nanjekyejoannah'
dependencies = []
files = ['48011', '48057']
hgrepos = []
issue_num = 35537
keywords = ['patch', 'patch', 'patch']
message_count = 61.0
messages = ['332151', '332204', '332210', '332212', '332213', '332216', '332217', '332219', '332222', '332223', '332235', '332236', '332237', '332238', '332239', '332266', '332270', '332272', '332393', '332565', '333123', '333126', '333127', '333131', '333571', '333629', '333646', '333656', '333657', '333738', '333739', '333740', '333744', '333745', '333749', '333771', '333794', '333795', '333799', '333800', '333803', '333805', '333820', '333825', '333839', '333889', '333902', '333994', '334268', '334269', '334288', '334292', '334305', '334337', '334661', '334662', '334685', '340838', '344683', '345617', '345621']
nosy_count = 9.0
nosy_names = ['gregory.p.smith', 'vstinner', 'serhiy.storchaka', 'koobs', 'izbyshev', 'pablogsal', 'miss-islington', 'nanjekyejoannah', 'kevans']
pr_nums = ['11242', '11452', '11575', '11579', '11582', '11608', '11668', '11684', '11686', '11718', '11719', '11721', '11917', '14093', '18792']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35537'
versions = ['Python 3.8']

[nanjekyejoannah]

On Linux, posix_spawn() uses vfork() instead of fork() which blocks
the parent process. The child process executes exec() early and so we
don't pay the price of duplicating the memory pages (the thing which
tracks memory pages of a process).

On macOS, posix_spawn() is a system call, so the kernel is free to use
fast-paths to optimize it as they want.

posix_spawn() is faster but it's also safer: it allows us to do a lot of
"actions" before exec(), before executing the new program. For
example, you can close files and control signals. Again, on macOS,
these actions are "atomic" since it's a system call. On Linux, glibc uses a very good implementation which has no race condition.

Currently, Python uses a C extension _posixsubprocess which more or
less reimplements posix_spawn(): close file descriptors, make some
file descriptors inheritable or not, etc. It is very tricky to write
correct code: code run around fork() is very fragile. In theory, the
POSIX specification only allows to use "async-signal-safe" functions
after fork()...

So it would be great to avoid _posixsubprocess whenever possible for
(1) speed (2) correctness.

[vstinner]

"posix_spawn() is faster"

This assumption needs a benchmark!

I ran a benchmark on Linux (Fedora 29, Linux kernel 4.19.9-300.fc29.x86_64, glibc 2.28) using attached subprocess_bench.py.

I rebased the PR 11242 on master and run the benchmark in virtual environment:

git co pr/11242
# current branch: PR 11242
make distclean
./configure
make
./python -m venv env
env/bin/python -m pip install perf
env/bin/python subprocess_bench.py -v -o posix_spawn.json
git co master
env/bin/python subprocess_bench.py -v -o fork_exec.json
env/bin/python -m perf compare_to fork_exec.json posix_spawn.json

The result is quite explicit: the PR makes subprocess.Popen 61x faster!!!

$ env/bin/python -m perf compare_to fork_exec.json posix_spawn.json 
Mean +- std dev: [fork_exec] 27.1 ms +- 0.4 ms -> [posix_spawn] 447 us +- 163 us: 60.55x faster (-98%)

That's the best case:

• The parent process (Python) allocated 2 GiB of memory: that's not uncommon for large application. On OpenStack for example, it's common that a process takes more than 1 GiB.
• The child process has a short execution time and allocates few memory.

On my config (Fedora 29, Linux kernel 4.19.9-300.fc29.x86_64, glibc 2.28), os.posix_spawn() uses vfork():

$ strace -o trace ./python -c 'import subprocess; subprocess.run(["/bin/true"], close_fds=False, restore_signals=False)'
$ grep clone trace
clone(child_stack=0x7fab28ac0ff0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 23073

I guess that the 61x speedup mostly comes from vfork().

See also bpo-34663 for previous discussion about vfork() and os.posix_spawn(). In short, the glibc is smart and detects when vfork() can be used or not.

[vstinner]

Benchmark on macOS Mojave (10.14.2), x86_64:

$ ./python.exe -m perf compare_to fork_exec.json posix_spawn.json 
Mean +- std dev: [fork_exec] 2.79 ms +- 0.06 ms -> [posix_spawn] 1.64 ms +- 0.03 ms: 1.70x faster (-41%)

The speedup is "only" 1.7x faster, it's less impressive. Maybe fork()+exec() is more efficient on macOS than on Linux.

[vstinner]

Benchmark on FreeBSD:

$ env/bin/python -m perf compare_to fork_exec.json posix_spawn.json 
Mean +- std dev: [fork_exec] 52.8 ms +- 4.7 ms -> [posix_spawn] 499 us +- 45 us: 105.91x faster (-99%)

Wow! 106x faster!

[vstinner]

See also:

"How a fix in Go 1.9 sped up our Gitaly service by 30x" (hint: posix_spawn)
https://about.gitlab.com/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/

fork/exec vs. posix_spawn benchmarks:
https://github.com/rtomayko/posix-spawn#benchmarks

[serhiy-storchaka]

We need to add tests for all corner cases. I.e. disabling any of conditions for posix_spawn should cause the failure.

We need to do also something with PyOS_BeforeFork/PyOS_AfterFork_Child/PyOS_AfterFork_Parent.

[vstinner]

We need to add tests for all corner cases. I.e. disabling any of conditions for posix_spawn should cause the failure.

Do you mean running tests twice, one with posix_spawn(), one without?

We need to do also something with PyOS_BeforeFork/PyOS_AfterFork_Child/PyOS_AfterFork_Parent.

Can you please elaborate? posix_spawn() may or may not use fork() internally.

[serhiy-storchaka]

Do you mean running tests twice, one with posix_spawn(), one without?

I mean that after writing tests they can be tested manually by disabling conditions for posix_spawn one by one. I.e. some tests should fail if remove "stdout is None" and some tests should fail if remove "not close_fds", etc.

Can you please elaborate? posix_spawn() may or may not use fork() internally.

_posixsubprocess.fork_exec() calls PyOS_BeforeFork/PyOS_AfterFork_Child/PyOS_AfterFork_Parent. If use os.posix_spawn(), these calls will be omitted. This is a behavior change. We should either call these functions manually before/after os.posix_spawn() (but I do not know what to do with PyOS_AfterFork_Child), or disable using os.posix_spawn() if non-trivial callbacks were added, or use other methods for calling registered callbacks. But the stdlib already adds callbacks in the threading and random modules.

[izbyshev]

Serhiy, PyOS_* functions are called only if preexec_fn != None. But it will never be possible to implement support for preexec_fn (and some other subprocess features, e.g. close_fds) if processes are run via posix_spawn, so I don't see why anything should be done with those functions.

[serhiy-storchaka]

Oh, nice! So this is not an obstacle for using os.posix_spawn().

[izbyshev]

Victor and Joannah, thanks for working on adding vfork() support to subprocess. Regarding speedups in the real world, I can share a personal anecdote. Back at the time when AOSP was built with make (I think it was AOSP 5) I've observed ~2x slowdown (or something close to that) if fork() is used instead of vfork() in make. This is the slowdown of the *whole* build (not just process creation time), and it's dramatic given the amount of pure computation (compilation of C++/Java code) involved in building AOSP. The underlying reason was that make merged all AOSP subproject makefiles into a single gigantic one, so make consumed more than 1 GB of RAM, and each shell invocation in recipes resulted in copying a large page table.

That said, I'm not sure that the chosen approach of adding posix_spawn() "fast paths" for particular combinations of arguments is an optimal way to expose vfork() benefits to most users. For example, close_fds is True by default in subprocess, and I don't think there is a reliable way to use posix_spawn() in this case. So users would need to use "magic" combinations of arguments to subprocess.Popen to benefit from vfork(). Another example is "cwd" parameter: there is no standard posix_spawn file action to change the working directory in the child, so such a seemingly trivial operation would trigger the "slow" fork-exec path.

Another approach would be to use vfork-exec instead fork-exec in _posixsubprocess. I know that previous discussions were resolved against vfork(), but I'm still not convinced and suggest to reevaluate those decisions again. Some arguments:

1. While POSIX contain scare-wording forbidding to do pretty anything in a vfork-spawned child, real systems where posix_spawn() is not a system call call vfork() and then execute all specified actions in the child. Cases where vfork() is avoided seem to be a quality-of-implementation issue, not a fundamental issue (for example, until recently glibc used fork() in some cases because of heap memory allocations, but they could be avoided). In practice, there is no problem with calling at least async-signal-safe functions in vfork-children, and there is no technical reason why there would be any problem.

2. _posixsubprocess is already very careful and calls only async-signal-safe functions in almost all cases (an obvious exception is preexec_fn, which is discouraged anyway).

3. Our use case for vfork() is restricted, so some concerns don't apply to us. For example, the setuid() problem outlined in Rich Felker's post (https://ewontfix.com/7) doesn't seem to apply. The problem with signal handlers from the same post should be possible to avoid except for preexec_fn, but we'd have to fallback to fork() in this case anyway due to memory allocation.

4. In the standard example of fork() unsafety in multithreaded processes (state of memory-based locks is copied to the child, and there could be nobody to unlock them), vfork() is *safer* than fork() because it still shares memory with the parent, and all threads other than the parent one are running. In particular, it should be perfectly safe to use any memory allocator that protects its state with something like futexes and atomics in a vfork-child even if other threads of the parent are using it concurrently.

5. Other language runtimes are already using vfork(). Java has been doing it for ages, and Victor referenced Go.

Some possible counter-arguments:

1. We seem to be reimplementing posix_spawn(). In fact, due to (2) above, we can fully reuse the existing fork-exec code, with additional tweaks that doesn't seem hard at this point.

2. My points above are based on Linux, but I don't know much about other Unix-likes, so in theory additional complications could exist. I can't, however, imagine any fundamental complication.

In the end, I think that migrating subprocess to vfork-exec would have more impact for users than adding "fast paths" and have consistent performance regardless of subprocess.Popen arguments (with few exceptions). Please consider it. Thanks!