bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread by asvetlov · Pull Request #14344 · python/cpython

asvetlov · 2019-06-24T15:30:15Z

The resurrection of #13630
Now without dangling threads in tests :)

https://bugs.python.org/issue35621

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

asvetlov · 2019-06-24T15:38:22Z

I tested it with manual adding 1-second timeout in spawned waiter threads to emulate slow test machine.
After making sure that ./python -Werror -m test -v --fail-env-changed test_asyncio doesn't produce an error I removed the sleep.

asvetlov · 2019-06-24T15:43:13Z

 def add_child_handler(self, pid, callback, *args):
 assert self._forks, "Must use the context manager"

- if self._loop is None:


The check is replaced by watcher.is_active() call in subprocess transport

1st1

LGTM overall. Thanks for pushing this through, Andrew.

Co-Authored-By: Yury Selivanov <yury@magic.io>

asvetlov · 2019-06-30T09:00:21Z

Thanks for review!

miss-islington · 2019-06-30T09:55:02Z

Thanks @asvetlov for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

bedevere-bot · 2019-06-30T09:55:16Z

GH-14484 is a backport of this pull request to the 3.8 branch.

…ted in non-main thread (pythonGH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

…ted in non-main thread (pythonGH-14344)

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com>

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com> (cherry picked from commit 7080a8b)

With Python 3.7 asynchronous tests randomly fail with "RuntimeError: Event loop is closed" during shutdown. Backtrace of the error includes `SafeChildWatcher` calls. Python 3.8 has replaced `SafeChildWatcher` with a new `ThreadedChildWatcher` by default [1] as a bugfix for "asyncio.create_subprocess_exec() only works with main event loop" bug [2]. Python 3.7 scheduled end of life is 2023-06-27 according to <https://devguide.python.org/versions/>. [1] python/cpython#14344 [2] https://bugs.python.org/issue35621

asvetlov added 7 commits June 4, 2019 02:32

Close pid waiters on watcher.close()
3695efd

Revert "Revert "bpo-35621: Support running subprocesses in asyncio wh…
3559581
…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

Merge branch 'dangling-threads-in-watcher' into threaded-waiter
b18e6b9

Merge remote-tracking branch 'upstream/master' into threaded-waiter
1bc1d8e

Work on
ef1e21c

Merge branch 'master' into threaded-waiter
c982e4a

Fix dangling thread warnings in tests
6912ec7

asvetlov requested review from 1st1 and gpshead as code owners June 24, 2019 15:30

the-knights-who-say-ni added the CLA signed label Jun 24, 2019

bedevere-bot added the awaiting core review label Jun 24, 2019

asvetlov commented Jun 24, 2019
View reviewed changes

Update documentation
efb19c0

asvetlov added topic-asyncio needs backport to 3.8 labels Jun 24, 2019

Add docstrings
eb07403

1st1 approved these changes Jun 30, 2019
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jun 30, 2019

asvetlovand others added 8 commits June 30, 2019 11:50

Update Doc/library/asyncio-policy.rst
6aa3357
Co-Authored-By: Yury Selivanov <yury@magic.io>

Fix doc text
25d9ac2

Update Doc/library/asyncio-policy.rst
ae3a706
Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Doc/library/asyncio-policy.rst
8baf48c
Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Doc/library/asyncio-policy.rst
6e6a783
Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Lib/asyncio/unix_events.py
1e1281a
Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Lib/asyncio/unix_events.py
936aeb0
Co-Authored-By: Yury Selivanov <yury@magic.io>

Update docs
cc5a752

asvetlov merged commit 0d671c0 into python:masterJun 30, 2019

bedevere-bot removed the awaiting merge label Jun 30, 2019

asvetlov deleted the threaded-waiter branch June 30, 2019 09:55

bedevere-bot removed the needs backport to 3.8 label Jun 30, 2019

miss-islington added a commit that referenced this pull request Jun 30, 2019
bpo-35621: Support running subprocesses in asyncio when loop is execu…
bf8cb31
…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
bpo-35621: Support running subprocesses in asyncio when loop is execu…
af4d7c9
…ted in non-main thread (pythonGH-14344)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
bpo-35621: Support running subprocesses in asyncio when loop is execu…
b340eb2
…ted in non-main thread (pythonGH-14344)

webknjaz mentioned this pull request Jan 25, 2021
pytest-xdist breaks asyncio-based code expecting to be run in the main thread pytest-dev/pytest-xdist#620
Closed

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021
Use ThreadedChildWatcher in tests where available
a7f5a2a
Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021
Use ThreadedChildWatcher in tests where available
7b7a324
Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz mentioned this pull request Jan 25, 2021
Re-introduce pytest-xdist in supported envs aio-libs/aiohttp#5431
Merged
5 tasks

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021
Use ThreadedChildWatcher in tests where available
e3df8bc
Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Feb 21, 2021
Use ThreadedChildWatcher in tests where available
0df32d5
Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Feb 24, 2021
Use ThreadedChildWatcher in tests where available
bea959a
Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

link2xt mentioned this pull request May 19, 2023
"Event loop is closed" error in _provide_clean_event_loop()pytest-dev/pytest-asyncio#548
Closed

link2xt mentioned this pull request May 19, 2023
ci: require Python 3.8 for deltachat-rpc-client chatmail/core#4410
Merged

gvanrossum mentioned this pull request Oct 15, 2023
test_asyncio.test_subprocess leaked dangling threads on PPC64LE Fedora Stable 3.x: #110205
Closed

Navigation Menu

Uh oh!

Uh oh!

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread - #14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344
asvetlov merged 17 commits into
python:masterfrom
asvetlov:threaded-waiter

asvetlov commented Jun 24, 2019•
edited by bedevere-bot
Loading

Uh oh!

asvetlov commented Jun 24, 2019

Uh oh!

asvetlovJun 24, 2019

Uh oh!

1st1 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asvetlov commented Jun 30, 2019

Uh oh!

miss-islington commented Jun 30, 2019

Uh oh!

bedevere-bot commented Jun 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Uh oh!

Conversation

asvetlov commented Jun 24, 2019• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asvetlov commented Jun 24, 2019

Uh oh!

asvetlovJun 24, 2019

Choose a reason for hiding this comment

Uh oh!

1st1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asvetlov commented Jun 30, 2019

Uh oh!

miss-islington commented Jun 30, 2019

Uh oh!

bedevere-bot commented Jun 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

asvetlov commented Jun 24, 2019•
edited by bedevere-bot
Loading