gh-108520: Fix bad fork detection in nested multiprocessing use case by albanD · Pull Request #108568 · python/cpython

albanD · 2023-08-28T14:36:41Z

gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the is_fork_ctx attribute would be left missing after the first deserialization.

Issue: Nested multiprocessing leads to AttributeError: is_fork_ctx with forkserver or spawn methods #108520

albanD · 2023-08-28T22:14:14Z

I'm not sure how to read the test failure? Are they actually relevant?

albanD · 2023-08-29T20:52:43Z

Rebased to check fi the windows test failure was just flaky.

albanD · 2023-08-30T15:17:44Z

All the updates are done.
Let me know if the a blurb is needed?
Also I guess commits are squashed before merging? let me know if you want me to cleanup the commits in the branch.

pitrou · 2023-08-30T15:18:28Z

Yes, a blurb is needed especially as it fixes a regression.
Yes, commits will be squashed automatically when merging :-)

pitrou

+1, thanks for the quick fix @albanD

pitrou · 2023-08-30T16:24:40Z

Ok, it took me a couple attempts to get the comments right from the GitHub UI :-) Will merge if CI is green.

bedevere-bot · 2023-08-30T17:16:37Z

There's a new commit after the PR has been approved.

@pitrou: please review the changes made to this pull request.

miss-islington · 2023-08-30T17:18:30Z

Thanks @albanD for the PR, and @pitrou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

… case (pythonGH-108568) pythongh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>

bedevere-bot · 2023-08-30T17:29:37Z

GH-108691 is a backport of this pull request to the 3.12 branch.

* main: pythongh-108520: Fix bad fork detection in nested multiprocessing use case (python#108568) pythongh-108590: Revert pythongh-108657 (commit 400a1ce) (python#108686) pythongh-108494: Argument Clinic: Document how to generate code that uses the limited C API (python#108584) Document Python build requirements (python#108646) pythongh-101100: Fix Sphinx warnings in the Logging Cookbook (python#108678) Fix typo in multiprocessing docs (python#108666) pythongh-108669: unittest: Fix documentation for TestResult.collectedDurations (python#108670) pythongh-108590: Fix sqlite3.iterdump for invalid Unicode in TEXT columns (python#108657) Revert "pythongh-103224: Use the realpath of the Python executable in `test_venv` (pythonGH-103243)" (pythonGH-108667) pythongh-106320: Remove private _Py_ForgetReference() (python#108664) Mention Ellipsis pickling in the docs (python#103660) Revert "Use non alternate name for Kyiv (pythonGH-108533)" (pythonGH-108649) pythongh-108278: Deprecate passing the first param of sqlite3.Connection callback APIs by keyword (python#108632) pythongh-108455: peg_generator: install two stubs packages before running mypy (python#108637) pythongh-107801: Improve the accuracy of io.IOBase.seek docs (python#108268)

bedevere-bot · 2023-08-30T18:10:56Z

GH-108692 is a backport of this pull request to the 3.11 branch.

…e case (GH-108568) (#108692) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>

…e case (GH-108568) (#108691) gh-108520: Fix bad fork detection in nested multiprocessing use case (GH-108568) gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization. --------- (cherry picked from commit add8d45) Co-authored-by: albanD <desmaison.alban@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>

vstinner · 2023-09-22T01:30:28Z

The test fails randomly. Can you please have a look? I proposed a fix.

bedevere-bot mentioned this pull request Aug 28, 2023
Nested multiprocessing leads to AttributeError: is_fork_ctx with forkserver or spawn methods #108520
Closed
2 tasks

bedevere-bot added the awaiting review label Aug 28, 2023

Support nesting and add test
ec4e7e0

albanD force-pushed the fix-issue-108520 branch from 9d43084 to ec4e7e0Compare August 29, 2023 20:52

pitrou reviewed Aug 30, 2023
View reviewed changes

Comment threadLib/multiprocessing/synchronize.py Outdated

pitrou reviewed Aug 30, 2023
View reviewed changes

Comment threadLib/test/_test_multiprocessing.py Outdated

pitrou reviewed Aug 30, 2023
View reviewed changes

Comment threadLib/test/_test_multiprocessing.py

Cleanup test code
1b62ab0

albanD commented Aug 30, 2023
View reviewed changes

Comment threadLib/multiprocessing/synchronize.py Outdated

albanD added 2 commits August 30, 2023 11:15

Properly set is_fork_ctx in set_state
9212635

Rename is_fork_ctx to _is_fork_ctx
c0ee443

📜🤖 Added by blurb_it.
19225fc

pitrou added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Aug 30, 2023

Blurb nits
61d402d

pitrou approved these changes Aug 30, 2023
View reviewed changes

pitrou reviewed Aug 30, 2023
View reviewed changes

Comment threadLib/multiprocessing/synchronize.py Outdated

pitrou added 2 commits August 30, 2023 18:19

Improve co
6eae967

Merge branch 'main' into fix-issue-108520
11e6b02

bedevere-bot added awaiting merge and removed awaiting review labels Aug 30, 2023

pitrou reviewed Aug 30, 2023
View reviewed changes

Comment threadLib/multiprocessing/synchronize.py Outdated

Update comment
051f326

pitrou enabled auto-merge (squash) August 30, 2023 16:25

pitrou merged commit add8d45 into python:mainAug 30, 2023

bedevere-bot added awaiting core review and removed awaiting merge labels Aug 30, 2023

bedevere-bot requested a review from pitrou August 30, 2023 17:16

bedevere-bot removed the awaiting core review label Aug 30, 2023

bedevere-bot removed the needs backport to 3.12 only security fixes label Aug 30, 2023

bedevere-bot removed the needs backport to 3.11 only security fixes label Aug 30, 2023

albanD deleted the fix-issue-108520 branch August 30, 2023 19:23

vstinner mentioned this pull request Sep 22, 2023
test_multiprocessing_fork: test_nested_startmethod() fails randomly. #109706
Closed

Uh oh!

Uh oh!

Conversation

albanD commented Aug 28, 2023• edited by pitrou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

albanD commented Aug 28, 2023

Uh oh!

albanD commented Aug 29, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albanD commented Aug 30, 2023

Uh oh!

pitrou commented Aug 30, 2023

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pitrou commented Aug 30, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Aug 30, 2023

Uh oh!

miss-islington commented Aug 30, 2023

Uh oh!

bedevere-bot commented Aug 30, 2023

Uh oh!

bedevere-bot commented Aug 30, 2023

Uh oh!

vstinner commented Sep 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

albanD commented Aug 28, 2023•
edited by pitrou
Loading

pitrou commented Aug 30, 2023•
edited
Loading