gh-137400: Fix a crash when disabling profiling across all threads by colesbury · Pull Request #137471 · python/cpython

colesbury · 2025-08-06T15:43:46Z

The PyEval_SetProfileAllThreads function and other related functions had a race condition on tstate->c_profilefunc that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call.

There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads.

Issue: sys._setprofileallthreads race condition #137400

The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads.

colesbury · 2025-08-07T15:28:27Z

~~Closing in favor of the more complete PR: #137518~~

bedevere-bot · 2025-08-07T18:57:23Z

🤖 New build scheduled with the buildbot fleet by @colesbury for commit 5e5e4f1 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F137471%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

miss-islington-app · 2025-08-11T15:41:48Z

Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

…ads (pythongh-137471) The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads. (cherry picked from commit 3626928) Co-authored-by: Sam Gross <colesbury@gmail.com>

miss-islington-app · 2025-08-11T15:42:00Z

Sorry, @colesbury, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 362692852f13cdd1d33cc7ed35c0cbac7af1a785 3.13

bedevere-app · 2025-08-11T15:42:04Z

GH-137648 is a backport of this pull request to the 3.14 branch.

…ll threads (pythongh-137471) The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads. (cherry picked from commit 3626928) Co-authored-by: Sam Gross <colesbury@gmail.com>

bedevere-app · 2025-08-11T15:45:38Z

GH-137649 is a backport of this pull request to the 3.13 branch.

…eads (gh-137471) (gh-137649) The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads. (cherry picked from commit 3626928)

…eads (gh-137471) (#137648) Co-authored-by: Sam Gross <colesbury@gmail.com>

bedevere-bot · 2025-08-12T14:53:20Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Android 3.14 (tier-3) has failed when building commit 4ebd928.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1794/builds/435) and take a look at the build logs.
Check if the failure is related to this commit (4ebd928) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1794/builds/435

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 24, done. remote: Counting objects: 4% (1/24) remote: Counting objects: 8% (2/24) remote: Counting objects: 12% (3/24) remote: Counting objects: 16% (4/24) remote: Counting objects: 20% (5/24) remote: Counting objects: 25% (6/24) remote: Counting objects: 29% (7/24) remote: Counting objects: 33% (8/24) remote: Counting objects: 37% (9/24) remote: Counting objects: 41% (10/24) remote: Counting objects: 45% (11/24) remote: Counting objects: 50% (12/24) remote: Counting objects: 54% (13/24) remote: Counting objects: 58% (14/24) remote: Counting objects: 62% (15/24) remote: Counting objects: 66% (16/24) remote: Counting objects: 70% (17/24) remote: Counting objects: 75% (18/24) remote: Counting objects: 79% (19/24) remote: Counting objects: 83% (20/24) remote: Counting objects: 87% (21/24) remote: Counting objects: 91% (22/24) remote: Counting objects: 95% (23/24) remote: Counting objects: 100% (24/24) remote: Counting objects: 100% (24/24), done. remote: Compressing objects: 7% (1/13) remote: Compressing objects: 15% (2/13) remote: Compressing objects: 23% (3/13) remote: Compressing objects: 30% (4/13) remote: Compressing objects: 38% (5/13) remote: Compressing objects: 46% (6/13) remote: Compressing objects: 53% (7/13) remote: Compressing objects: 61% (8/13) remote: Compressing objects: 69% (9/13) remote: Compressing objects: 76% (10/13) remote: Compressing objects: 84% (11/13) remote: Compressing objects: 92% (12/13) remote: Compressing objects: 100% (13/13) remote: Compressing objects: 100% (13/13), done. remote: Total 13 (delta 11), reused 0 (delta 0), pack-reused 0 (from 0) 
From https://github.com/python/cpython
* branch 3.14 -> FETCH_HEAD
Note: switching to '4ebd928b0096ca1dd78d5d3e6c787dbc40a43c11'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 4ebd928b009 [3.14] gh-137400: Fix a crash when disabling profiling across all threads (gh-137471) (#137648)
Switched to and reset branch '3.14'
configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 192k 100 192k 0 0 1474k 0 --:--:-- --:--:-- --:--:-- 1474k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 42455 100 42455 0 0 318k 0 --:--:-- --:--:-- --:--:-- 318k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
6 5041k 6 333k 0 0 2019k 0 0:00:02 --:--:-- 0:00:02 2019k
100 5041k 100 5041k 0 0 24.6M 0 --:--:-- --:--:-- --:--:-- 135M
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1257k 100 1257k 0 0 9634k 0 --:--:-- --:--:-- --:--:-- 9634k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 635k 100 635k 0 0 5066k 0 --:--:-- --:--:-- --:--:-- 5066k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 479k 100 479k 0 0 2199k 0 --:--:-- --:--:-- --:--:-- 2199k
../../configure: line 4071: pkg-config: command not foundconfigure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
../../Python/fileutils.c:460:1: warning: unused function 'decode_current_locale' [-Wunused-function]
460 | decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen,
|^~~~~~~~~~~~~~~~~~~~~
../../Python/fileutils.c:679:1: warning: unused function 'encode_current_locale' [-Wunused-function]
679 | encode_current_locale(const wchar_t *text, char **str,
|^~~~~~~~~~~~~~~~~~~~~
2 warnings generated.
../../Modules/_localemodule.c:148:1: warning: unused function 'is_all_ascii' [-Wunused-function]
148 | is_all_ascii(const char *str)
|^~~~~~~~~~~~
1 warning generated.
../../Modules/_hacl/Lib_Memzero0.c:66:6: warning: "Your platform does not support any safe implementation of memzero -- consider a pull request!" [-W#warnings]
66 | #warning "Your platform does not support any safe implementation of memzero -- consider a pull request!"
|^
1 warning generated.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 8784 100 8784 0 0 67630 0 --:--:-- --:--:-- --:--:-- 68093
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2894 100 2894 0 0 9811 0 --:--:-- --:--:-- --:--:-- 9843
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 43504 100 43504 0 0 696k 0 --:--:-- --:--:-- --:--:-- 708k

bedevere-bot · 2025-08-12T15:54:57Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Android 3.14 (tier-3) has failed when building commit 4ebd928.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1794/builds/436) and take a look at the build logs.
Check if the failure is related to this commit (4ebd928) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1794/builds/436

Summary of the results of the build (if available):

Click to see traceback logs

Note: switching to '4ebd928b0096ca1dd78d5d3e6c787dbc40a43c11'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 4ebd928b009 [3.14] gh-137400: Fix a crash when disabling profiling across all threads (gh-137471) (#137648)
Switched to and reset branch '3.14'
configure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 192k 100 192k 0 0 1201k 0 --:--:-- --:--:-- --:--:-- 1201k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 42455 100 42455 0 0 419k 0 --:--:-- --:--:-- --:--:-- 419k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 5041k 100 5041k 0 0 30.7M 0 --:--:-- --:--:-- --:--:-- 30.7M
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1257k 100 1257k 0 0 11.6M 0 --:--:-- --:--:-- --:--:-- 11.6M
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 635k 100 635k 0 0 4794k 0 --:--:-- --:--:-- --:--:-- 4794k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 479k 100 479k 0 0 4090k 0 --:--:-- --:--:-- --:--:-- 4090k
../../configure: line 4071: pkg-config: command not foundconfigure: WARNING: no system libmpdec found; falling back to bundled libmpdec (deprecated and scheduled for removal in Python 3.16)configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
../../Python/fileutils.c:460:1: warning: unused function 'decode_current_locale' [-Wunused-function]
460 | decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen,
|^~~~~~~~~~~~~~~~~~~~~
../../Python/fileutils.c:679:1: warning: unused function 'encode_current_locale' [-Wunused-function]
679 | encode_current_locale(const wchar_t *text, char **str,
|^~~~~~~~~~~~~~~~~~~~~
2 warnings generated.
../../Modules/_localemodule.c:148:1: warning: unused function 'is_all_ascii' [-Wunused-function]
148 | is_all_ascii(const char *str)
|^~~~~~~~~~~~
1 warning generated.
../../Modules/_hacl/Lib_Memzero0.c:66:6: warning: "Your platform does not support any safe implementation of memzero -- consider a pull request!" [-W#warnings]
66 | #warning "Your platform does not support any safe implementation of memzero -- consider a pull request!"
|^
1 warning generated.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 8784 100 8784 0 0 142k 0 --:--:-- --:--:-- --:--:-- 142k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2894 100 2894 0 0 55255 0 --:--:-- --:--:-- --:--:-- 55653
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 43504 100 43504 0 0 926k 0 --:--:-- --:--:-- --:--:-- 944k

* main: pythongh-137288: Update 3.14 magic numbers (pythonGH-137665) pythongh-135228: When @DataClass(slots=True) replaces a dataclass, make the original class collectible (take 2) (pythonGH-137047) pythongh-126008: Improve docstrings for Tkinter cget and configure methods (pythonGH-133303) pythongh-131885: Use positional-only markers for ``max()`` and ``min()`` (python#131868) pythonGH-137426: Remove code deprecation of `importlib.abc.ResourceLoader` (pythonGH-137567) pythongh-125897: Mark range function parameters as positional only (python#125945) pythongh-137400: Fix a crash when disabling profiling across all threads (pythongh-137471) pythongh-115766: Fix IPv4Interface.is_unspecified (pythonGH-137326) pythongh-128813: cleanup C-API docs for PyComplexObject (pythonGH-137579) pythongh-135953: Profile a module or script with sampling profiler (python#136777) Fix documentation of hash in PyHash_FuncDef (python#137595)

hugovk · 2025-08-13T10:58:03Z

(These were temporary failures, fixed by a buildbot restart: #137186 (comment))

…ads (pythongh-137471) The `PyEval_SetProfileAllThreads` function and other related functions had a race condition on `tstate->c_profilefunc` that could lead to a crash when disable profiling or tracing on all threads while another thread is starting to profile or trace a a call. There are still potential crashes when threads exit concurrently with profiling or tracing be enabled/disabled across all threads.

…ll threads (pythongh-137471) (python#137648) Co-authored-by: Sam Gross <colesbury@gmail.com>

bedevere-appBot mentioned this pull request Aug 6, 2025
sys._setprofileallthreads race condition #137400
Closed

godlygeek mentioned this pull request Aug 6, 2025
gh-137400: Stop the world when swapping profile functions #137472
Closed

colesbury closed this Aug 7, 2025

colesbury reopened this Aug 7, 2025

colesbury marked this pull request as ready for review August 7, 2025 18:55

bedevere-appBot added the awaiting core review label Aug 7, 2025

colesbury added topic-free-threading needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Aug 7, 2025

colesbury requested review from hugovk and pablogsal August 7, 2025 18:57

colesbury added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Aug 7, 2025

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Aug 7, 2025

kumaraditya303 approved these changes Aug 11, 2025
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting core review labels Aug 11, 2025

colesbury merged commit 3626928 into python:mainAug 11, 2025
107 checks passed

colesbury deleted the gh-137400-profileall branch August 11, 2025 15:41

bedevere-appBot removed the awaiting merge label Aug 11, 2025

bedevere-appBot removed the needs backport to 3.14 bugs and security fixes label Aug 11, 2025

bedevere-appBot removed the needs backport to 3.13 bugs and security fixes label Aug 11, 2025

hugovk pushed a commit that referenced this pull request Aug 12, 2025
[3.14]gh-137400: Fix a crash when disabling profiling across all thr…
4ebd928
…eads (gh-137471) (#137648) Co-authored-by: Sam Gross <colesbury@gmail.com>

kumaraditya303 pushed a commit to miss-islington/cpython that referenced this pull request Sep 9, 2025
[3.14]pythongh-137400: Fix a crash when disabling profiling across a…
90dc6d8
…ll threads (pythongh-137471) (python#137648) Co-authored-by: Sam Gross <colesbury@gmail.com>

Navigation Menu

Uh oh!

Uh oh!

gh-137400: Fix a crash when disabling profiling across all threads - #137471

gh-137400: Fix a crash when disabling profiling across all threads#137471
colesbury merged 1 commit into
python:mainfrom
colesbury:gh-137400-profileall

colesbury commented Aug 6, 2025•
edited by bedevere-app Bot
Loading

Uh oh!

colesbury commented Aug 7, 2025•
edited
Loading

Uh oh!

bedevere-bot commented Aug 7, 2025

Uh oh!

Uh oh!

miss-islington-appBot commented Aug 11, 2025

Uh oh!

miss-islington-appBot commented Aug 11, 2025

Uh oh!

bedevere-appBot commented Aug 11, 2025

Uh oh!

bedevere-appBot commented Aug 11, 2025

Uh oh!

bedevere-bot commented Aug 12, 2025

Uh oh!

bedevere-bot commented Aug 12, 2025

Uh oh!

hugovk commented Aug 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

colesbury commented Aug 6, 2025• edited by bedevere-app BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colesbury commented Aug 7, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Aug 7, 2025

Uh oh!

Uh oh!

miss-islington-appBot commented Aug 11, 2025

Uh oh!

miss-islington-appBot commented Aug 11, 2025

Uh oh!

bedevere-appBot commented Aug 11, 2025

Uh oh!

bedevere-appBot commented Aug 11, 2025

Uh oh!

bedevere-bot commented Aug 12, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Aug 12, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

hugovk commented Aug 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

colesbury commented Aug 6, 2025•
edited by bedevere-app Bot
Loading

colesbury commented Aug 7, 2025•
edited
Loading