gh-140815: Fix faulthandler for invalid/freed frame by vstinner · Pull Request #140921 · python/cpython

vstinner · 2025-11-02T18:15:12Z

faulthandler now detects if a frame or a code object is invalid or freed.

Add helper functions:

_PyCode_SafeAddr2Line()
_PyFrame_SafeGetCode()
_PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now also considers the pointer 0x1 as freed.

Issue: SEGV in faulthandler.dump_traceback_later #140815

faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetBytecode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now also considers the pointer 0x1 as freed.

sergey-miryanov · 2025-11-02T18:58:40Z

I believe this PR minimizes the chances of getting a segfault. However, I have a feeling that it does not eliminate the possibility entirely. Unfortunately, I don't have a better solution.

sergey-miryanov · 2025-11-02T19:07:12Z

Perhaps we could add a new function that attempts to attach the thread state with a timeout. If successful, we will have a more reliable solution. WDYT?

vstinner · 2025-11-02T20:10:28Z

Perhaps we could add a new function that attempts to attach the thread state with a timeout. If successful, we will have a more reliable solution. WDYT?

I don't think that it's worth it.

Another faulthandler feature is faulthandler.enable() and faulthandler.register(): register signal handlers. A Unix signal can occur anytime, so we cannot acquire the GIL or any other lock.

faulthandler is a debugger, it's not designed to be 100% accurate, but to "just work" in any case (GIL or not, in a signal handler or not, etc.) without taking any lock.

vstinner · 2025-11-02T20:17:43Z

@sergey-miryanov @picnixz: I pushed a change to address your reviews.

efimov-mikhail · 2025-11-02T20:31:50Z

LGTM, but I have very little expertise here, so I'll not push approve button :-).

vstinner · 2025-11-02T20:32:28Z

I believe this PR minimizes the chances of getting a segfault. However, I have a feeling that it does not eliminate the possibility entirely. Unfortunately, I don't have a better solution.

faulthandler is implemented with multiple heuristics and best effort. It's ok if it does crash, since it should only be triggered when something already goes wrong.

Only faulthandler.dump_traceback() should be safe since it's called with an attached thread safe.

sergey-miryanov

From point of my little experience, it looks good to me.

sergey-miryanov · 2025-11-02T20:44:21Z

Thanks!

ZeroIntensity · 2025-11-02T21:23:41Z

+ if (_PyMem_IsPtrFreed(ptr)) {
+ return NULL;
+ }
+ PyObject *executable = PyStackRef_AsPyObjectBorrow(f->f_executable);
+ if (_PyObject_IsFreed(executable)) {


Does this patch work on release builds? I thought _PyMem_IsPtrFreed and _PyObject_IsFreed only worked if debug allocators are enabled.

Oh. I didn't try on a release build! _PyMem_IsPtrFreed() and _PyObject_IsFreed() work better with debug allocators, you're correct. But they detect freed (reused) memory in some cases in a release build as well. They are just less reliable in release mode.
I pushed more changes to make the heuristics stricter.

vstinner · 2025-11-03T12:25:03Z

@picnixz @ZeroIntensity: Do these changes look reasonable to you? Even if they are not perfect, they allow dumping the traceback in more cases.

I wrote a script to run the reproducer in a loop (see below). I ran the stress test 3 times.

Debug build:
- Without these changes: crash after 1 run, crash after 6 runs, crash after 1 run
- With these changes: crash after 3970 runs, <stopped after 10,000 success without crash>, <stopped after 10,000 success without crash>.
Release build:
- Without these changes: crash after 6 runs, crash after 59 runs, crash after 36 runs
- With these changes: crash after 550 runs, crash after 3750 runs, crash after 7845 runs.

Stress test:

importsubprocess, syscmd=[sys.executable, 'reproducer.py']
success=0whileTrue:
proc=subprocess.run(cmd)
ifproc.returncode!=1:
print()
print(f"Exitcode {proc.returncode}")
breaksuccess+=1print(success)

reproducer.py:

importfaulthandlerimportsysfaulthandler.dump_traceback_later(10*1e-308, exit=True, file=sys.__stderr__)
assertFalse

ZeroIntensity

I think this is a good idea anyway, yeah. We might want to document that faulthandler.dump_traceback_later can sometimes crash during interpreter finalization.

ZeroIntensity · 2025-11-03T15:26:08Z

+// Similar to _PyFrame_GetCode(), but return NULL if the frame is invalid or
+// freed. Used by dump_frame() in Python/traceback.c.


It might be worth adding a comment that this isn't completely reliable.

Good idea. I added a sentence for that in the 3 added functions.

vstinner · 2025-11-04T10:48:46Z

Merged, thanks for your reviews!

miss-islington-app · 2025-11-04T10:51:52Z

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

…40921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c) Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-app · 2025-11-04T10:51:59Z

GH-140981 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2025-11-04T11:16:20Z

GH-140985 is a backport of this pull request to the 3.13 branch.

) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c)

#140981) gh-140815: Fix faulthandler for invalid/freed frame (GH-140921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c) Co-authored-by: Victor Stinner <vstinner@python.org>

…140985) gh-140815: Fix faulthandler for invalid/freed frame (#140921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c)

) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed.

vstinner requested review from ericsnowcurrently and markshannon as code owners November 2, 2025 18:15

bedevere-appBot added the awaiting core review label Nov 2, 2025

bedevere-appBot mentioned this pull request Nov 2, 2025
SEGV in faulthandler.dump_traceback_later#140815
Closed

Add NEWS entry
4059837

vstinner mentioned this pull request Nov 2, 2025
GH-140815: Acquire GIL or stop the world when calling _Py_DumpTracebackThreads from faulthandler_thread #140895
Closed

sergey-miryanov reviewed Nov 2, 2025
View reviewed changes

Comment threadInclude/internal/pycore_interpframe.h

sergey-miryanov reviewed Nov 2, 2025
View reviewed changes

Comment threadPython/traceback.c Outdated

picnixz reviewed Nov 2, 2025
View reviewed changes

Comment threadInclude/internal/pycore_interpframe.h Outdated
Comment threadInclude/internal/pycore_interpframe.h
Comment threadInclude/internal/pycore_interpframe.h

Address reviews
9b26187

efimov-mikhail reviewed Nov 2, 2025
View reviewed changes

Comment threadInclude/internal/pycore_code.h Outdated

Update Include/internal/pycore_code.h
7d9a4b8

sergey-miryanov approved these changes Nov 2, 2025
View reviewed changes

picnixz reviewed Nov 2, 2025
View reviewed changes

Comment threadPython/traceback.c Outdated
Comment threadPython/traceback.c

ZeroIntensity reviewed Nov 2, 2025
View reviewed changes

vstinner added 3 commits November 3, 2025 12:44

Enhance _PyMem_IsPtrFreed()
c208539

Stricter _PyFrame_SafeGetCode()
c273f99

coding style
3764548

ZeroIntensity approved these changes Nov 3, 2025
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting core review labels Nov 3, 2025

Document that safe functions are not 100% reliable
6239435

vstinner merged commit a84181c into python:mainNov 4, 2025
46 checks passed

vstinner deleted the faulthandler_freed branch November 4, 2025 10:48

bedevere-appBot removed the awaiting merge label Nov 4, 2025

vstinner added the needs backport to 3.14 bugs and security fixes label Nov 4, 2025

bedevere-appBot removed the needs backport to 3.14 bugs and security fixes label Nov 4, 2025

sergey-miryanov mentioned this pull request Nov 5, 2025
Segfault in faulthandler signal handler with threads #116008
Closed

		// Similar to _PyFrame_GetCode(), but return NULL if the frame is invalid or
		// freed. Used by dump_frame() in Python/traceback.c.

Uh oh!

Uh oh!

Conversation

vstinner commented Nov 2, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sergey-miryanov commented Nov 2, 2025

Uh oh!

sergey-miryanov commented Nov 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinner commented Nov 2, 2025

Uh oh!

vstinner commented Nov 2, 2025

Uh oh!

Uh oh!

efimov-mikhail commented Nov 2, 2025

Uh oh!

vstinner commented Nov 2, 2025

Uh oh!

sergey-miryanov left a comment

Choose a reason for hiding this comment

Uh oh!

sergey-miryanov commented Nov 2, 2025

Uh oh!

Uh oh!

Uh oh!

ZeroIntensityNov 2, 2025

Choose a reason for hiding this comment

Uh oh!

vstinnerNov 3, 2025

Choose a reason for hiding this comment

Uh oh!

vstinner commented Nov 3, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

ZeroIntensityNov 3, 2025

Choose a reason for hiding this comment

Uh oh!

vstinnerNov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner commented Nov 4, 2025

Uh oh!

miss-islington-appBot commented Nov 4, 2025

Uh oh!

bedevere-appBot commented Nov 4, 2025

Uh oh!

bedevere-appBot commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vstinner commented Nov 2, 2025•
edited
Loading

vstinner commented Nov 3, 2025•
edited
Loading