gh-117657: Fix some races in tracebacks by Fidget-Spinner · Pull Request #121748 · python/cpython

Fidget-Spinner · 2024-07-14T09:35:53Z

TSAN reported race:

WARNING: ThreadSanitizer: data race (pid=31237)
Read of size 8 at 0x7fffb9ac6148 by thread T333:
#0 PyException_GetTraceback /home/ken/Documents/GitHub/cpython/Objects/exceptions.c:415:12 (python+0x27c20b) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#1 _PyErr_SetObject /home/ken/Documents/GitHub/cpython/Python/errors.c:242:14 (python+0x52cf07) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#2 do_raise /home/ken/Documents/GitHub/cpython/Python/ceval.c:2031:5 (python+0x4dd3c9) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#3 _PyEval_EvalFrameDefault /home/ken/Documents/GitHub/cpython/Python/generated_cases.c.h:5532:21 (python+0x4dd3c9)
#4 _PyEval_EvalFrame /home/ken/Documents/GitHub/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x4b39b3) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#5 _PyEval_Vector /home/ken/Documents/GitHub/cpython/Python/ceval.c:1866:12 (python+0x4b39b3)
#6 _PyFunction_Vectorcall /home/ken/Documents/GitHub/cpython/Objects/call.c (python+0x25b2ac) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#7 _PyObject_VectorcallTstate /home/ken/Documents/GitHub/cpython/./Include/internal/pycore_call.h:167:11 (python+0x261410) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#8 method_vectorcall /home/ken/Documents/GitHub/cpython/Objects/classobject.c:70:20 (python+0x25f8fd) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#9 _PyVectorcall_Call /home/ken/Documents/GitHub/cpython/Objects/call.c:273:16 (python+0x25ac2a) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#10 _PyObject_Call /home/ken/Documents/GitHub/cpython/Objects/call.c:348:16 (python+0x25aed3) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#11 PyObject_Call /home/ken/Documents/GitHub/cpython/Objects/call.c:373:12 (python+0x25b0b7) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#12 thread_run /home/ken/Documents/GitHub/cpython/./Modules/_threadmodule.c:337:21 (python+0x6cbcd8) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#13 pythread_wrapper /home/ken/Documents/GitHub/cpython/Python/thread_pthread.h:243:5 (python+0x604b0b) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
Previous write of size 8 at 0x7fffb9ac6148 by main thread:
#0 BaseException_set_tb /home/ken/Documents/GitHub/cpython/Objects/exceptions.c:327:9 (python+0x27c4d8) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#1 PyException_SetTraceback /home/ken/Documents/GitHub/cpython/Objects/exceptions.c:422:12 (python+0x27c2fc) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#2 PyTraceBack_Here /home/ken/Documents/GitHub/cpython/Python/traceback.c:273:5 (python+0x605636) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#3 _PyEval_EvalFrameDefault /home/ken/Documents/GitHub/cpython/Python/ceval.c:927:17 (python+0x4e365d) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#4 _PyEval_EvalFrame /home/ken/Documents/GitHub/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x4b39b3) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#5 _PyEval_Vector /home/ken/Documents/GitHub/cpython/Python/ceval.c:1866:12 (python+0x4b39b3)

and

WARNING: ThreadSanitizer: data race (pid=31237)
Atomic read of size 4 at 0x7fffb9a8439c by thread T333:
#0 _Py_atomic_load_uint32_relaxed /home/ken/Documents/GitHub/cpython/./Include/cpython/pyatomic_gcc.h:339:10 (python+0x27c222) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#1 Py_INCREF /home/ken/Documents/GitHub/cpython/./Include/refcount.h:227:22 (python+0x27c222)
#2 Py_XINCREF /home/ken/Documents/GitHub/cpython/./Include/refcount.h:449:9 (python+0x27c222)
#3 _Py_XNewRef /home/ken/Documents/GitHub/cpython/./Include/refcount.h:481:5 (python+0x27c222)
#4 PyException_GetTraceback /home/ken/Documents/GitHub/cpython/Objects/exceptions.c:415:12 (python+0x27c222)
#5 _PyErr_SetObject /home/ken/Documents/GitHub/cpython/Python/errors.c:242:14 (python+0x52cf07) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#6 do_raise /home/ken/Documents/GitHub/cpython/Python/ceval.c:2031:5 (python+0x4dd3c9) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#7 _PyEval_EvalFrameDefault /home/ken/Documents/GitHub/cpython/Python/generated_cases.c.h:5532:21 (python+0x4dd3c9)
#8 _PyEval_EvalFrame /home/ken/Documents/GitHub/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x4b39b3) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#9 _PyEval_Vector /home/ken/Documents/GitHub/cpython/Python/ceval.c:1866:12 (python+0x4b39b3)
#10 _PyFunction_Vectorcall /home/ken/Documents/GitHub/cpython/Objects/call.c (python+0x25b2ac) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#11 _PyObject_VectorcallTstate /home/ken/Documents/GitHub/cpython/./Include/internal/pycore_call.h:167:11 (python+0x261410) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#12 method_vectorcall /home/ken/Documents/GitHub/cpython/Objects/classobject.c:70:20 (python+0x25f8fd) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#13 _PyVectorcall_Call /home/ken/Documents/GitHub/cpython/Objects/call.c:273:16 (python+0x25ac2a) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#14 _PyObject_Call /home/ken/Documents/GitHub/cpython/Objects/call.c:348:16 (python+0x25aed3) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#15 PyObject_Call /home/ken/Documents/GitHub/cpython/Objects/call.c:373:12 (python+0x25b0b7) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#16 thread_run /home/ken/Documents/GitHub/cpython/./Modules/_threadmodule.c:337:21 (python+0x6cbcd8) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#17 pythread_wrapper /home/ken/Documents/GitHub/cpython/Python/thread_pthread.h:243:5 (python+0x604b0b) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
Previous write of size 4 at 0x7fffb9a8439c by main thread:
#0 new_reference /home/ken/Documents/GitHub/cpython/Objects/object.c:2367:22 (python+0x32793a) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#1 _Py_NewReference /home/ken/Documents/GitHub/cpython/Objects/object.c:2386:5 (python+0x32793a)
#2 _PyObject_Init /home/ken/Documents/GitHub/cpython/./Include/internal/pycore_object.h:271:5 (python+0x544ee6) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#3 _PyObject_GC_New /home/ken/Documents/GitHub/cpython/Python/gc_free_threading.c:1677:5 (python+0x544ee6)
#4 tb_create_raw /home/ken/Documents/GitHub/cpython/Python/traceback.c:53:10 (python+0x605418) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#5 _PyTraceBack_FromFrame /home/ken/Documents/GitHub/cpython/Python/traceback.c:257:12 (python+0x605306) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#6 PyTraceBack_Here /home/ken/Documents/GitHub/cpython/Python/traceback.c:267:23 (python+0x6055a4) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
#7 _PyEval_EvalFrameDefault /home/ken/Documents/GitHub/cpython/Python/ceval.c:927:17 (python+0x4e365d) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)

Issue: Make TSAN tests pass with the GIL disabled in free-threaded builds #117657

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

iritkatriel · 2024-07-22T17:05:17Z

LGTM, but I'd suggest to have another reviewer with more knowledge of the thread safe patterns.

colesbury

I think the races in our tests involving tracebacks are broader than this.

The critical sections on getters and setters probably still makes sense, but I don't think they'll fix the traceback races.

The big problem is using sys._current_exceptions() to access exceptions for other threads while they are running. That function runs in a stop-the-world context, but the exception tracebacks are created lazily when other threads are resumed. I think that will still not be thread-safe even with this change.

colesbury · 2024-07-22T17:48:39Z

 op->ob_refcnt = 1;
 #else
- op->ob_tid = _Py_ThreadId();
+_Py_atomic_store_uintptr_release(&op->ob_tid, _Py_ThreadId());


I don't think we want to do this, at least not for now. I want to wait until we fix all the other races before we address new_reference because there are perf implications and the race is in practice benign.
The "release" memory order is also stronger (and more expensive on arm64) than necessary.

python-cla-bot · 2025-04-18T09:35:53Z

All commit authors signed the Contributor License Agreement.

kumaraditya303 · 2025-07-03T10:17:00Z

This is fixed on main under #128421

Fix races in tracebacks
6cb8897

Fidget-Spinner added the skip news label Jul 14, 2024

Fidget-Spinner requested a review from iritkatriel as a code owner July 14, 2024 09:35

Fidget-Spinner added the topic-free-threading label Jul 14, 2024

bedevere-appBot added the awaiting core review label Jul 14, 2024

bedevere-appBot mentioned this pull request Jul 14, 2024
Make TSAN tests pass with the GIL disabled in free-threaded builds #117657
Closed

Fidget-Spinner added 2 commits July 14, 2024 17:41

use atomics in new_reference
5bc855c

Add more atomics
da8415d

iritkatriel reviewed Jul 22, 2024
View reviewed changes

Comment threadInclude/cpython/pyatomic_msc.h Outdated

Update Include/cpython/pyatomic_msc.h
ef99cb9
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

Fidget-Spinner requested a review from colesbury July 22, 2024 17:37

colesbury reviewed Jul 22, 2024
View reviewed changes

Fidget-Spinner added 2 commits July 24, 2024 21:28

revert atomics
951a34d

Use argumentclinic
6e0e176

Fidget-Spinner changed the title ~~gh-117657: Fix races in tracebacks~~gh-117657: Fix some races in tracebacksJul 24, 2024

Fidget-Spinner closed this Jul 3, 2025

Navigation Menu

Uh oh!

Uh oh!

gh-117657: Fix some races in tracebacks - #121748

gh-117657: Fix some races in tracebacks#121748
Fidget-Spinner wants to merge 6 commits into
python:mainfrom
Fidget-Spinner:baseexception_tb

Fidget-Spinner commented Jul 14, 2024•
edited
Loading

Uh oh!

Uh oh!

iritkatriel commented Jul 22, 2024

Uh oh!

colesbury left a comment

Uh oh!

colesburyJul 22, 2024

Uh oh!

Uh oh!

python-cla-botBot commented Apr 18, 2025•
edited
Loading

Uh oh!

kumaraditya303 commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Uh oh!

Conversation

Fidget-Spinner commented Jul 14, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

iritkatriel commented Jul 22, 2024

Uh oh!

colesbury left a comment

Choose a reason for hiding this comment

Uh oh!

colesburyJul 22, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

python-cla-botBot commented Apr 18, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumaraditya303 commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Fidget-Spinner commented Jul 14, 2024•
edited
Loading

python-cla-botBot commented Apr 18, 2025•
edited
Loading