Uh oh!
There was an error while loading. Please reload this page.
gh-149816: Fix race between setattr and object.__dict__ update - #152296
Open
LindaSummer wants to merge 17 commits into
Open
gh-149816: Fix race between setattr and object.__dict__ update#152296LindaSummer wants to merge 17 commits into
setattr and object.__dict__ update#152296LindaSummer wants to merge 17 commits into
Conversation
24 tasks
LindaSummer
commented
Jun 26, 2026
ContributorAuthor
I also constructed an unstable sample during test this case. It slightly changed the #!/usr/bin/env python3"""Crash-biased instance setattr / __dict__ replacement race reproducer.Expected on fixed or GIL builds: the script runs to completion.Symptom on affected free-threaded debug/ASAN builds: intermittent crash,abort, or sanitizer report while setattr() inserts into a stale dict."""importargparseimportfaulthandlerimportosimportsysimportsysconfigimportthreadingimporttimeErrorLog=list[tuple[str, str]]
classTarget:
passPROOF_PREFIX="__stale_write_probe_"defgil_state() ->str:
is_gil_enabled=getattr(sys, "_is_gil_enabled", None)
ifis_gil_enabledisnotNone:
try:
return"enabled"ifis_gil_enabled() else"disabled"exceptException:
passifsysconfig.get_config_var("Py_GIL_DISABLED"):
return"unknown (free-threaded build)"return"unknown (regular build)"defpositive_int(text: str) ->int:
value=int(text, 0)
ifvalue<=0:
raiseargparse.ArgumentTypeError("must be positive")
returnvaluedefpositive_float(text: str) ->float:
value=float(text)
ifvalue<=0:
raiseargparse.ArgumentTypeError("must be positive")
returnvaluedefattr_writer(
obj: Target,
stop: threading.Event,
started: threading.Barrier,
errors: ErrorLog,
writer_yield_stride: int,
) ->None:
index=0ident=threading.get_ident()
started.wait()
whilenotstop.is_set():
try:
setattr(obj, f"{PROOF_PREFIX}{ident}_{index}", index)
exceptBaseExceptionasexc:
errors.append((threading.current_thread().name, repr(exc)))
stop.set()
returnindex+=1ifindex%writer_yield_stride==0:
time.sleep(0)
defdict_replacer(
obj: Target,
names: list[str],
stop: threading.Event,
started: threading.Barrier,
errors: ErrorLog,
dict_size: int,
churn: int,
replacer_yield_stride: int,
) ->None:
index=0started.wait()
whilenotstop.is_set():
try:
old_dict=obj.__dict__obj.__dict__= {
names[(index+offset) %len(names)]: (index, offset)
foroffsetinrange(dict_size)
}
delold_dict# Reuse freshly released dict/key memory quickly, so a stale# borrowed dict pointer is more likely to become a hard failure.junk= [
{names[(index+offset) %len(names)]: offset}
foroffsetinrange(churn)
]
deljunkexceptBaseExceptionasexc:
errors.append((threading.current_thread().name, repr(exc)))
stop.set()
returnindex+=1ifindex%replacer_yield_stride==0:
time.sleep(0)
defparse_args() ->argparse.Namespace:
cpu_count=os.cpu_count() or2parser=argparse.ArgumentParser(
description="Crash-biased setattr(obj, name, value) racing obj.__dict__ replacement."
)
parser.add_argument("--seconds", type=positive_float, default=60.0)
parser.add_argument("--writers", type=positive_int, default=max(4, cpu_count*2))
parser.add_argument("--replacers", type=positive_int, default=max(4, cpu_count*2))
parser.add_argument("--names", type=positive_int, default=4096)
parser.add_argument("--dict-size", type=positive_int, default=1)
parser.add_argument("--churn", type=positive_int, default=128)
parser.add_argument("--writer-yield-stride", type=positive_int, default=64)
parser.add_argument("--replacer-yield-stride", type=positive_int, default=4)
returnparser.parse_args()
defmain() ->int:
faulthandler.enable()
args=parse_args()
obj=Target()
names= [f"a{i}"foriinrange(args.names)]
obj.__dict__= {name: Nonefornameinnames[: args.dict_size]}
stop=threading.Event()
started=threading.Barrier(args.writers+args.replacers+1)
errors: ErrorLog= []
threads: list[threading.Thread] = []
print("finding: crash-biased instance setattr / __dict__ replacement race", flush=True)
print(
f"parameters: seconds={args.seconds:g}; GIL={gil_state()}; "f"writers={args.writers}; replacers={args.replacers}; "f"names={args.names}; dict_size={args.dict_size}; churn={args.churn}; "f"writer_yield_stride={args.writer_yield_stride}; "f"replacer_yield_stride={args.replacer_yield_stride}",
flush=True,
)
print("-"*72, flush=True)
forindexinrange(args.writers):
threads.append(
threading.Thread(
target=attr_writer,
name=f"writer-{index}",
args=(obj, stop, started, errors, args.writer_yield_stride),
)
)
forindexinrange(args.replacers):
threads.append(
threading.Thread(
target=dict_replacer,
name=f"replacer-{index}",
args=(
obj,
names,
stop,
started,
errors,
args.dict_size,
args.churn,
args.replacer_yield_stride,
),
)
)
forthreadinthreads:
thread.start()
started.wait()
deadline=time.monotonic() +args.secondstry:
whiletime.monotonic() <deadlineandnotstop.is_set():
time.sleep(0.05)
finally:
stop.set()
forthreadinthreads:
thread.join()
iferrors:
forname, errorinerrors:
print(f"{name} raised {error}", file=sys.stderr)
return1print("finished without a visible crash or sanitizer abort", flush=True)
return0if__name__=="__main__":
raiseSystemExit(main())Here is the crash report. no-gil-build/bin/python3 edward_borrow_dict.py
finding: 108 instance setattr / __dict__ replacement race
expected signal: stale writer key appears in a detached old __dict__, or native crash/abort/sanitizer report
parameters: seconds=30; GIL=disabled; writers=48; replacers=32; dict_size=1
------------------------------------------------------------------------
Fatal Python error: Segmentation fault
<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
File "/home/someuser/projects/cpython/cpython/edward_borrow_dict.py", line 71 in attr_writer
File "/home/someuser/projects/cpython/cpython/no-gil-build/lib/python3.16t/threading.py", line 1160 in run
File "/home/someuser/projects/cpython/cpython/no-gil-build/lib/python3.16t/threading.py", line 1218 in _bootstrap_inner
File "/home/someuser/projects/cpython/cpython/no-gil-build/lib/python3.16t/threading.py", line 1180 in _bootstrap
Current thread's C stack trace (most recent call first):
Binary file "no-gil-build/bin/python3", at ___interceptor_backtrace+0x46 [0x5639f2f67556]
Binary file "no-gil-build/bin/python3", at _Py_DumpStack+0x138 [0x5639f3727568]
Binary file "no-gil-build/bin/python3", at +0xaab7df [0x5639f37777df]
Binary file "/usr/lib64/libc.so.6", at +0x3d310 [0x7f934f127310]
Binary file "no-gil-build/bin/python3", at +0x6876fa [0x5639f33536fa]
Binary file "no-gil-build/bin/python3", at +0x60d0f5 [0x5639f32d90f5]
Binary file "no-gil-build/bin/python3", at +0x6174c7 [0x5639f32e34c7]
Binary file "no-gil-build/bin/python3", at +0x616ca3 [0x5639f32e2ca3]
Binary file "no-gil-build/bin/python3", at +0x5fa28a [0x5639f32c628a]
Binary file "no-gil-build/bin/python3", at +0x60ec81 [0x5639f32dac81]
Binary file "no-gil-build/bin/python3", at _PyObject_GenericSetAttrWithDict+0x58a [0x5639f332604a]
Binary file "no-gil-build/bin/python3", at PyObject_SetAttr+0x2b1 [0x5639f33205d1]
Binary file "no-gil-build/bin/python3", at +0x87656d [0x5639f354256d]
Binary file "no-gil-build/bin/python3", at _PyEval_EvalFrameDefault+0x1f6cb [0x5639f356df6b]
Binary file "no-gil-build/bin/python3", at +0x87e334 [0x5639f354a334]
Binary file "no-gil-build/bin/python3", at +0x52170d [0x5639f31ed70d]
Binary file "no-gil-build/bin/python3", at +0x9161a3 [0x5639f35e21a3]
Binary file "no-gil-build/bin/python3", at PyObject_Vectorcall+0xf4 [0x5639f31e99b4]
Binary file "no-gil-build/bin/python3", at _Py_VectorCallInstrumentation_StackRefSteal+0x3eb [0x5639f354b59b]
Binary file "no-gil-build/bin/python3", at _PyEval_EvalFrameDefault+0x155fd [0x5639f3563e9d]
Binary file "no-gil-build/bin/python3", at +0x87e334 [0x5639f354a334]
Binary file "no-gil-build/bin/python3", at +0x52170d [0x5639f31ed70d]
Binary file "no-gil-build/bin/python3", at +0xbc218f [0x5639f388e18f]
Binary file "no-gil-build/bin/python3", at +0xa548d1 [0x5639f37208d1]
Binary file "no-gil-build/bin/python3", at +0x3ae605 [0x5639f307a605]
Binary file "/usr/lib64/libc.so.6", at +0x91873 [0x7f934f17b873]
Binary file "/usr/lib64/libc.so.6", at +0x11098c [0x7f934f1fa98c]
AddressSanitizer:DEADLYSIGNAL
=================================================================
==3383997==ERROR: AddressSanitizer: SEGV on unknown address 0x03e80033a2bd (pc 0x7f934f17d54c bp 0x7b934cb8a770 sp 0x7b934cb8a690 T42)
==3383997==The signal is caused by a READ memory access.
#0 0x7f934f17d54c (/usr/lib64/libc.so.6+0x9354c)
#1 0x7f934f1271e5 in raise (/usr/lib64/libc.so.6+0x3d1e5)
#2 0x5639f3777842 in faulthandler_fatal_error /home/someuser/projects/cpython/cpython/./Modules/faulthandler.c:434:5
#3 0x7f934f12730f (/usr/lib64/libc.so.6+0x3d30f)
#4 0x5639f33536f9 in mi_block_nextx /home/someuser/projects/cpython/cpython/./Include/internal/mimalloc/mimalloc/internal.h:640:23
#5 0x5639f33536f9 in mi_block_next /home/someuser/projects/cpython/cpython/./Include/internal/mimalloc/mimalloc/internal.h:669:10
#6 0x5639f33536f9 in _mi_page_malloc /home/someuser/projects/cpython/cpython/Objects/mimalloc/alloc.c:49:16
#7 0x5639f33536f9 in mi_heap_malloc_small_zero /home/someuser/projects/cpython/cpython/Objects/mimalloc/alloc.c:127:19
#8 0x5639f33536f9 in _mi_heap_malloc_zero_ex /home/someuser/projects/cpython/cpython/Objects/mimalloc/alloc.c:156:12
#9 0x5639f33536f9 in _mi_heap_malloc_zero /home/someuser/projects/cpython/cpython/Objects/mimalloc/alloc.c:179:10
#10 0x5639f33536f9 in mi_heap_malloc /home/someuser/projects/cpython/cpython/Objects/mimalloc/alloc.c:183:10
#11 0x5639f33536f9 in _PyMem_MiMalloc /home/someuser/projects/cpython/cpython/Objects/obmalloc.c:265:12
#12 0x5639f32d90f4 in new_keys_object /home/someuser/projects/cpython/cpython/Objects/dictobject.c:817:14
#13 0x5639f32e34c6 in dictresize /home/someuser/projects/cpython/cpython/Objects/dictobject.c:2147:15
#14 0x5639f32e2ca2 in insertion_resize /home/someuser/projects/cpython/cpython/Objects/dictobject.c:1838:12
#15 0x5639f32e2ca2 in insert_combined_dict /home/someuser/projects/cpython/cpython/Objects/dictobject.c:1855:13
#16 0x5639f32c6289 in insertdict /home/someuser/projects/cpython/cpython/Objects/dictobject.c:1978:13
#17 0x5639f32dac80 in store_instance_attr_dict /home/someuser/projects/cpython/cpython/Objects/dictobject.c:7368:15
#18 0x5639f32dac80 in _PyObject_StoreInstanceAttribute /home/someuser/projects/cpython/cpython/Objects/dictobject.c:7389:16
#19 0x5639f3326049 in _PyObject_GenericSetAttrWithDict /home/someuser/projects/cpython/cpython/Objects/object.c:2058:19
#20 0x5639f33205d0 in PyObject_SetAttr /home/someuser/projects/cpython/cpython/Objects/object.c:1533:15
#21 0x5639f354256c in builtin_setattr_impl /home/someuser/projects/cpython/cpython/Python/bltinmodule.c:1814:9
#22 0x5639f354256c in builtin_setattr /home/someuser/projects/cpython/cpython/Python/clinic/bltinmodule.c.h:789:20
#23 0x5639f356df6a in _Py_BuiltinCallFast_StackRef /home/someuser/projects/cpython/cpython/Python/ceval.c:815:11
#24 0x5639f356df6a in _PyEval_EvalFrameDefault /home/someuser/projects/cpython/cpython/Python/generated_cases.c.h:2420:35
#25 0x5639f354a333 in _PyEval_EvalFrame /home/someuser/projects/cpython/cpython/./Include/internal/pycore_ceval.h:122:16
#26 0x5639f354a333 in _PyEval_Vector /home/someuser/projects/cpython/cpython/Python/ceval.c:2134:12
#27 0x5639f31ed70c in _PyObject_VectorcallTstate /home/someuser/projects/cpython/cpython/./Include/internal/pycore_call.h:144:11
#28 0x5639f31ed70c in _PyObject_VectorcallPrepend /home/someuser/projects/cpython/cpython/Objects/call.c:855:20
#29 0x5639f35e21a2 in _PyObject_VectorcallTstate /home/someuser/projects/cpython/cpython/./Include/internal/pycore_call.h:144:11
#30 0x5639f35e21a2 in context_run /home/someuser/projects/cpython/cpython/Python/context.c:728:29
#31 0x5639f31e99b3 in _PyObject_VectorcallTstate /home/someuser/projects/cpython/cpython/./Include/internal/pycore_call.h:144:11
#32 0x5639f31e99b3 in PyObject_Vectorcall /home/someuser/projects/cpython/cpython/Objects/call.c:327:12
#33 0x5639f354b59a in _Py_VectorCallInstrumentation_StackRefSteal /home/someuser/projects/cpython/cpython/Python/ceval.c:766:11
#34 0x5639f3563e9c in _PyEval_EvalFrameDefault /home/someuser/projects/cpython/cpython/Python/generated_cases.c.h:1846:35
#35 0x5639f354a333 in _PyEval_EvalFrame /home/someuser/projects/cpython/cpython/./Include/internal/pycore_ceval.h:122:16
#36 0x5639f354a333 in _PyEval_Vector /home/someuser/projects/cpython/cpython/Python/ceval.c:2134:12
#37 0x5639f31ed70c in _PyObject_VectorcallTstate /home/someuser/projects/cpython/cpython/./Include/internal/pycore_call.h:144:11
#38 0x5639f31ed70c in _PyObject_VectorcallPrepend /home/someuser/projects/cpython/cpython/Objects/call.c:855:20
#39 0x5639f388e18e in thread_run /home/someuser/projects/cpython/cpython/./Modules/_threadmodule.c:388:21
#40 0x5639f37208d0 in pythread_wrapper /home/someuser/projects/cpython/cpython/Python/thread_pthread.h:234:5
#41 0x5639f307a604 in asan_thread_start(void*) asan_interceptors.cpp.o
#42 0x7f934f17b872 (/usr/lib64/libc.so.6+0x91872)
#43 0x7f934f1fa98b (/usr/lib64/libc.so.6+0x11098b)
==3383997==Register values:
rax = 0x0000000000000000 rbx = 0x000000000033a2e7 rcx = 0x00007f934f17d54c rdx = 0x000000000000000b rdi = 0x000000000033a2bd rsi = 0x000000000033a2e7 rbp = 0x00007b934cb8a770 rsp = 0x00007b934cb8a690 r8 = 0x000000000000005e r9 = 0x00007b930e239a00 r10 = 0x6e8687225753e38c r11 = 0x0000000000000246 r12 = 0x00000f724b2ca3b0 r13 = 0x000000000000000b r14 = 0x000000000000000b r15 = 0x00007b9259651d80 AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib64/libc.so.6+0x9354c) Thread T42 created by T0 here:
#0 0x5639f305bb01 in pthread_create (/home/someuser/projects/cpython/cpython/no-gil-build/bin/python3.16+0x38fb01)
#1 0x5639f371f004 in do_start_joinable_thread /home/someuser/projects/cpython/cpython/Python/thread_pthread.h:281:14
#2 0x5639f371ebf2 in PyThread_start_joinable_thread /home/someuser/projects/cpython/cpython/Python/thread_pthread.h:323:9
#3 0x5639f388dc64 in ThreadHandle_start /home/someuser/projects/cpython/cpython/./Modules/_threadmodule.c:475:9
#4 0x5639f388d316 in do_start_new_thread /home/someuser/projects/cpython/cpython/./Modules/_threadmodule.c:1919:9
#5 0x5639f388b9f8 in thread_PyThread_start_joinable_thread /home/someuser/projects/cpython/cpython/./Modules/_threadmodule.c:2042:14
#6 0x5639f330ddbf in cfunction_call /home/someuser/projects/cpython/cpython/Objects/methodobject.c:564:18
#7 0x5639f31e8429 in _PyObject_MakeTpCall /home/someuser/projects/cpython/cpython/Objects/call.c:242:18
#8 0x5639f354aa88 in _Py_VectorCall_StackRefSteal /home/someuser/projects/cpython/cpython/Python/ceval.c:724:11
#9 0x5639f3561cbd in _PyEval_EvalFrameDefault /home/someuser/projects/cpython/cpython/Python/generated_cases.c.h:3528:35
#10 0x5639f3549aa7 in _PyEval_EvalFrame /home/someuser/projects/cpython/cpython/./Include/internal/pycore_ceval.h:122:16
#11 0x5639f3549aa7 in _PyEval_Vector /home/someuser/projects/cpython/cpython/Python/ceval.c:2134:12
#12 0x5639f3549aa7 in PyEval_EvalCode /home/someuser/projects/cpython/cpython/Python/ceval.c:677:21
#13 0x5639f36de954 in run_mod /home/someuser/projects/cpython/cpython/Python/pythonrun.c:1472:19
#14 0x5639f36d7d76 in pyrun_file /home/someuser/projects/cpython/cpython/Python/pythonrun.c:1296:15
#15 0x5639f36d7d76 in _PyRun_SimpleFileObject /home/someuser/projects/cpython/cpython/Python/pythonrun.c:518:13
#16 0x5639f36d7219 in _PyRun_AnyFileObject /home/someuser/projects/cpython/cpython/Python/pythonrun.c:81:15
#17 0x5639f3750fda in pymain_run_file_obj /home/someuser/projects/cpython/cpython/Modules/main.c:411:15
#18 0x5639f3750fda in pymain_run_file /home/someuser/projects/cpython/cpython/Modules/main.c:430:15
#19 0x5639f374f217 in pymain_run_python /home/someuser/projects/cpython/cpython/Modules/main.c:715:21
#20 0x5639f374f217 in Py_RunMain /home/someuser/projects/cpython/cpython/Modules/main.c:796:5
#21 0x5639f3750239 in pymain_main /home/someuser/projects/cpython/cpython/Modules/main.c:826:12
#22 0x5639f37503e2 in Py_BytesMain /home/someuser/projects/cpython/cpython/Modules/main.c:850:12
#23 0x7f934f11116d (/usr/lib64/libc.so.6+0x2716d)
==3383997==ABORTING |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:aa631bcbfc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+7515
to
+7517
| while (!(valid = FT_ATOMIC_LOAD_UINT8(values->valid))) { | ||
| // Retry if the managed dict changes before we can lock and validate it. | ||
| try_res = try_store_instance_attr_invalid_inline(obj, name, value); |
There was a problem hiding this comment.
When setattr reads values->valid as true, this new retry path is skipped entirely, but the object can still have a materialized __dict__ pointing at the inline values. If another thread replaces obj.__dict__ after that load and before the later store_instance_attr_dict() lock, _PyObject_SetManagedDict() detaches the old dict and the setter can then write into that detached dict; this is the same stale-dict mutation the patch is trying to prevent, just starting from the valid-inline branch.
Useful? React with 👍 / 👎.
LindaSummer
commented
Jun 29, 2026
ContributorAuthor
Hi @markshannon and @methane , Could you take a look at this PR when you have a chance? I found this issue in the I'd really appreciate your thoughts. Thanks, and have a great day! |
LindaSummer
commented
Jul 5, 2026
ContributorAuthor
Hi @kumaraditya303 , Sorry to bother you. Wish you a good day! |
| typedef struct { | ||
| try_store_instance_attr_status try_status; | ||
| int res; | ||
| } try_store_instance_attr_result_t; |
Contributor
There was a problem hiding this comment.
It seems too complicated to add a new type just to return for a function which is supposed to be local, can you simplify it to avoid this?
ContributorAuthor
There was a problem hiding this comment.
Hi @kumaraditya303 ,
Thanks very much for your suggestion!
This struct has been removed, and a pointer is in parameter as the res for simplicity.
ContributorAuthor
Hi @kumaraditya303 , Thanks very for your review! Wish you a good day! |
LindaSummer
commented
Jul 14, 2026
ContributorAuthor
Hi @kumaraditya303 , Sorry to ping you again. Thanks and wish you a good day! |
Contributor
@mpage Can you take a look at this? |
LindaSummer
commented
Jul 31, 2026
ContributorAuthor
Hi @mpage , I resolved the merge conflicts yesterday, so this PR is up to date now. |
mpage
requested changes
Aug 3, 2026
mpage
left a comment
Contributor
There was a problem hiding this comment.
Sorry for the late response here (I was on vacation) and thanks for taking the time to dig into this! After reading through the original report, I think we can probably come up with a simpler fix by ensuring that we always use a strong reference to the managed dict after releasing the per-object lock.
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
#149816 sub-case 108
Root Cause
This race is caused between
_PyObject_SetDictand_PyObject_StoreInstanceAttribute.Inside the
!FT_ATOMIC_LOAD_UINT8(values->valid)branch theobjanddictare not locked in a consistent lock.So it is possible that we get a stale dict and if it is not a strong ref, the dict could be inconsistent even an invalid one.
cpython/Objects/dictobject.c
Lines 7558 to 7570 in 3ffda34
Proposed Changes
Py_BEGIN_CRITICAL_SECTION2(obj, dict)to make theobjanddictconsistent