Abort/Segfault: corrupted object freelist in clear_freelist

[sergey-miryanov]

Bug report

Bug description:

Under OOM the free-threaded build's per-PyThreadState freelists desync size from the chain: PyList_New's Py_DECREF(op) failure path re-pushes a list with a stale dangling ob_item, tripping the clear_freelist size assertion (or a free_list_items use-after-free SEGV on the next pop).

AI Disclaimer: this gist was drafted by Claude Code, which also generated the reduced reproducer.

Crash report

Under sustained allocation failure the free-threaded build's per-PyThreadState object freelists become internally inconsistent: the size counter and the actual freelist chain disagree. This corruption surfaces two ways from the same root: (1) the GC's freelist sweep aborts on assert(freelist->size == 0 || freelist->size == -1) in clear_freelist (Objects/object.c:909) — the recorded crash for all three vehicles — and (2) a subsequent PyList_New pops a bogus "list" off the corrupted lists freelist whose stale ob_item points at already-freed (0xcd...-filled) array memory; when the immediately-following list_allocate_array() fails under OOM, the cleanup Py_DECREF(op) calls free_list_items() on that dangling pointer -> use-after-free SEGV in _PyMem_DebugFree. The assertion is debug-only (NDEBUG drops it), but the underlying freelist corruption is build-agnostic and crashes release, JIT and upstream builds too.

Reproducer

import gc
from _testcapi import set_nomemory, remove_mem_hooks

def go():
 objs = []
 for _ in range(120):
 a = (1.0, 2.0, 3.0) # tuple + float freelists
 b = [a[0], a[1], a[2]] # list freelist (PyList_New)
 objs.append(b[0:2]) # slice -> another PyList_New
 gc.collect() # GC sweeps freelists -> clear_freelist
 return objs

for start in range(1, 900): # crashes at start=4
 set_nomemory(start, 0) # fail every allocation from #start onward
 try:
 try:
 go()
 finally:
 remove_mem_hooks()
 except BaseException:
 pass

Deterministic at start=4 on the free-threaded debug+ASan build, and also crashes (SEGV) on free-threaded release, JIT, and upstream. A single set_nomemory(0,0) does not suffice — the budget has to let the early freelist pushes/pops succeed and then fail mid-sequence, hence the sweep.

The original fuzzing vehicles instead drained/realloc'd enough lists that the corruption surfaced one step earlier, inside the GC's clear_freelist, producing the Objects/object.c:909 assertion abort (see the recorded faulthandler stack in backtrace.txt). Same defect, different surface.

Backtrace

Deterministic SEGV surface from the reduced reproducer (gdb, 3.16_ft_debug_asan, commit 15d7406) — byte-identical to the captured vehicle backtrace:

#0 _PyMem_DebugCheckAddress Objects/obmalloc.c:3344 (p = 0xcdcdcdcdcdcdcdc5 <- freed-fill)
#1 _PyMem_DebugRawFree Objects/obmalloc.c:3166
#2 _PyMem_DebugFree Objects/obmalloc.c:3316
#3 free_list_items Objects/listobject.c:68 (PyMem_Free(dangling ob_item))
#4 list_dealloc Objects/listobject.c:569
#5 _Py_Dealloc Objects/object.c:3319
#6 Py_DECREF Include/refcount.h:359
#7 PyList_New Objects/listobject.c:262 (Py_DECREF(op) after list_allocate_array fail)
#8 _PyList_FromStackRefStealOnSuccess Objects/listobject.c:3298

Recorded abort surface (faulthandler, original tarfile vehicle):

python: Objects/object.c:909: void clear_freelist(struct _Py_freelist *, int, freefunc):
 Assertion `freelist->size == 0 || freelist->size == -1' failed.
Fatal Python error: Aborted
 ... _Py_HandlePending -> GC -> _PyObject_ClearFreeLists -> clear_freelist

Root cause

Objects/object.c, clear_freelist (L901):

static void
clear_freelist(struct _Py_freelist *freelist, int is_finalization, freefunc dofree)
{
 void *ptr;
 while ((ptr = _PyFreeList_PopNoStats(freelist)) != NULL) { // drains chain
 dofree(ptr);
 }
 assert(freelist->size == 0 || freelist->size == -1); // L909: size desynced -> abort
 assert(freelist->freelist == NULL);
 ...
}

_PyFreeList_Push/_PyFreeList_PopNoStats (Include/internal/pycore_freelist.h) keep fl->size in lockstep with the singly-linked fl->freelist chain (the "next" pointer is stored in the first word of each freed object). The clear_freelist loop pops until the chain is NULL and then asserts size == 0. The abort means size no longer matches the chain length: the chain terminated (or was diverted) before size reached zero.

Two ingredients, both triggered by the OOM injection in the free-threaded build:

1. Freelist objects are not "next"-cleared on the failure path. In PyList_New (Objects/listobject.c:248) the list object is popped from the freelist before the size-dependent list_allocate_array() runs. When that allocation fails under OOM, the cleanup is Py_DECREF(op) (L262), so op goes straight back through list_dealloc -> _Py_FREELIST_FREE(lists, op, ...) (L573) and back onto the same freelist. The repeated pop/push-on-failure churn under a failing allocator is where size and the chain drift apart, leaving a node whose stale ob_item (L266 was never reached) still points at a freed array.

2. A later consumer trusts the desynced freelist. The very next PyList_New pops that bad node; with op->ob_item dangling and list_allocate_array again failing, Py_DECREF(op) reaches free_list_items(op->ob_item, ...) on freed (0xcd) memory -> use-after-free. If a GC sweep runs first instead, clear_freelist trips its size/chain assertion.

(The assertion is compiled out under NDEBUG, which is why release/JIT/upstream don't abort there — but they still crash on the same corruption, as a SEGV.)

Suggested fix

The robust fix is to make freelist accounting self-healing and to stop trusting size independently of the chain. Concretely:

• In clear_freelist, drive the loop and the counter off the chain and reset size unconditionally instead of asserting it, so a desync can't corrupt later allocations:

 void *ptr;
 while ((ptr = _PyFreeList_PopNoStats(freelist)) != NULL) {
 dofree(ptr);
 }
 assert(freelist->freelist == NULL);
 freelist->size = is_finalization ? -1 : 0; /* reset, don't assert size */

• More importantly, fix the producer: in PyList_New (listobject.c:248), reset op->ob_item = NULL (and Py_SET_SIZE(op, 0); op->allocated = 0;) on the freelist-pop path before the fallible list_allocate_array(), so the OOM Py_DECREF(op) cleanup never sees a stale ob_item:

 PyListObject *op = _Py_FREELIST_POP(PyListObject, lists);
 if (op == NULL) {
 op = PyObject_GC_New(PyListObject, &PyList_Type);
 if (op == NULL) {
 return NULL;
 }
 }
 op->ob_item = NULL; /* defensive: pop'd object may carry stale state */
 Py_SET_SIZE(op, 0);
 op->allocated = 0;

The deeper invariant to audit is every _Py_FREELIST_POP / _Py_FREELIST_FREE pair on a fallible allocation path: an object must be in a clean, decref-safe state before any allocation that can fail under OOM, since the failure cleanup pushes it back onto the freelist.

Notes

Found by OOM-injection fuzzing (_testcapi.set_nomemory). Free-threading-specific in its abort surface (the clear_freelist assert and the per-PyThreadState freelist machinery are Py_GIL_DISABLED-only), but the reduced reproducer crashes all four local builds (debug+ASan abort/SEGV, release SEGV, JIT SEGV, upstream SEGV), so the underlying freelist corruption is not debug-only. Three vehicles in python-4/ recorded the identical Objects/object.c:909 assertion; re-running any of them now non-deterministically lands on the sibling PyList_New/free_list_items use-after-free SEGV instead — confirming a single shared root cause. Distinct from the _co_unique_id (codeobject.c:2440) assertion that the same vehicles can also hit; that is a separate FT unique-id-pool invariant, not this freelist one.

Versions

• main (3.16.0a0, commit 15d7406). Reproduced (reduced repro repro.py) on free-threaded debug+ASan (SEGV; abort on the vehicle's clear_freelist surface), free-threaded release (SEGV), JIT (SEGV), and upstream (SEGV).

---

Part of python/cpython#151763 — an umbrella tracking 35 OOM-related crash findings.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

[Abhi210]

@devdanzin Just a question. There are two fixes; in my PR, I fixed only the second part, which was in PyList_New. The reason being, I do not see why we need to use the second fix:

 void *ptr;
 while ((ptr = _PyFreeList_PopNoStats(freelist)) != NULL) {
 dofree(ptr);
 }
 assert(freelist->freelist == NULL);
 freelist->size = is_finalization ? -1 : 0; /* reset, don't assert size */

Just using the fix in PyList_New prevented any crashes.

From what I understood by looking at the code, PyList_New leads to an uninitialized ob_item being freed, which, under some heap layouts, corrupts nearby allocator/freelist state rather than segfaulting immediately; that corruption is what the assertion later catches. Indeed in the following code:

PyObject *
PyList_New(Py_ssize_t size)
{
 if (size < 0) {
 PyErr_BadInternalCall();
 return NULL;
 }

 PyListObject *op = _Py_FREELIST_POP(PyListObject, lists);
 if (op == NULL) {
 op = PyObject_GC_New(PyListObject, &PyList_Type);
 if (op == NULL) {
 return NULL;
 }
 }
 if (size <= 0) {
 op->ob_item = NULL;
 }
 else {
#ifdef Py_GIL_DISABLED
 _PyListArray *array = list_allocate_array(size);
 if (array == NULL) {
 Py_DECREF(op);
 return PyErr_NoMemory();
 }
 memset(&array->ob_item, 0, size * sizeof(PyObject *));
 op->ob_item = array->ob_item;

We see that the main cause of the crash is op = PyObject_GC_New(PyListObject, &PyList_Type); followed by an inability to allocate memory in _PyListArray *array = list_allocate_array(size);. If we fix this in Objects/listobject, clear_freelist would not catch any error. So, in my view, we do not need the first fix. Fixing only the PyList_New should be sufficient to resolve this issue. Or did I miss something here?

[devdanzin]

You're right. The clear_freelist assertion is a downstream symptom that won't be triggered with your fix, so PyList_New-only fix is sufficient.

[devdanzin]

New investigation found something significant.

Cross-linking a non-OOM way to reach the same clear_freelist assertion, since the discussion above turned on whether it's exclusively a downstream symptom of the PyList_New OOM bug.

To be clear up front: the PyList_New fix (#154019) is the right fix for the OOM root here, and I'm not arguing to reopen that. The one thing I'd refine is "with the PyList_New fix, clear_freelist would not catch any error" — that holds for the OOM root, but the Objects/object.c:909 assertion (freelist->size == 0 || freelist->size == -1) is a generic freelist-corruption detector, and it's independently reachable with no allocation failure at all, via #154130: sharing a dict iterator across threads makes dictiter_iternext_threadsafe's exhaustion (fail:) path double-DECREF di_dict. That double-frees the dict, desyncs the dict freelist's size, and the sweep aborts on this exact assertion. The PyList_New change doesn't touch that path.

Minimal non-OOM reproducer (free-threaded build, PYTHON_GIL=0) — no set_nomemory, no third-party module. Aborts on the assertion ~100% within a few seconds on a debug FT build:

import threading

WORKERS = 8
INNER = 200
ROUNDS = 40

def mk():
 return iter({}) # dict_keyiterator over an empty dict: StopIteration at once

def run_round():
 cell = [mk()]
 bar = threading.Barrier(WORKERS)
 def worker():
 bar.wait()
 for _ in range(INNER):
 it = cell[0]
 try:
 next(it)
 except StopIteration:
 cell[0] = mk() # drop the exhausted iterator while siblings still hold it
 ts = [threading.Thread(target=worker) for _ in range(WORKERS)]
 for t in ts:
 t.start()
 for t in ts:
 t.join()

for _ in range(ROUNDS):
 run_round()
print("survived") # rarely reached

Faces observed, all at the next() line (same underlying double-free):

• debug free-threaded (--with-pydebug): Objects/object.c:909 clear_freelist ... Assertion 'freelist->size == 0 || freelist->size == -1' failed (~90% of runs), otherwise _Py_NegativeRefcount in Objects/dictobject.c (the iterator itself driven to a negative refcount).
• release free-threaded (assertion compiled out): still crashes — Fatal Python error: PyMutex_Unlock: unlocking mutex that is not locked (the dict's per-object critical-section mutex, left inconsistent by the race) or a hard SIGSEGV. So the corruption is build-agnostic, not a debug-only artifact.
• ThreadSanitizer: data races in dictiter_iternextkey / dictiter_iternext_threadsafe (Objects/dictobject.c) and the free-threading shared-refcount merge _Py_ExplicitMergeRefcount (Objects/object.c).

Net: once both #154019 (this issue's OOM root) and #154130 (the dict-iterator double-free) land, the assertion should stop firing for both known roots. I mainly wanted the #152125 ↔ #154130 link on record, so a clear_freelist abort isn't automatically read as the PyList_New OOM bug — today it's also a live detector for the #154130 race. Whether that additionally justifies the belt-and-braces freelist->size reset is your call.

AI note: the reproducer was reduced from a fusil fuzzing vehicle and this comment drafted with Claude Code.