Skip to content

Fix debug-mode segfault when the asyncgen finalizer hook runs during frame teardown - #756

Open
david-runlayer wants to merge 1 commit into
MagicStack:masterfrom
david-runlayer:fix-asyncgen-finalizer-upstream
Open

Fix debug-mode segfault when the asyncgen finalizer hook runs during frame teardown#756
david-runlayer wants to merge 1 commit into
MagicStack:masterfrom
david-runlayer:fix-asyncgen-finalizer-upstream

Conversation

@david-runlayer

@david-runlayerdavid-runlayer commented Jul 20, 2026

Copy link
Copy Markdown

Summary

In debug mode, Loop._asyncgen_finalizer_hook can segfault the interpreter. The hook is invoked by the GC from a dealloc — which can happen while CPython is mid-way through clearing the current interpreter frame — and the call_soon_threadsafe it performs creates a Handle whose debug source-traceback capture calls sys._getframe(), dereferencing the half-cleared frame.

We hit this in production on CPython 3.13.14 with uvloop 0.22.1: enabling loop.set_debug(True) on a Starlette/FastAPI service with streaming request bodies (request.stream() is an async generator; under disconnect/cancellation churn they are GC'd constantly) produced worker SIGSEGVs within seconds. Reproduces on 3.13.14 and 3.14.4, x86_64 and aarch64. It appears to be the same underlying weakness as #715 (there, on 3.14, the capture path surfaces a non-frame object to traceback.walk_stack).

Symbolized backtrace (uvloop built -O0 -g3, CPython 3.13.14)

#3 ?? libpython3.13 (sys._getframe impl; SEGV reading frame->frame_obj)
#4 ?? libpython3.13
#5 __pyx_f_6uvloop_4loop_extract_stack () at cbhandles.pyx (extract_stack)
#6 __pyx_f_6uvloop_4loop_6Handle__set_loop () at cbhandles.pyx (Handle._set_loop)
#7 __pyx_f_6uvloop_4loop_new_Handle ()
#8 __pyx_pf_6uvloop_4loop_4Loop_14call_soon_threadsafe ()
#11 __pyx_pf_6uvloop_4loop_4Loop_140_asyncgen_finalizer_hook ()
#14 PyObject_CallOneArg
#16 PyObject_CallFinalizerFromDealloc <- asyncgen GC'd from a dealloc...
#19 _PyEval_FrameClearAndPop <- ...while this frame is being cleared
#20 _PyEval_EvalFrameDefault

A production core dump shows the same failure from the release wheel: SIGSEGV (SEGV_MAPERR) inside sys._getframe incref'ing frame->frame_obj containing a torn value (0x3000000010).

Fix

Reproducer

Crashes in <15 s unpatched (verified end-to-end from a clean directory containing only these files: 271 Fatal Python error lines in a 60 s window, current PyPI starlette/fastapi/uvicorn); clean with this fix (10 min, ~259k streamed requests, verified on 3.13.14 aarch64 and x86_64).

app.py + load.py + docker command
# app.py — needs: pip install uvicorn starlette fastapi httpx uvloopfromfastapiimportFastAPI, Requestfromstarlette.middleware.baseimportBaseHTTPMiddlewareapp=FastAPI()
classPassthrough(BaseHTTPMiddleware):
asyncdefdispatch(self, request, call_next):
returnawaitcall_next(request)
app.add_middleware(Passthrough)
app.add_middleware(Passthrough)
@app.post("/upload")asyncdefupload(request: Request) ->dict:
total=0asyncforchunkinrequest.stream():
total+=len(chunk)
return {"received": total}
# loop_factory.pyimportuvloopdefnew_event_loop():
loop=uvloop.new_event_loop()
loop.set_debug(True) # <- the triggerreturnloop
# load.py — 32 concurrent chunked uploads in a loopimportasyncio, httpxasyncdefbody():
for_inrange(200):
yieldb"x"*2048asyncdefworker(client):
whileTrue:
try:
awaitclient.post("http://localhost:8000/upload", content=body())
exceptException:
awaitasyncio.sleep(0.05)
asyncdefmain():
asyncwithhttpx.AsyncClient(timeout=30) asc:
awaitasyncio.gather(*[worker(c) for_inrange(32)])
asyncio.run(main())
docker run --rm -v $PWD:/r python:3.13.14-slim sh -c ' pip install -q uvloop==0.22.1 uvicorn starlette fastapi httpx httptools cd /r PYTHONFAULTHANDLER=1 uvicorn app:app --workers 4 --port 8000 \ --loop loop_factory:new_event_loop --http httptools & sleep 8 python load.py & sleep 60 # -> "Fatal Python error: Segmentation fault" within ~15s of load starting'

Performance

Stock = master merge base (e8efea4), patched = this branch (including the save/restore hook); both built from the same tree with identical flags (aarch64, CPython 3.13.14, best-of-3):

pathstockpatcheddelta
call_soon + cancel, non-debug0.125 µs0.129 µsnoise (≤4 ns, sign flips between sessions)
asyncgen create→finalize, non-debug1.491 µs1.631 µs+140 ns (three thread-local ops in the hook)
call_soon + cancel, debug3.337 µs3.376 µs~1% (within noise)
asyncgen create→finalize, debug11.836 µs7.368 µs−38% (skips the now-pointless capture)

Non-debug hot paths are untouched by construction (extract_stack is only reachable under loop._debug); the only non-debug cost is the thread-local save/set/restore per GC'd async generator, on a path whose real cost is dominated by uv_async_send plus the subsequently scheduled create_task/aclose().

Behavior change

In debug mode, the handle scheduling agen.aclose() from the finalizer hook no longer carries _source_traceback. Previously it captured the stack of whichever code happened to be running when GC fired — misleading rather than useful. Task-level debug tracebacks (create_task runs later in normal loop context) and firstiter capture are unaffected.

Validated with the reproducer plus targeted checks (normal handles still get _source_traceback in debug mode; asyncgen finalizer path executes cleanly); relying on CI for the full test matrix.

@david-runlayer

Copy link
Copy Markdown
Author

CI note: the test (3.14t, macos-latest) failure is the pre-existing flake family, not this change — Test_AIO_Process.test_process_streams_redirect (stdlib-asyncio variant; this patch isn't in that code path) failed because a debug-mode Executing <Task ...> took N seconds slow-callback warning leaked into the redirected stderr on the slow free-threaded runner. The same warning-pollution appears in master's latest Tests run (29349960216, 3.14t job, test_process_delayed_stdio*), which failed before this PR existed. Re-pushed to retrigger.

@david-runlayer
david-runlayerforce-pushed the fix-asyncgen-finalizer-upstream branch 4 times, most recently from 474e3dd to 3f175adCompareJuly 20, 2026 22:47
…frame teardown
The asyncgen finalizer hook can be invoked by a dealloc while CPython is
mid-way through clearing the current interpreter frame
(_PyEval_FrameClearAndPop -> PyObject_CallFinalizerFromDealloc ->
_asyncgen_finalizer_hook). In debug mode the hook's call_soon_threadsafe
creates a Handle whose source-traceback capture calls sys._getframe(),
which dereferences the half-cleared frame (garbage frame_obj) and
segfaults the interpreter.
Observed in production on CPython 3.13.14 under streaming HTTP load
(starlette request.stream() async generators being GC'd under
cancellation churn): worker SIGSEGV within seconds of enabling
loop.set_debug(True). Reproduces on 3.13 and 3.14, x86_64 and aarch64.
Fix: skip source-traceback capture (per-thread flag) for handles created
from the finalizer hook -- the traceback of a GC point is meaningless for
debugging anyway -- and make extract_stack() tolerate AttributeError/
ValueError/TypeError from walk_stack, which py3.14 can raise when
non-frame objects (e.g. _asyncio.TaskStepMethWrapper) appear in the
stack (MagicStack#715).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@david-runlayer