[3.14] GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. (GH-141711) by encukou · Pull Request #141944 · python/cpython

encukou · 2025-11-25T13:26:06Z

Only raises if the stack pointer is both below the limit and above the stack base. This prevents false positives for user-space threads, as the stack pointer will be outside those bounds if the stack has been swapped.

Cherry-picked from commit c25a070

I'm also including the whole page of internal documentation, rather than ignore the conflict there.

Issue: Python 3.14 stack overflow detection is incompatible with C++ Boost make_fcontext() coroutines #139653

… the stack pointer is about to overflow the stack. (pythonGH-141711) Only raises if the stack pointer is both below the limit *and* above the stack base. This prevents false positives for user-space threads, as the stack pointer will be outside those bounds if the stack has been swapped. Cherry-picked from commit c25a070

vstinner

LGTM. Good idea to backport the whole internal documentation.

hroncok · 2025-11-25T15:21:38Z

Backporting this (together with #141661) makes KiCad work for me on Fedora 43.

markshannon

Thanks for doing this. Adding the docs is a good idea.

bedevere-bot · 2025-11-25T22:12:05Z

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 0728b62 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F141944%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

CPython 3.14.0/3.14.1 fatally crash Ray async actors running on boost fiber stacks (python/cpython#141944, fixed upstream in 3.14.2), which kills the JobSupervisor on every Anyscale job. The py3.14 images are stuck on 3.14.0 for two reasons: the unpinned 'conda install python=3.14' resolve is frozen in the wanda layer cache, and the libffi=3.4.6 pin in base-deps forces conda to pick python 3.14.0 even on a fresh build (newer 3.14.x conda-forge builds require libffi>=3.5). Pin python 3.14.6 in base-deps and base-slim, and let 3.14 bring its own libffi. Verified: the async-actor repro crashes in rayproject/ray:nightly-py314-cpu (3.14.0) and passes with the same nightly cp314 wheel on 3.14.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

CPython 3.14.0/3.14.1 fatally crash Ray async actors running on boost fiber stacks (python/cpython#141944, fixed upstream in 3.14.2), which kills the JobSupervisor on every Anyscale job. The py3.14 base image resolves python 3.14.0 because the exact libffi=3.4.6 pin, installed as a separate conda step, re-solves the env and downgrades python to the only 3.14 build compatible with libffi<3.5. Replace the two-step install with a single solve using a libffi floor (>=3.4.6, keeping the intent of the original pin): every python version then resolves its newest patch release with a compatible libffi — 3.14.6 + libffi 3.5.2 for py3.14. Verified via dry-run for 3.10-3.14 and ctypes smoke tests on 3.10 (libffi 3.7.0) and 3.14 (libffi 3.5.2); the async-actor repro crashes on 3.14.0 and passes with the same nightly cp314 wheel on 3.14.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…mage resolving CPython 3.14.0 (#64857) ## Description ### Release smoke test Adds a `hello_world_py314` nightly smoke release test (aws variation only), mirroring the existing `hello_world_py313` entry: - `release/ray_release/schema.json` — add `"3.14"` to the `python` enum (release-test config validation rejects `python: "3.14"` without this). - `release/ray_release/config.py` — add `"3.14"` to the cpu/cu123 BYOD python allowlist (the parallel gate the release runner walks). - `release/release_tests.yaml` — new `hello_world_py314` test, nightly, `byod: {}`, same `hello_world_compute_config.yaml` as the other hello_world tests. The py3.14 `ray-anyscale` release-test images (cpu + cuda) are already built and published on master via `.buildkite/release/build.rayci.yml`, so no image plumbing is needed here. ### Base-image fix (what made the smoke test fail) The first run of this test failed with the JobSupervisor dying at job startup: `Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable stack overflow`. Root cause: the py3.14 images ship **CPython 3.14.0**, which fatally crashes any Ray async actor running on a boost fiber stack (python/cpython#141944, fixed upstream in **3.14.2**). Why the images resolve 3.14.0: `docker/base-deps/Dockerfile` installs an exact `libffi=3.4.6` pin as a *separate* conda step after installing python. That second solve downgrades python to the only 3.14 build compatible with `libffi<3.5` — which is 3.14.0. (The stale wanda layer cache compounds this, but even a fresh rebuild today re-resolves 3.14.0 because of the pin.) Fix: replace the two-step install with a **single solve using a libffi floor** (`libffi>=3.4.6`, preserving the intent of the original pin — the 3.4.2/defaults-channel libffi was buggy). No per-version special casing; every python version resolves its newest patch release with a compatible libffi. Resolved versions today: py3.10→3.10.20 (libffi 3.7.0), py3.11→3.11.15, py3.12→3.12.13, py3.13→3.13.14 (libffi 3.5.2), py3.14→**3.14.6** (libffi 3.5.2). The Dockerfile change also busts the stale wanda cache. Note: #64772 (fiber stack-protection re-anchoring) is complementary, not a fix for this crash — its `PyUnstable_ThreadState_SetStackProtection` call only exists on 3.14.2+, so it no-ops on the 3.14.0 currently in the images. Once this lands, #64772 fixes the remaining per-task async-actor memory leak. ## Verification - Reproduced the crash: async-actor repro (`ray.get(A.remote().hi.remote())` with an `async def` method) dies in `rayproject/ray:nightly-py314-cpu` (CPython 3.14.0) with the exact failure signature from release-test job `prodjob_d4dctduzm3h6eu812vrrehiuzl`. - Verified the fix: the same unpatched nightly cp314 wheel on CPython 3.14.6 (`python:3.14-slim`) runs the repro successfully. - Verified the combined solve under miniforge 24.11.3-0 (same as the Dockerfile): dry-runs for python 3.10–3.14 all resolve (versions above), plus real installs with `ctypes` smoke tests on 3.10 (libffi 3.7.0) and 3.14.6 (libffi 3.5.2). - `python -m pytest -q release/ray_release/tests/test_config.py` — 23 passed; full collection validates (319 tests) including `hello_world_py314.aws`. ## Duplicate-work note #63237 contains an earlier version of the release-test config bundled with image-build plumbing that has since landed on master through other PRs. This PR carves out the remaining release-test config plus the base-image fix; #63237 can be closed or rebased down to the raylet fix. AI assistance (Claude Code) was used for this PR; all changes reviewed by the submitter. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…protection to fiber stacks (#64772) # Description On Python 3.14 + Linux, every async-actor task permanently leaks ~518 KiB of live malloc (the per-task `asyncio.Task`, `concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack `Packer`s with 256 KiB internal buffers). Closes#63290 ### Root cause **1. CPython 3.14 changed how it avoids stack overflow when freeing objects.** Freeing one object can recursively free many others (a dict frees its values, which free their contents, …), and each level is a nested C call. To keep that from overflowing the C stack, CPython has long had a safety mechanism (the "trashcan"): when it decides it's too deep, it doesn't free the object right away. Instead it parks the object on a per-thread *delete-later* list and drains the list once there's stack headroom again. Up to 3.13, "too deep" was a simple recursion counter. In 3.14 it's decided by comparing the actual machine **stack pointer** against the stack bounds CPython recorded for the thread when it attached (from pthreads, on Linux). **2. Ray async actors don't run task code on the thread's normal stack.** Each task executes on a small 256 KiB boost fiber stack allocated elsewhere in memory. The problem is that CPython still thinks the thread runs on its original pthread stack. So while a task runs on a fiber, every "am I near the stack limit?" check compares the fiber's stack pointer against the *pthread* stack's bounds. On Linux, fiber stacks happen to be allocated at lower addresses than the pthread stack, so CPython concludes the stack is hopelessly overflowed and parks **every** object freed during the task (including return-value serialization and end-of-task cleanup) on the delete-later list. That list is only ever drained by a later free on the same thread state at a healthy stack margin, which never happens here as the Ray thread only runs on fibers and Ray creates a fresh Python thread state per task and destroys it at task end. This means that CPython destroys a thread state **without draining its delete-later list** and the parked objects are orphaned permanently. That's the leak. Why the confusing symptoms: - `boost::make_fcontext` in the issue's flamegraphs just marks *where* the leaked allocations were made (on a fiber stack); the fiber stacks themselves are freed correctly. - macOS is unaffected only by luck: fiber stacks there land at *higher* addresses than the pthread stack, so the check passes. - 3.13 and earlier are unaffected because their trashcan uses the counter, not the stack pointer. ### Fix CPython 3.14.2 added an official API for exactly this situation: `PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets an embedder tell CPython "this thread is currently executing on *this* stack." We call it with the fiber's stack bounds: - at async-actor task entry in `task_execution_handler`, and - whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers share the thread state, so each must re-register its own stack). With the bounds correct, the near-limit check returns to normal behavior: objects are freed immediately, and the rare genuinely-deep free is parked and then properly drained. Implementation notes: the symbol is looked up via `dlsym`, so `_raylet` still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a more severe, since-fixed stack-check bug anyway, python/cpython#141944). No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are derived from the current stack pointer minus a conservative allowance for stack already used, so the protection errs toward triggering slightly early rather than missing an overflow. Side benefit: fibers gain real C-stack overflow protection (RecursionError) on 3.14, which they currently lack entirely (`boost::fibers::fixedsize_stack` has no guard pages). Also makes `FiberState::kStackSize` public so the anchoring uses the real fiber stack size. ## Related issue number Closes#63290. Supersedes #63284 (same diagnosis direction, but hand-rolled `_PyThreadStateImpl` offsets, a deliberate `gilstate_counter` leak that freezes non-main threads, and a crash premise that CPython 3.14.2 already fixed upstream). ## Checks - Verified with a locally built cp314 Linux (aarch64, python:3.14.6 docker) wheel: - refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks) - `__del__` deferral probe: dealloc during return serialization on the fiber **deferred → immediate** - live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3 KiB/task** across async call → dict/bytes, async generator, sync generator on async actor - reporter-shaped streaming workload (400 tasks, 10 concurrent sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped regions 0 → 0 - async-actor smoke: correctness (echo, state, async generators, recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s) - throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs baseline, same container image): fixed 5624–6076 tasks/s vs unpatched 4551–4825 tasks/s meaning no regression (the unpatched build is slower while leaking) - baseline (unpatched) wheel from the same tree reproduces the bug: +4.00 refs/task, fiber dealloc deferred=True note: fable did a majority of the heavy lifting in this investigation with prompting on what to check next and validate the solution --------- Signed-off-by: Mark Towers <mark@anyscale.com> Signed-off-by: myan <myan@anyscale.com> Co-authored-by: Mark Towers <mark@anyscale.com> Co-authored-by: myan <myan@anyscale.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>

#65177) …protection to fiber stacks (#64772) # Description On Python 3.14 + Linux, every async-actor task permanently leaks ~518 KiB of live malloc (the per-task `asyncio.Task`, `concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack `Packer`s with 256 KiB internal buffers). Closes#63290 ### Root cause **1. CPython 3.14 changed how it avoids stack overflow when freeing objects.** Freeing one object can recursively free many others (a dict frees its values, which free their contents, …), and each level is a nested C call. To keep that from overflowing the C stack, CPython has long had a safety mechanism (the "trashcan"): when it decides it's too deep, it doesn't free the object right away. Instead it parks the object on a per-thread *delete-later* list and drains the list once there's stack headroom again. Up to 3.13, "too deep" was a simple recursion counter. In 3.14 it's decided by comparing the actual machine **stack pointer** against the stack bounds CPython recorded for the thread when it attached (from pthreads, on Linux). **2. Ray async actors don't run task code on the thread's normal stack.** Each task executes on a small 256 KiB boost fiber stack allocated elsewhere in memory. The problem is that CPython still thinks the thread runs on its original pthread stack. So while a task runs on a fiber, every "am I near the stack limit?" check compares the fiber's stack pointer against the *pthread* stack's bounds. On Linux, fiber stacks happen to be allocated at lower addresses than the pthread stack, so CPython concludes the stack is hopelessly overflowed and parks **every** object freed during the task (including return-value serialization and end-of-task cleanup) on the delete-later list. That list is only ever drained by a later free on the same thread state at a healthy stack margin, which never happens here as the Ray thread only runs on fibers and Ray creates a fresh Python thread state per task and destroys it at task end. This means that CPython destroys a thread state **without draining its delete-later list** and the parked objects are orphaned permanently. That's the leak. Why the confusing symptoms: - `boost::make_fcontext` in the issue's flamegraphs just marks *where* the leaked allocations were made (on a fiber stack); the fiber stacks themselves are freed correctly. - macOS is unaffected only by luck: fiber stacks there land at *higher* addresses than the pthread stack, so the check passes. - 3.13 and earlier are unaffected because their trashcan uses the counter, not the stack pointer. ### Fix CPython 3.14.2 added an official API for exactly this situation: `PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets an embedder tell CPython "this thread is currently executing on *this* stack." We call it with the fiber's stack bounds: - at async-actor task entry in `task_execution_handler`, and - whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers share the thread state, so each must re-register its own stack). With the bounds correct, the near-limit check returns to normal behavior: objects are freed immediately, and the rare genuinely-deep free is parked and then properly drained. Implementation notes: the symbol is looked up via `dlsym`, so `_raylet` still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a more severe, since-fixed stack-check bug anyway, python/cpython#141944). No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are derived from the current stack pointer minus a conservative allowance for stack already used, so the protection errs toward triggering slightly early rather than missing an overflow. Side benefit: fibers gain real C-stack overflow protection (RecursionError) on 3.14, which they currently lack entirely (`boost::fibers::fixedsize_stack` has no guard pages). Also makes `FiberState::kStackSize` public so the anchoring uses the real fiber stack size. ## Related issue number Closes#63290. Supersedes #63284 (same diagnosis direction, but hand-rolled `_PyThreadStateImpl` offsets, a deliberate `gilstate_counter` leak that freezes non-main threads, and a crash premise that CPython 3.14.2 already fixed upstream). ## Checks - Verified with a locally built cp314 Linux (aarch64, python:3.14.6 docker) wheel: - refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks) - `__del__` deferral probe: dealloc during return serialization on the fiber **deferred → immediate** - live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3 KiB/task** across async call → dict/bytes, async generator, sync generator on async actor - reporter-shaped streaming workload (400 tasks, 10 concurrent sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped regions 0 → 0 - async-actor smoke: correctness (echo, state, async generators, recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s) - throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs baseline, same container image): fixed 5624–6076 tasks/s vs unpatched 4551–4825 tasks/s meaning no regression (the unpatched build is slower while leaking) - baseline (unpatched) wheel from the same tree reproduces the bug: +4.00 refs/task, fiber dealloc deferred=True note: fable did a majority of the heavy lifting in this investigation with prompting on what to check next and validate the solution --------- (cherry picked from commit 35591ba) Signed-off-by: Mark Towers <mark@anyscale.com> Signed-off-by: myan <myan@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Mark Towers <mark.m.towers@gmail.com> Co-authored-by: Mark Towers <mark@anyscale.com> Co-authored-by: myan <myan@anyscale.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>

encukou requested review from markshannon and vstinner November 25, 2025 13:26

bedevere-appBot added the awaiting core review label Nov 25, 2025

This was referenced Nov 25, 2025
GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. #141711
Merged
Python 3.14 stack overflow detection is incompatible with C++ Boost make_fcontext() coroutines #139653
Open

vstinner approved these changes Nov 25, 2025
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting core review labels Nov 25, 2025

markshannon approved these changes Nov 25, 2025
View reviewed changes

markshannon added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 25, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 25, 2025

encukou merged commit 69021e9 into python:3.14Nov 26, 2025
126 checks passed

bedevere-appBot removed the awaiting merge label Nov 26, 2025

encukou deleted the backport-c25a070-3.14 branch November 26, 2025 11:55

cjdoris mentioned this pull request Nov 28, 2025
Intermittent CI failures: Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable stack overflow JuliaPy/PythonCall.jl#694
Closed

pseudo-rnd-thoughts mentioned this pull request Jul 15, 2026
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack protection to fiber stacks ray-project/ray#64772
Merged

elliot-barn mentioned this pull request Jul 17, 2026
[release] Add hello_world_py314 smoke release test; fix py3.14 base image resolving CPython 3.14.0 ray-project/ray#64857
Merged

elliot-barn mentioned this pull request Aug 3, 2026
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack … ray-project/ray#65177
Merged

Navigation Menu

Uh oh!

Uh oh!

[3.14] GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. (GH-141711) - #141944

[3.14] GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. (GH-141711)#141944
encukou merged 1 commit into
python:3.14from
encukou:backport-c25a070-3.14

encukou commented Nov 25, 2025•
edited by bedevere-app Bot
Loading

Uh oh!

vstinner left a comment

Uh oh!

hroncok commented Nov 25, 2025

Uh oh!

markshannon left a comment

Uh oh!

bedevere-bot commented Nov 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Uh oh!

Conversation

encukou commented Nov 25, 2025• edited by bedevere-app BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

hroncok commented Nov 25, 2025

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Nov 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

encukou commented Nov 25, 2025•
edited by bedevere-app Bot
Loading