gh-149101: Implement PEP 788

[ZeroIntensity]

Hugo has graciously given me permission to backport this if we don't make the May 5th deadline, but let's try to get this done in time!

I will write a full tutorial and migration guide once this is merged; I want to first make sure that this lands before the beta freeze.

• Issue: Implement PEP 788 -- Protecting the C API from Interpreter Finalization #149101

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #32572982 | 📁 Comparing 53fa94b against main (40dc61a)

🔍 Preview build

76 files changed · + 1 added · ± 75 modified

+Added

• c-api/slots.html

±Modified

• glossary.html
• c-api/concrete.html
• c-api/exceptions.html
• c-api/extension-modules.html
• c-api/index.html
• c-api/interp-lifecycle.html
• c-api/lifecycle.html
• c-api/module.html
• c-api/perfmaps.html
• c-api/stable.html
• and 65 more...

[encukou]

Thanks for adding these!

I'll send notes for Doc/ now; code review coming up.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit bc78c10 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F149116%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[encukou]

for buildbots: The RHEL8 failures aren't relevant. Refleaks are worrying though.

[encukou]

Refleaks are worrying though.

Never mind; main currently leaks (#149179).

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit a34dbc6 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F149116%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[colesbury]

This looks good to me. The only remaining issues I see are minor. GitHub is kind of broken so here are my comments:

• Some things found by Claude: https://gist.github.com/colesbury/e3ecafad30bb291d205c7be88c7fe7d4 (same as I messaged Peter in Discord)
• In make_pre_finalization_calls, the atomic compare exchange on finalization_guards should happen before HEAD_UNLOCK(interp->runtime). In other words, we should do both the checks and state transition under the same locks.
• We don't need both NO_TSTATE_SENTINEL and DETACHED_TSTATE_SENTINEL. You can just keep one or the other.

The comment about tstate->ensure.owned_guard = NULL (" This is usually done by PyThreadState_Clear()...") is confusing to me. The owned_guard = NULL in PyThreadState_Clear is both useless and not a concern of PyThreadState_Clear. These fields are managed by Ensure/Release, not PyThreadState_New/Clear/Delete and we are not clearing or touching any of the other ensure fields. I'd remove that line from PyThreadState_Clear and write the relevant part of PyThreadState_Release as:

if (tstate->ensure.delete_on_release) {
++tstate->ensure.counter;
PyThreadState_Clear(tstate);
--tstate->ensure.counter;
}
elseif (owned_guard!=NULL) {
tstate->ensure.owned_guard=NULL;
}

[ZeroIntensity]

Thanks for the reviews, both of you! Sorry again about the rushing.