gh-140550: Update xxlimited with 3.15 limited API & PEP 697 by encukou · Pull Request #142827 · python/cpython

encukou · 2025-12-16T14:21:03Z

Copy the existing xxlimited to xxlimited_3_13: this module serves as a rudimentary check that we don't break older Limited API. This is similar to xxlimited_35. (Nowadays we know that it's good to have a separator between the major/minor versions, but renaming to xxlimited_3_5 isn't worth the churn.)
Reorganize test_xxlimited to make similar copies easier in the future
Switch xxlimited to Limited API 3.15, PyModExport (PEP-793), standalone instance struct with extended basicsize (PEP-697)
treat PyObject as opaque, as proposed in PEP-803 and PEP-809, using a private knob (_Py_OPAQUE_PYOBJECT) for now to ensure we don't rely on PyObject layout details accidentally. (I'll replace/remove the private API in 3.15 beta at latest.)

Issue: Implement PEP 793 – PyModExport #140550

This adds the experimental free-threading support.

ZeroIntensity · 2025-12-16T16:08:04Z

+ /* TODO: This is not quite true yet: there is a race in Xxo_setattro
+ * for example.
+ */


Wouldn't it be better to mark this as Py_MOD_GIL_USED in the meantime, or are you trying to avoid the warning?

AFAIK, the situation is unchanged since Py_MOD_GIL_NOT_USED was added everywhere in #116882.
I think the intention is/was to make everything thread-safe before some stage of free-threading implementation?

I think we added that everywhere under the assumption that everything was already thread-safe, and anything that should be reported to us as a bug. I'm not sure that applies with our own test suite.

This is not really a test suite, it supposed to be an example/template showing how to use the Limited API.
The limited API was never compatible with free-threading (yet), so I don't think there's a bug?
There's an earlier PR, #110764, which did this (Py_NOGIL was later renamed to Py_GIL_DISABLED):
+#ifndef Py_NOGIL #define Py_LIMITED_API 0x030d0000 +#endif
... but that seems to defeat the point of this module :/
Maybe we should have not built these modules at all in free-threading builds, and skip the relevant tests?
Doing that now seems like wasted effort though, when mutexes/critical sections are a genuine TODO item for the limited API.

If it's a template, I'm a little concerned that people will see this and think "oh, this module is thread-safe", when it's really not. The comment helps, but I don't know what percentage of people will ignore it.

Before PyCriticalSection is added to the Stable ABI, we can use PyThread_acquire_lock for per-instance locking. (It's also a nice example of a non-PyObject resource that needs freeing.)

encukou · 2025-12-16T17:57:52Z

I'll mark this draft -- this PR makes some shortcomings quite visible, and I guess it's better to address them before suggesting the new API :)

encukou · 2026-01-06T12:24:24Z

 // traverse: Visit all references from an object, including its type
 static int
-Xxo_traverse(PyObject *op, visitproc visit, void *arg)
+Xxo_traverse(PyObject *self, visitproc visit, void *arg)


@colesbury, regarding your post on Discourse:
I think that’s just PyType_GetBaseByToken and PyType_GetModuleByToken. It seems pretty rare to need to call them from a tp_traverse handler. If really necessary, I think you can work around that in extensions by calling other APIs.
How would you implement the traverse function here?
Xxo is a subclassable type, and it should work on ABIs with different sizeof(PyObject) (i.e. free-threaded & GIL-ful).

By workaround I mean something like:
static Xxo_Type_spec = { ... .flags = Py_TPFLAGS_IMMUTABLETYPE | ... }; ... static PyTypeObject *Xxo_Type; ... Xxo_Type = state->Xxo_Type = (PyTypeObject*)PyType_FromModuleAndSpec(...);
static XxoObject_Data * Xxo_get_data(PyObject *self) { return PyObject_GetTypeData(self, Xxo_Type); }
We could also avoid all this if we had a PyObject_GetTypeDataByToken().

static PyTypeObject *Xxo_Type; ... Xxo_Type = state->Xxo_Type = (PyTypeObject*)PyType_FromModuleAndSpec(...);
That'll overwrite the pointer in a subinterpreter, or if you otherwise load several copies of the module. It's not something we can recommend.

vstinner · 2026-05-06T14:09:06Z

 }
- if (PyModule_AddType(m, (PyTypeObject*)Str_Type) < 0) {
+ if (PyModule_AddType(m, Str_Type) < 0) {
 return -1;


Don't we have to call Py_DECREF(Str_Type) here (on error)?

Thanks for the catch!

vstinner · 2026-05-06T14:10:23Z

+ PySlot_FUNC(Py_mod_exec, xx_modexec),
+
+ /* Module state and associated functions */
+ PySlot_SIZE(Py_mod_state_size, (void*)sizeof(xx_state)),


I don't think that the (void*) cast is needed here:
Suggested change
PySlot_SIZE(Py_mod_state_size, (void*)sizeof(xx_state)),
PySlot_SIZE(Py_mod_state_size, sizeof(xx_state)),

vstinner · 2026-05-06T14:10:59Z

+static PySlot xx_slots[] = {
+ /* Basic metadata */
+ PySlot_STATIC_DATA(Py_mod_name, "xxlimited"),
+ PySlot_STATIC_DATA(Py_mod_doc, (void*)module_doc),


Is the (void*) cast useful here?
Suggested change
PySlot_STATIC_DATA(Py_mod_doc, (void*)module_doc),
PySlot_STATIC_DATA(Py_mod_doc, module_doc),

vstinner

LGTM

miss-islington-app · 2026-05-13T16:35:55Z

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

bedevere-app · 2026-05-13T16:36:13Z

GH-149785 is a backport of this pull request to the 3.15 branch.

…H-149785) (cherry picked from commit fa81cd9) Co-authored-by: Petr Viktorin <encukou@gmail.com>

encukou added 6 commits December 15, 2025 16:40

Copy xxlimited to xxlimited_3_13
8147b92

Run tests for all the xxlimited* modules
65f3fd0

Adjust module names in the 3_13 module
aa43538

Update xxlimited to 3.15 limited API (with opaque PyObject)
323695f
This adds the experimental free-threading support.

Comments & cleanups
597652b

Add Py_mod_abi
22464eb

encukou requested review from gpshead and vstinner December 16, 2025 14:21

encukou requested review from a team, AA-Turner, corona10, emmatyping, ericsnowcurrently and erlend-aasland as code owners December 16, 2025 14:21

bedevere-appBot added the awaiting core review label Dec 16, 2025

bedevere-appBot mentioned this pull request Dec 16, 2025
Implement PEP 793 – PyModExport #140550
Open
25 tasks

encukou mentioned this pull request Dec 16, 2025
gh-124153: Add Py_tp_token support to xxlimited module #141644
Draft

encukou added 2 commits December 16, 2025 15:33

Remove unused import
cc5e8f3

Don't flag static PyABIInfo variables
945b450

encukou added the skip news label Dec 16, 2025

encukou requested a review from ZeroIntensity December 16, 2025 14:44

erlend-aasland removed their request for review December 16, 2025 14:45

ZeroIntensity reviewed Dec 16, 2025
View reviewed changes

encukou marked this pull request as draft December 16, 2025 17:58

bedevere-appBot removed the awaiting core review label Dec 16, 2025

encukou added 3 commits December 17, 2025 12:02

Refcounting fix
3b3ae7c

Style
ff0f300

Expand TODO
29f1690

vstinner reviewed Dec 17, 2025
View reviewed changes

Comment threadModules/xxlimited.c Outdated
Comment threadModules/xxlimited.c
Comment threadModules/xxlimited.c Outdated

encukou added 2 commits December 17, 2025 14:56

Remove comment that might be misleading
d9728de

PyUnicode_EqualToUTF8 can't error (and doesn't type-check)
0bc0198

encukou commented Jan 6, 2026
View reviewed changes

encukou mentioned this pull request Jan 13, 2026
Fix and test PyABIInfo_VAR#142225
Closed

encukou added 4 commits April 8, 2026 15:06

Merge in the main branch
0ae6c58

Use _DuringGC API in traversal functions
619eac7

Don't clobber current exception in tp_dealloc
d680b17

Avoid data race in Xxo_setattro
38e5fea

encukou marked this pull request as ready for review April 10, 2026 12:37

bedevere-appBot added the awaiting core review label Apr 10, 2026

encukou added 4 commits May 5, 2026 17:37

Merge in the main branch
e6f10df

Update xxlimited to PyCriticalSection
e2903da

Update comment
330bb76

Update C globals checker
62f8b5b

vstinner reviewed May 6, 2026
View reviewed changes

encukou added 3 commits May 7, 2026 13:15

Address review comments
f08fac8

Merge in the main branch
f3bb5a1

Merge in the main branch
5b0cd4f

encukou added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label May 8, 2026

Avoid warning
c52a25a

vstinner approved these changes May 13, 2026
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting core review labels May 13, 2026

encukou merged commit fa81cd9 into python:mainMay 13, 2026
66 checks passed

encukou deleted the xxlimited-3.15 branch May 13, 2026 16:35

bedevere-appBot removed the awaiting merge label May 13, 2026

bedevere-appBot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label May 13, 2026

encukou added a commit that referenced this pull request May 13, 2026
[3.15]gh-140550: Update xxlimited with 3.15 limited API (GH-142827) (G…
b650305
…H-149785) (cherry picked from commit fa81cd9) Co-authored-by: Petr Viktorin <encukou@gmail.com>

	PySlot_SIZE(Py_mod_state_size, (void*)sizeof(xx_state)),
	PySlot_SIZE(Py_mod_state_size, sizeof(xx_state)),

	PySlot_STATIC_DATA(Py_mod_doc, (void*)module_doc),
	PySlot_STATIC_DATA(Py_mod_doc, module_doc),

Uh oh!

Uh oh!

Conversation

encukou commented Dec 16, 2025• edited by bedevere-app BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukouDec 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou commented Dec 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

encukouJan 6, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinnerMay 6, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-appBot commented May 13, 2026

Uh oh!

bedevere-appBot commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

encukou commented Dec 16, 2025•
edited by bedevere-app Bot
Loading

encukouDec 16, 2025•
edited
Loading

encukouJan 6, 2026•
edited
Loading

vstinnerMay 6, 2026•
edited
Loading