gh-103509: PEP 697 -- Limited C API for Extending Opaque Types

[encukou]

PEP 697 describes the change. Some details it doesn't cover:

• _PyHeapType_GET_MEMBERS is moved from Include/internal to the only .c file that uses it.
• Tests demonstrate the API is usable for the purpose, and verify the layout with brutish-force.

TODO:

• What's New entry
• Fix the alignof & max_align_t

• Issue: PEP 697 -- Limited C API for Extending Opaque Types #103509

[encukou]

typeobject.c includes <stddef.h> and <stdalign.h>, which should define max_align_t and alignof in C11. Yet, the Windows build complains:

D:\a\cpython\cpython\Objects\typeobject.c(3558,34): warning C4013: 'alignof' undefined; assuming extern returning int [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
D:\a\cpython\cpython\Objects\typeobject.c(3558,53): error C2065: 'max_align_t': undeclared identifier [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

What am I missing?
@zooba, could you help?

(The reference I linked to lists alignof as removed in C23, which is a bit misleading. The macro will be removed because it will become a keyword.)

[zooba]

Looks like you need stdalign.h for alignof (defines the alignof macro to _Alignof), and max_align_t only seems to exist for C++ via cstddef, where it's mapped to double.

[encukou]

That means I can't use max_align_t in CPython? Well, I can work around it.
But I was under the impression that MSVC did support C11. I can start the discussion to add max_align_t (and possibly alignof?) as exceptions to PEP 7. Do you know of any other C11 required features MSVC (or our configuration of it) doesn't support?

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit ec9d5a8 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[arhadthedev]

The TraceRefs buildbot failure:

Modules/gcmodule.c:450: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
Enable tracemalloc to get the memory block allocation traceback

is likely caused by gh-103621.

[arhadthedev]

void* cannot participate in address arithmetics. You need to cast both data_ptr and instance to char * to get distance in bytes.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit 402ecc6 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit db5d49b 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[erlend-aasland]

Very nice!

Sorry for the late review; it took some time to dig through everything.

[encukou]

Thank you for the thorough review!
This should address the remaining comments.

[erlend-aasland]

Thanks! I've got some more comments, but we can deal with those in a follow-up PR if you think it's worth it. Let's land this before the feature freeze kicks in.

(I'm on mobile now; I won't be able to check the changes until tomorrow.)

[arhadthedev]

Please, keep in mind that I neither participated in the PR discussion nor seen the PEP before. So I looked at the rendered docs in the same fashion as an outside user would do.

Here are three optional rough proposals and a forgotten full stop. However, they are non-binding suggestions that can be properly considered later, after 3.12b1 is out (otherwise the two remaining weeks will pass swiftly in the bikeshedding).

I've limited myself to the documentation because I have no expertise in the C API to check such a volume of code in such a short period of time.

[encukou]

Thank you! Fresh eyes is exactly what the documentation needs.

It will also need a tutorial to bring the concepts together -- but when trying to write a tutorial, I keep finding rough edges that should be fixed first :)

[encukou]

Thanks for the reviews!

I'll merge, and leave the issues open for other PRs.

[erlend-aasland]

Thanks for the reviews!

I'll merge, and leave the issues open for other PRs.

Good job! This is a very nice feature.

[encukou]

Thank you!

FWIW, here's a follow-up on the alignof(max_align_t): https://discuss.python.org/t/requiring-compilers-c11-standard-mode-to-build-cpython/26481