gh-135906: Test the internal C API in test_cext

[vstinner]

• Issue: Use _PyObject_CAST in internal headers #135906

[vstinner]

I tried to check for C++ compiler warnings in internal Python C headers, but I got many C++ compiler errors in mimalloc and I don't know how to handle them. Examples:

Infileincludedfrom /usr/include/features.h:524,
from /usr/include/assert.h:35,
from /home/vstinner/python/main/Include/Python.h:19,
fromextension.cpp:13:
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/types.h:73:33: error: expectedunqualified-idbefore 'noexcept'
73 | #definemi_decl_throw__THROW
| ^~~~~~~
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/types.h:611:31: note: inexpansionofmacro'mi_decl_throw'611 | mi_decl_noreturnmi_decl_coldmi_decl_throw
| ^~~~~~~~~~~~~

and

Infileincludedfrom /home/vstinner/python/main/Include/internal/pycore_mimalloc.h:44,
from /home/vstinner/python/main/Include/internal/pycore_tstate.h:13,
from /home/vstinner/python/main/Include/internal/pycore_pystate.h:13,
fromextension.cpp:20:
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/internal.h: Infunction 'bool_mi_is_aligned(void*, size_t)':
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/types.h:613:49: error: '_mi_assert_fail'wasnotdeclaredinthisscope; didyoumean'__assert_fail'?
613 | #definemi_assert(expr) ((expr) ? (void)0 : _mi_assert_fail(#expr,__FILE__,__LINE__,__func__))
| ^~~~~~~~~~~~~~~
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/types.h:619:31: note: inexpansionofmacro'mi_assert'619 | #definemi_assert_internal mi_assert
| ^~~~~~~~~
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/internal.h:273:3: note: inexpansionofmacro'mi_assert_internal'273 | mi_assert_internal(alignment!=0);
| ^~~~~~~~~~~~~~~~~~

[vstinner]

I updated test_cppext to have a bare minimum test of the internal C API for now: test including pycore_frame.h. We can extend the test later.

[vstinner]

Note to mysel: GitHub Action skips these tests which require the cpu test resource (-u cpu or -u all). Only buildbots run these slow tests.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 69588da 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[vstinner]

test_cext fails on on multiple buildbots (s390x Fedora Stable Clang PR, AMD64 FreeBSD PR, x86-64 macOS PR, ...) with:

 In file included from extension.c:21:
In file included from /home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang/build/Include/internal/pycore_gc.h:11:
/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang/build/Include/internal/pycore_interp_structs.h:132:16: error: redefinition of typedef 'atexit_datacallbackfunc' is a C11 feature [-Werror,-Wtypedef-redefinition]
132 | typedef void (*atexit_datacallbackfunc)(void *);
| ^
/home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang/build/Include/cpython/pylifecycle.h:87:16: note: previous definition is here
87 | typedef void (*atexit_datacallbackfunc)(void *);
| ^

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 49faa0c 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[vstinner]

test_cext fails on ARM64 Windows Non-Debug PR:

 "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX86\ARM64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\ARM64 -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\build\test_python_28188�\tempcwd\env\include -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PC -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PC "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\ATLMFC\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\cppwinrt" /Tcextension.c /Fobuild\temp.win-arm64-cpython-315\Release\extension.obj /W4 /WX -DMODULE_NAME=_test_cext -DTEST_INTERNAL_C_API=1
cl : Command line warning D9025 : overriding '/W3' with '/W4'
extension.c
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(509): error C2220: the following warning is treated as an error
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(509): warning C4324: 'mi_abandoned_pool_s': structure was padded due to alignment specifier
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(512): warning C4324: 'mi_abandoned_pool_s': structure was padded due to alignment specifier
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(513): warning C4324: 'mi_abandoned_pool_s': structure was padded due to alignment specifier
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(518): warning C4324: 'mi_abandoned_pool_s': structure was padded due to alignment specifier
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_qsbr.h(75): warning C4200: nonstandard extension used: zero-sized array in struct/union
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_backoff.h(68): warning C4244: 'function': conversion from 'int' to 'uint16_t', possible loss of data
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_code.h(114): warning C4201: nonstandard extension used: nameless struct/union
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_object_deferred.h(20): warning C4100: 'op': unreferenced formal parameter
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_stackref.h(792): warning C4100: 'tstate': unreferenced formal parameter
C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_stackref.h(803): warning C4100: 'tstate': unreferenced formal parameter
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX86\\ARM64\\cl.exe' failed with exit code 2
error: subprocess-exited-with-error

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 7870668 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[vstinner]

ARM64 Windows Non-Debug PR failed. .TestInteralCAPI.test_build() failed with:

 extension.obj : error LNK2001: unresolved external symbol PyLong_FromLong
extension.obj : error LNK2001: unresolved external symbol PyModuleDef_Init
extension.obj : error LNK2001: unresolved external symbol PyArg_ParseTuple
build\lib.win-arm64-cpython-315\_test_cext.cp315-win_arm64.pyd : fatal error LNK1120: 3 unresolved externals

and TestInteralCAPI.test_build_c11() fails with:

 extension.c
C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um\winbase.h(9531): error C2220: the following warning is treated as an error
C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um\winbase.h(9531): warning C5105: macro expansion producing 'defined' has undefined behavior
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX86\\ARM64\\cl.exe' failed with exit code 2

[vstinner]

!buildbot Windows

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 30c6265 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

The command will test the builders whose names match following regular expression: Windows

The builders matched are:

• AMD64 Windows PGO NoGIL PR
• AMD64 Windows Server 2022 NoGIL PR
• AMD64 Windows10 PR
• ARM64 Windows Non-Debug PR
• AMD64 Windows11 Refleaks PR
• AMD64 Windows11 Non-Debug PR
• AMD64 Windows11 Bigmem PR
• AMD64 Windows PGO PR
• ARM64 Windows PR

[vstinner]

On ARM64 Windows Non-Debug PR, TestInteralCAPI.test_build_c11() failed with:

 extension.c
C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um\winbase.h(9531): error C2220: the following warning is treated as an error
C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um\winbase.h(9531): warning C5105: macro expansion producing 'defined' has undefined behavior
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX86\\ARM64\\cl.exe' failed with exit code 2

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 9d4d697 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.