gh-93939: Build C extensions without setup.py

[tiran]

Combines GH-93940, GH-94452, and GH-94433

• Issue: Build shared extensions without distutils #93939

Automerge-Triggered-By: GH:tiran

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @tiran for commit 1469caf9703e411b73a00169a6bf0c5e688f5e72 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @tiran for commit 68b3032229ee0185f029772e5e43f72cbb4b402d 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[merwok]

Is there a need for force pushes? Github notifications lead to nothing, which is a little annoying.

[pradyunsg]

FWIW, You can click on "force-pushed" in the event about the force push, to see what changed between the two force-pushes.

Submitting feedback to GitHub to have the notifications for force-pushed go to that, instead of "oh, I can't show you what changed" may help make GitHub prioritze fixing that higher.

[tiran]

I only force push to rebase. It is the easiest and fastest way for me to resolve merge conflicts. The PR has frequent conflicts, even right now.

[tiran]

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @tiran for commit 87f243d764d4909388aed35e59f177d598c096ca 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[tiran]

setup.py's check_extension_import feature is now provided by a script. Here is an example output with tk-dev missing and a modified _sqlite that is missing _sqlite/util.c dependency.

[ERROR] _sqlite3 failed to import: build/lib.linux-x86_64-3.12/_sqlite3.cpython-312-x86_64-linux-gnu.so: undefined symbol: _pysqlite_seterror
The necessary bits to build these optional modules were not found:
_tkinter To find the necessary bits, look in configure.ac and config.log.
Following modules built successfully but were removed because they could not be imported:
_sqlite3 Checked 109 modules (30 built-in, 76 shared, 1 n/a on linux-x86_64, 0 disabled, 1 missing, 1 failed on import)

validate_tzpath is performed by configure.ac

xxsubtype is now an optional extension. It is skipped when Python is built without test C extension modules.

[miss-islington]

Sorry, I can't merge this PR. Reason: Base branch was modified. Review and try the merge again..

[tiran]

#94848 addresses the build issue with LTO + pydebug.

[vstinner]

My little precious setup.py... YOU WILL NEVER BE FORGOTTEN 😭😭😭

_bootsubprocess.py (that I wrote)... I hate you, I'm glad that you're dead! If I recall correctly, I had to write _bootsubprocess to use the clean subprocess module, rather than naive fork+exec code, in distutils. distutils... I also hate you by the way.

[ayappanec]

This broke the AIX build with the below error.

./python -E -S -m sysconfig --generate-posix-vars ;
if test $? -ne 0 ; then
echo "generate-posix-vars failed" ;
rm -f ./pybuilddir.txt ;
exit 1 ;
fi

Traceback (most recent call last):
File "/home/buildusr/cpython_master/cpython/Lib/_aix_support.py", line 7, in
import subprocess
File "/home/buildusr/cpython_master/cpython/Lib/subprocess.py", line 104, in
from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/buildusr/cpython_master/cpython/Lib/sysconfig.py", line 851, in
_main()
File "/home/buildusr/cpython_master/cpython/Lib/sysconfig.py", line 839, in _main
_generate_posix_vars()
File "/home/buildusr/cpython_master/cpython/Lib/sysconfig.py", line 511, in _generate_posix_vars
pybuilddir = f'build/lib.{get_platform()}-{_PY_VERSION_SHORT}'
^^^^^^^^^^^^^^
File "/home/buildusr/cpython_master/cpython/Lib/sysconfig.py", line 781, in get_platform
from _aix_support import aix_platform
File "/home/buildusr/cpython_master/cpython/Lib/_aix_support.py", line 11, in
import _bootsubprocess as subprocess
ModuleNotFoundError: No module named '_bootsubprocess'
generate-posix-vars failed
gmake: *** [Makefile:871: pybuilddir.txt] Error 1
gmake: *** Waiting for unfinished jobs....

[tiran]

Please open a new bug for the problem.

[hroncok]

I found a regression, reported as #98707

[haampie]

Thanks for doing this!