bpo-44133: Link Python executable with object files

[vstinner]

When Python is built without --enable-shared, the "python" program is
now linked to object files, rather than be linked the Python static
library (libpython.a), to make sure that all symbols are exported.
Previously, the linker omitted some symbols like the Py_FrozenMain()
function.

When Python is configured with --without-static-libpython, the Python
static library (libpython.a) is no longer built.

• Reorganize configure.ac to check --without-static-libpython
  earlier.
• Add LINK_PYTHON_OBJS and LINK_PYTHON_DEPS variables to Makefile.

https://bugs.python.org/issue44133

[vstinner]

With this change, import ctypes; assert hasattr(ctypes.pythonapi, 'Py_FrozenMain') no longer fails: the symbol is now available in all tests. I tested 4 cases.

The "python" file only exports the symbol when it's not linked to libpython.so.

When libpython.so exists, il always exports the symbol.

Case 1 (no option):

$ ls *.a *.so
# no file
# grep '^LINK_PYTHON_' Makefile
LINK_PYTHON_DEPS=$(LIBRARY_DEPS)
LINK_PYTHON_OBJS=$(LIBRARY_OBJS)
++ objdump -T ./python | grep Py_FrozenMain
00000000006722b2 g DF .text	0000000000000285 Base Py_FrozenMain
# ./python test.py
python binary has Py_FrozenMain symbol? True

Case 2, --without-static-libpython --enable-shared:

$ ls *.a *.so
libpython3.11d.so
$ grep '^LINK_PYTHON_' Makefile
LINK_PYTHON_DEPS=$(LIBRARY_DEPS)
LINK_PYTHON_OBJS=$(BLDLIBRARY)
$ objdump -T ./python | grep Py_FrozenMain
# not found
$ objdump -T libpython3.11d.so | grep Py_FrozenMain
00000000003b368a g DF .text	00000000000002a9 Base Py_FrozenMain
# ./python test.py
python binary has Py_FrozenMain symbol? True

Case 3, --without-static-libpython:

$ ls *.a *.so
# no file
$ grep '^LINK_PYTHON_' Makefile
LINK_PYTHON_DEPS=$(LIBRARY_DEPS)
LINK_PYTHON_OBJS=$(LIBRARY_OBJS)
$ objdump -T ./python | grep Py_FrozenMain
00000000006722b2 g DF .text	0000000000000285 Base Py_FrozenMain
# ./python test.py
python binary has Py_FrozenMain symbol? True

Case 4, --enable-shared:

$ ls *.a *.so
libpython3.11d.a libpython3.11d.so
$ grep '^LINK_PYTHON_' Makefile
LINK_PYTHON_DEPS=$(LIBRARY_DEPS)
LINK_PYTHON_OBJS=$(BLDLIBRARY)
$ objdump -T ./python | grep Py_FrozenMain
# no result
$ objdump -T libpython3.11d.so | grep Py_FrozenMain
00000000003b368a g DF .text	00000000000002a9 Base Py_FrozenMain
$ ./python test.py
python binary has Py_FrozenMain symbol? True

[vstinner]

I don't know how to test this change on macOS to build Python with the "framework".

[vstinner]

@tiran@erlend-aasland: Would you mind to review this change?

cc @encukou

[vstinner]

On my Fedora 35, if I compare objdump -T ./python output on Python configure with --without-static-libpython, the difference with this PR is a single additional symbol: Py_FrozenMain() is now exported. Hopefully, no other symbol was missing previously.

[vstinner]

PR rebased to fix merge conflicts.

[vstinner]

I may be a good idea to test this build change on buildbots before merging the PR :-) Fedora and RHEL buildbots use --without-static-libpython.

[shihai1991]

The gate failed. the test of this PR should be skipped in wins?

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @shihai1991 for commit c5c7102 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[vstinner]

@shihai1991: "The gate failed. the test of this PR should be skipped in wins?"

No, the symbol must be present on all platforms. I pushed a fix.

[vstinner]

Ah, adding frozenmain.c to the pythoncore project makes the Windows build to fail:

frozenmain.obj : error LNK2001: unresolved external symbol _PyWinFreeze_ExeTerm [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
frozenmain.obj : error LNK2001: unresolved external symbol _PyInitFrozenExtensions [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
frozenmain.obj : error LNK2001: unresolved external symbol _PyWinFreeze_ExeInit [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]

[vstinner]

Ok, I give up for this PR (I skip the test on Windows): I have no idea how to add Py_FrozenMain() on Windows if it requires 3 symbols when Python is linked.

Maybe Py_FrozenMain() should be modified to load these symbols at runtime on Windows?