Some C-API symbols (e.g. Py_FrozenMain) are not always exported on Unix

[vstinner]

BPO	44133
Nosy	@vstinner, @encukou, @miss-islington, @kulikjak, @shihai1991
PRs	bpo-44133: Export Py_FrozenMain() symbol #26130 [3.10] bpo-43795: Add a test for Stable ABI symbol availability using ctypes (GH-26354) #29148 bpo-44133: Export Py_FrozenMain() symbol. #29876 bpo-44133: Link Python executable with object files #30556 bpo-44133: Skip PyThread_get_thread_native_id() if not available #30636
Files	reproduce.tar.gz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-17.13:50:21.465>
created_at = <Date 2021-05-14.16:58:44.058>
labels = ['expert-C-API', '3.11']
title = 'Some C-API symbols (e.g. Py_FrozenMain) are not always exported on Unix'
updated_at = <Date 2022-04-01.08:55:02.796>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-04-01.08:55:02.796>
actor = 'kulikjak'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-17.13:50:21.465>
closer = 'vstinner'
components = ['C API']
creation = <Date 2021-05-14.16:58:44.058>
creator = 'vstinner'
dependencies = []
files = ['50043']
hgrepos = []
issue_num = 44133
keywords = ['patch']
message_count = 17.0
messages = ['393675', '393676', '393693', '404596', '404723', '404748', '406980', '410401', '410503', '410504', '410756', '410757', '410762', '410776', '410787', '416378', '416475']
nosy_count = 5.0
nosy_names = ['vstinner', 'petr.viktorin', 'miss-islington', 'kulikjak', 'shihai1991']
pr_nums = ['26130', '29148', '29876', '30556', '30636']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44133'
versions = ['Python 3.11']

[vstinner]

When Python is built without --enable-shared, the "Py_FrozenMain" is not exported.

When Python is built with --enable-shared, the "Py_FrozenMain" is exported as expected.

I can reproduce the issue with attached reproduce.tar.gz example:

• build.sh exports "func1" symbol
• build_ar.sh doesn't export the "func1" symbol

The difference is that build.sh links directly two object files (.o) into a binary, whereas build_ar.sh creates a static library (.a) and then link the static library into a binary.

Python is always built with a static library (libpythonVERSION.a) which causes the issue.

A solution is to pass -Wl,--whole-archive option to the linker to export *all* symbols exported by all object files, and not only export symbols of the object files which are used.

I'm not sure why, but "Py_FrozenMain" is the *only* impacted symbol of the whole C API.

This issue was discovered by Petr Viktorin who works on the stable ABI:
https://mail.python.org/archives/list/capi-sig@python.org/thread/5QLI3NUP3OSGLCCIBAQOTX4GEJQBWJ6F/

[vstinner]

The "python" binary exports 1610 symbols. With the attached fix, it exports 1611 symbols (+1): "Py_FrozenMain" is also exported :-)

[vstinner]

The symbol is not exported on Windows neither. My PR 26126 ("Test Py_FrozenMain()") fails on Windows with:

"_testembed.obj : error LNK2001: unresolved external symbol __imp_Py_FrozenMain"

("__imp_" prefix is added by Windows DLL loader.)

[encukou]

I'm not sure why, but "Py_FrozenMain" is the *only* impacted symbol of the whole C API.

Apparently, on some platforms PyModule_Create2 and PyModule_FromDefAndSpec2 don't appear either.

Should I rename the issue to cover these as well?

[encukou]

New changeset 276468d by Petr Viktorin in branch 'main':
bpo-43795: Add a test for Stable ABI symbol availability using ctypes (GH-26354)
276468d

[encukou]

Repurposing this bug to track all symbols that don't appear in some configurations.

• Py_FrozenMain should be added to stable ABI when this is fixed. It was left out of the 3.10 stable ABI for only this reason. See discussion here: https://mail.python.org/archives/list/capi-sig@python.org/thread/5QLI3NUP3OSGLCCIBAQOTX4GEJQBWJ6F/

• PyModule_Create2 and PyModule_FromDefAndSpec2 are "temporarily" excluded from tests in [Tools/scripts/stable_abi.py](https://github.com/python/cpython/blob/main/Tools/scripts/stable_abi.py)'s gen_ctypes_test, so that all the other symbols can get regression testing. When this issue is fixed, that special case should be removed.

[kulikjak]

On Solaris (and most likely several other platforms), PyThread_get_thread_native_id is also not available.

[vstinner]

On Solaris (and most likely several other platforms), PyThread_get_thread_native_id is also not available.

Oh, I added an explicit test for that in my PR 30556.

Py_FrozenMain should be added to stable ABI

I suggest to first fix the issue (export the symbol), and then write a second PR to add it to the stable ABI.

[vstinner]

New changeset 6be8489 by Victor Stinner in branch 'main':
bpo-44133: Link Python executable with object files (GH-30556)
6be8489