fix: PYBIND11_PLATFORM_ABI_ID Modernization Continued (platforms other than MSVC)

[rwgk]

Description

This PR updates PYBIND11_PLATFORM_ABI_ID, which determines cross-extension ABI compatibility, to improve accuracy across all platforms other than MSVC (which was modernized previously in #4953). For context on the importance of PYBIND11_PLATFORM_ABI_ID, please refer to PRs #5296 and #5375.

The accuracy improvements can be seen by comparing the PYBIND11_INTERNALS_ID summaries before and after:

master @ 741d86f (logs_32284154734.zip, logs_32284154873.zip):

cat *.txt | grep 'C.. Info: .* __pybind11_internals_v'| sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/'| cut -d'' -f1 | sort | uniq -c

 27 __pybind11_internals_vX_clang_libcpp_cxxabi1002__
5 __pybind11_internals_vX_clang_libstdcpp_cxxabi1002__
27 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1013__
3 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1014__
3 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1016__
2 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1017__
5 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1018__
2 __pybind11_internals_vX_icc_libstdcpp_cxxabi1010__
6 __pybind11_internals_vX_mingw_libstdcpp_cxxabi1019__
1 __pybind11_internals_vX_pgi_libstdcpp__

This PR (pr5439_logs_32281771697.zip, pr5439_logs_32281771739.zip):

 6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
26 __pybind11_internals_vX_system_libcpp_abi1__
1 __pybind11_internals_vX_system_libcpp_abi2__
48 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

Note that the compiler-specific identifiers clang, gcc, icc, pgi no longer appear. ABI compatibility is now determined primarily by libstdcpp (gcc) or libcpp (clang) ABI preprocessor macros.

It is expected that overrides like those in pytorch will not be needed anymore. However, this PR ensures that existing overrides will still function, allowing for gradual community adoption.

Similar to #4953, silent fallbacks for unexpected situations have been replaced with #error ... PLEASE REVISE THIS CODE. directives. For such cases, command-line overrides for the affected macros can serve as temporary workarounds. To eliminate the need for such workarounds, users are encouraged to contribute updates to pybind11/conduit/pybind11_platform_abi_id.h as needed.

For completeness, the counts for MSVC (unchanged):

 3 __pybind11_internals_vX_msvc_md_mscver19__
34 __pybind11_internals_vX_msvc_md_mscver19_debug__
3 __pybind11_internals_vX_msvc_mt_mscver1942__

The work on this PR was informed by:

• fix: fix MSVC MT/MD incompatibility in PYBIND11_BUILD_ABI #4953 (comment)
• fix: fix MSVC MT/MD incompatibility in PYBIND11_BUILD_ABI #4953 (comment)
• Build platform ABI tag improvements #5437

Suggested changelog entry:

``PYBIND11_PLATFORM_ABI_ID`` (which is used in composing ``PYBIND11_INTERNALS_ID``) was modernized to reflect actual ABI compatibility more accurately.

[rwgk]

With this PR at commit 14f8425 and logs_30720380481.zip downloaded from here:

• https://github.com/pybind/pybind11/actions/runs/11769449932/job/32780487991?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c

 26 __pybind11_internals_vX_clang_libcpp_libcpp1__
10 __pybind11_internals_vX_clang_libstdcpp_usecxx111__
40 __pybind11_internals_vX_gcc_libstdcpp_usecxx111__
2 __pybind11_internals_vX_icc_libstdcpp_usecxx111__
6 __pybind11_internals_vX_mingw_libstdcpp_usecxx111__
3 __pybind11_internals_vX_msvc_md_mscver19__
34 __pybind11_internals_vX_msvc_md_mscver19_debug__
3 __pybind11_internals_vX_msvc_mt_mscver1941__
1 __pybind11_internals_vX_pgi_libstdcpp__

This doesn't look right, _LIBCPP_ABI_VERSION and _GLIBCXX_USE_CXX11_ABI always appear to be 1.

@robertmaynard I was trying to follow your #4953 (comment) but maybe I'm misunderstanding?

(I want to leave NVHPC for later.)

@wjakob@henryiii@isuruf@cryos for awareness.

[robertmaynard]

With this PR at commit 14f8425 and logs_30720380481.zip downloaded from here:

• https://github.com/pybind/pybind11/actions/runs/11769449932/job/32780487991?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c

 26 __pybind11_internals_vX_clang_libcpp_libcpp1__
10 __pybind11_internals_vX_clang_libstdcpp_usecxx111__
40 __pybind11_internals_vX_gcc_libstdcpp_usecxx111__
2 __pybind11_internals_vX_icc_libstdcpp_usecxx111__
6 __pybind11_internals_vX_mingw_libstdcpp_usecxx111__
3 __pybind11_internals_vX_msvc_md_mscver19__
34 __pybind11_internals_vX_msvc_md_mscver19_debug__
3 __pybind11_internals_vX_msvc_mt_mscver1941__
1 __pybind11_internals_vX_pgi_libstdcpp__

This doesn't look right, _LIBCPP_ABI_VERSION and _GLIBCXX_USE_CXX11_ABI always appear to be 1.

@robertmaynard I was trying to follow your #4953 (comment) but maybe I'm misunderstanding?

(I want to leave NVHPC for later.)

@wjakob@henryiii@isuruf@cryos for awareness.

_GLIBCXX_USE_CXX11_ABI is a boolean which represents which ABI for a set of the C++ std containers you are using ( std::string, list? ). It is an independent axis compared to __GXX_ABI_VERSION.

__GXX_ABI_VERSION	_GLIBCXX_USE_CXX11_ABI	Notes
102	0	Only compatible with 102. Predates _GLIBCXX_USE_CXX11_ABI
1002	0	Compatible with >= 1002 && < 2000 + CXX11 abi of 0
1002	1	Compatible with >= 1002 && < 2000 + CXX11 abi of 1
1013	0	Compatible with >= 1002 && < 2000 + CXX11 abi of 0
1013	1	Compatible with >= 1002 && < 2000 + CXX11 abi of 1

This presumes that GCC will rev the major value of __GXX_ABI_VERSION when it next breaks ABI compat

[isuruf]

One more thing to fix would be to remove the compiler id _gcc or _clang as they are compatible with each other. I think the current checks are supposed to protect against the C standard library, but is too broad.

[wjakob]

I am surprised by __pybind11_internals_vX_msvc_mt_mscver1941__ -- how does the current code manage to generate the number 1941? (I thought 19 was hardcoded).

[rwgk]

I am surprised by __pybind11_internals_vX_msvc_mt_mscver1941__ -- how does the current code manage to generate the number 1941? (I thought 19 was hardcoded).

It's generated in this line:

pybind11/include/pybind11/conduit/pybind11_platform_abi_id.h

Line 67 in 08095d9

# definePYBIND11_BUILD_ABI "_mt_mscver" PYBIND11_PLATFORM_ABI_ID_TOSTRING(_MSC_VER)

[wjakob]

Ah, interesting. So for /MT, the whole _MSCVER should be included? @isuruf, can you confirm? I am not doing this in nanobind currently and will have to make a change if so.

[isuruf]

Yes. /MT means a static linkage of the C runtime, and it's better to have the full version.

[cryos]

I just pushed a new commit that produces something like

__pybind11_internals_v5_gcc_libstdcpp_gxx_abi_1_usecxx11_1__

It also ensures __GXX_ABI_VERSION is less than 2000, so labeling gxx_abi_1, maybe gxx_abi_1xxx would be better. The thought process was to include that as well should it be bumped in the future.

[rwgk]

Here are the results extracted from the latest CI logs:

• https://github.com/pybind/pybind11/actions/runs/11942204141

(Click on any job, then the gear that appears in the top-right corner of the window with the log for that job, then "Download log archive".)

$ mkdir junk && cd junk
$ unzip ../logs_31139436091.zip
$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c

 26 __pybind11_internals_vX_clang_libcpp_libcpp1__
10 __pybind11_internals_vX_clang_libstdcpp_gxx_abi_1_usecxx11_1__
40 __pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1_usecxx11_1__
2 __pybind11_internals_vX_icc_libstdcpp_gxx_abi_1_usecxx11_1__
6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1_usecxx11_1__
3 __pybind11_internals_vX_msvc_md_mscver19__
34 __pybind11_internals_vX_msvc_md_mscver19_debug__
3 __pybind11_internals_vX_msvc_mt_mscver1942__
1 __pybind11_internals_vX_pgi_libstdcpp__

On chat @cryos and I decided to go with gxx_abi_1xxx_ instead of gxx_abi_1_, for clarity.

Looking at these:

__pybind11_internals_vX_clang_libstdcpp_gxx_abi_1_usecxx11_1__
__pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1_usecxx11_1__

I think we want to omit clang and gcc here, based on my understanding of suggestions by @robertmaynard and @isuruf.

But what about
__pybind11_internals_vX_icc_libstdcpp_gxx_abi_1_usecxx11_1__
?

Does anyone know if icc-compiled code is ABI-compatible with clang and gcc?

And what should we do about this one? __pybind11_internals_vX_clang_libcpp_libcpp1__

Could/should we just omit these lines

# elif defined(_LIBCPP_ABI_VERSION) // https://libcxx.llvm.org/DesignDocs/ABIVersioning.html
# define PYBIND11_BUILD_ABI "_libcpp" PYBIND11_PLATFORM_ABI_ID_TOSTRING(_LIBCPP_ABI_VERSION)

and rely on the new __GXX_ABI_VERSION, _GLIBCXX_USE_CXX11_ABI logic instead?

[rwgk]

Here are the results extracted from the latest CI logs:

• https://github.com/pybind/pybind11/actions/runs/11944671063/job/33296033078?pr=5439 (attempt 1, 1 flake)
• https://github.com/pybind/pybind11/actions/runs/11944671063?pr=5439 (deflaked)

 26 __pybind11_internals_vX_clang_libcpp_abi1__
10 __pybind11_internals_vX_clang_libstdcpp_gxx_abi_1xxx_usecxx11_1__
42 __pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1xxx_usecxx11_1__
2 __pybind11_internals_vX_icc_libstdcpp_gxx_abi_1xxx_usecxx11_1__
6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_usecxx11_1__
3 __pybind11_internals_vX_msvc_md_mscver19__
34 __pybind11_internals_vX_msvc_md_mscver19_debug__
3 __pybind11_internals_vX_msvc_mt_mscver1942__
1 __pybind11_internals_vX_pgi_libstdcpp__

[rwgk]

The current changes affect icc, therefore it'll be best if we settle the PYBIND11_PLATFORM_ABI_ID also for that compiler in this PR. According to

https://www.intel.com/content/www/us/en/docs/dpcpp-cpp-compiler/developer-guide-reference/2023-2/gcc-compatibility-and-interoperability.html

"C language object files created with the compiler are binary compatible with the GCC and C/C++ language library."

AFAICT, the compiler also links against the systemlibstdcpp, therefore I figure it must be binary compatible for C++ as well.

Upshot: I think we should also omit icc in the PYBIND11_PLATFORM_ABI_ID.

[rwgk]

I just pushed up the experimental commit fe2dbcb which changes PYBIND11_COMPILER_TYPE.

Rationale:

• Test for special cases first, system compiler last.

• Replace "unknown" fall-through with #error, similar to the elimination of fall-throughs for PYBIND11_BUILD_ABI. (Possibly we may want to enable fall-throughs for releases, but always #error in the pybind11 CI jobs.)

• Assume __MINGW32__ and __CYGWIN__ are not binary compatible with MSVC.

• Use _system for gcc, clang, icc.

• Leave __PGI untouched for now. (Possibly after we're sure about gcc, clang, icc, we can also settle the case for PGI in this PR.)

I'll post a new PYBIND11_PLATFORM_ABI_ID overview when all jobs are done and green.

[rwgk]

Latest PYBIND11_INTERNALS_ID overview (this PR at commit fe2dbcb):

logs_31272968106.zip — https://github.com/pybind/pybind11/actions/runs/11999580827/job/33447666543?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | c
ut -d' ' -f1 | sort | uniq -c

 6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
3 __pybind11_internals_vX_msvc_md_mscver19__
34 __pybind11_internals_vX_msvc_md_mscver19_debug__
3 __pybind11_internals_vX_msvc_mt_mscver1942__
1 __pybind11_internals_vX_pgi_libstdcpp__
26 __pybind11_internals_vX_system_libcpp_abi1__
52 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

[rwgk]

@robertmaynard do you have a moment to look at this PR? (The changes are small and all in just one file.)

• Is the PYBIND11_COMPILER_TYPE and PYBIND11_BUILD_ABI logic good as-is for gcc and clang (Linux, macOS)?

• Do we have to worry about _DEBUG under Linux, macOS?

• Should we leave NVHPC (PGI) for later, or try to handle that in this PR as well?

[robertmaynard]

@robertmaynard do you have a moment to look at this PR? (The changes are small and all in just one file.)

• Is the PYBIND11_COMPILER_TYPE and PYBIND11_BUILD_ABI logic good as-is for gcc and clang (Linux, macOS)?

Looks good to me

• Do we have to worry about _DEBUG under Linux, macOS?

_DEBUG is a MSVCism. Linux and macOS have NDEBUG but that doesn't change ABI.

• Should we leave NVHPC (PGI) for later, or try to handle that in this PR as well?

NVHPC should be ABI compatible with gcc. I am fine either way in how you want to manage that integration from a PR perspective.

[rwgk]

Thanks a lot @robertmaynard for the review!

NVHPC should be ABI compatible with gcc.

Based on your comment, I added commit 9fc9515:

Add NVHPC (__PGI) to the list of compilers (assumed to be) compatible with system compiler.

[hpkfft]

In the interest of simplification, consider removing defined(__INTEL_COMPILER)

The Intel classic compilers, icc for C and icpc for C++, define __GNUC__ as well as __INTEL_COMPILER.
The last version was 2021.10 and it stopped shipping in 2023.

The "Intel(R) oneAPI DPC++/C++ Compilers", icx for C and icpx for C++, do not define __INTEL_COMPILER.
They do define all of these: __GNUC__, __clang__, __INTEL_CLANG_COMPILER, and __INTEL_LLVM_COMPILER.

[rwgk]

Done: 738b7ec

Thanks @hpkfft for the suggestion!

[wjakob]

This PR looks great to me now, are there any other planned changes?

[rwgk]

This PR looks great to me now, are there any other planned changes?

Only some work on the PR description. Sadly Flu + RSV have wiped out a lot of my time. I'll try to do it tonight.

[rwgk]

I updated the PR description.

Thanks so much Everybody for all the input and feedback!