Skip to content

fix: unblock the ExecuTorch runtime wheel configure (Python::Python on manylinux) - #4523

Merged
lanluo-nvidia merged 1 commit into
pytorch:mainfrom
shoumikhin:fix-executorch-runtime-pybind-configure
Aug 19, 2026
Merged

fix: unblock the ExecuTorch runtime wheel configure (Python::Python on manylinux)#4523
lanluo-nvidia merged 1 commit into
pytorch:mainfrom
shoumikhin:fix-executorch-runtime-pybind-configure

Conversation

@shoumikhin

@shoumikhinshoumikhin commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

The problem

The ExecuTorch runtime wheel has never built. executorch-runtime-build has failed
on every commit since it was added, on all ten matrix rows, and
executorch-runtime-test is gated on it:

# .github/workflows/ci-linux-x86_64.ymlexecutorch-runtime-test:
if: ... needs.executorch-runtime-build.result == 'success'

That test job is the only thing that runs tests/py/dynamo/executorch/ and the
reference-runner verification, so no ExecuTorch change is covered by CI today. And
because the job also fails on main, it reads as a safe pre-existing failure.

Why it fails

The build stops during CMake configure, before anything compiles:

Python_ADD_LIBRARY: dependent target 'Python::Python' is not defined.
Did you miss to request COMPONENT 'Development.Embed'?
Call Stack (most recent call first):
.../pybind11/tools/pybind11NewTools.cmake:269 (python_add_library)
.../executorch/codegen/tools/CMakeLists.txt:11 (pybind11_add_module)

ExecuTorch declares its Python extension modules as
pybind11_add_module(<target> SHARED ...). pybind11 maps a non-MODULE type onto
pybind11::embed, and CMake's python_add_library then requires Python::Python,
which exists only when Python is found with Development.Embed.

Requesting that component does not work here. It needs a libpython, and the
manylinux CPython has none, so the whole find_package fails instead:

Could NOT find Python (missing: Python_INCLUDE_DIRS Python_LIBRARIES ...)

pybind11 already handles this and asks for the component optionally on purpose:

# pybind11NewTools.cmake# Development.Module support (required for manylinux) started in 3.18set(_pybind11_dev_component Development.Module OPTIONAL_COMPONENTS Development.Embed)

codegen/tools cannot be switched off on its own. It is gated only on
EXECUTORCH_BUILD_PYBIND, which has to stay on because this build requires
portable_lib.

The change

Match pybind11's pattern, then fill the one gap it leaves:

  • Development.Module required, Development.Embed optional. Finding Python before
    pybind11 is the override pybind11 documents.
  • Where Development.Embed is genuinely absent, supply Python::Python as an empty
    INTERFACE, guarded on NOT TARGET so a real libpython wins when the image has
    one. It forwards Python::Module for include directories and links no libpython,
    which is correct for an extension module: its Python symbols resolve from the
    interpreter that loads it.

PYTHON_EXECUTABLE is bridged to Python_EXECUTABLE first, because the build
passes the interpreter under the former name while FindPython reads the latter.
ExecuTorch does the same bridge but only later, so without it this call could pick a
different interpreter than the rest of the build.

A test pins the invariant, added to the file that already pins eight properties of
this same CMakeLists as text: the component must stay optional, and the stand-in
must stay guarded.

The root fix belongs upstream

One keyword: selective_build, portable_lib and data_loader should be MODULE,
not SHARED. CMake skips the check entirely for MODULE, no Python::Python is
needed and no libpython is linked, which is what an extension module wants. That
fixes it for every consumer. This change defers to the real target if a later pin
provides one.

Testing

The configure failure only reproduces in the release image, so the mechanism was
verified separately with cmake 3.30 and 4.4:

  • Embed absent, stand-in used: configures and links.
  • Embed present, real target used: configures and links.
  • Requesting Embed as REQUIRED on an image without libpython fails, so the mechanism
    is not imagined.

The new test was verified in all three directions, because a test that cannot fail
proves nothing:

fixed tree 1 passed
Development.Embed made required 1 failed
stand-in removed 1 failed

On the readelf assertion, deliberately not here

An earlier revision asserted at build time, with readelf -d, that neither
extension links libpython. It is removed rather than kept, for three reasons:

  • It made binutils a hard build dependency on Linux for a check that cannot fire on
    the current ExecuTorch pin, where strip_python_lib() already removes both
    Python::Python and pybind11::embed from those targets
    (executorch/CMakeLists.txt:283, called at :1161 and :1195).
  • This repository already asserts that class of property from a test, not from the
    build: tests/py/core/test_libtorchtrt_linkage.py runs readelf -d on the shipped
    library with check=True.
  • Keeping it here mixed a packaging-hygiene concern into a configure fix.

It is worth having as a test against the built artifact, and belongs in its own
change. Flagging it rather than dropping it quietly.

What this does not fix

With the configure fixed, the build reaches 86% and stops at a pre-existing link
failure that was previously unreachable:

[ 86%] Linking CXX shared library _portable_lib.so
ld.gold: error: .../13/libstdc++.a(functexcept.o): multiple definition of
'std::__throw_bad_array_new_length()'
ld.gold: .../libstdc++_nonshared.a(functexcept80.o): previous definition here

That is not this change. git diff against the merge base touches no line of the
static-libstdc++ logic, and this change is purely additive. Every earlier build died
in configure, so nothing had reached the link step before. The wheel therefore needs
a second, separate fix before it builds.

Making that error visible at all required raising bazel's output limit, which is
#4524.

Also worth stating: a pull request push resolves the fast lane, so one matrix row is
scheduled rather than ten.

@github-actionsgithub-actionsBot added the component: api [Python] Issues re: Python API label Aug 19, 2026
@shoumikhinshoumikhin changed the title fix: request Development.Embed so the ExecuTorch runtime wheel configuresfix: unblock the ExecuTorch runtime wheel configure (Python::Python on manylinux)Aug 19, 2026
@shoumikhin
shoumikhinforce-pushed the fix-executorch-runtime-pybind-configure branch 3 times, most recently from c77fad4 to 7d01513CompareAugust 19, 2026 14:37
@github-actionsgithub-actionsBot added the component: tests Issues re: Tests label Aug 19, 2026
The ExecuTorch runtime wheel has never built. `executorch-runtime-build` has
failed on every commit since it was added, on all ten matrix rows, and
`executorch-runtime-test` is gated on it:
```yaml
# .github/workflows/ci-linux-x86_64.yml
executorch-runtime-test:
if: ... needs.executorch-runtime-build.result == 'success'
```
That test job is the only thing that runs `tests/py/dynamo/executorch/` and the
reference-runner verification, so no ExecuTorch change is covered by CI today.
Because the job also fails on main, it reads as a safe pre-existing failure.
## Why it fails
The build stops during CMake configure, before anything compiles:
```
Python_ADD_LIBRARY: dependent target 'Python::Python' is not defined.
Did you miss to request COMPONENT 'Development.Embed'?
Call Stack (most recent call first):
.../pybind11/tools/pybind11NewTools.cmake:269 (python_add_library)
.../executorch/codegen/tools/CMakeLists.txt:11 (pybind11_add_module)
```
ExecuTorch declares its Python extension modules as
`pybind11_add_module(<target> SHARED ...)`. pybind11 maps a non-MODULE type onto
`pybind11::embed`, and CMake's `python_add_library` then requires the
`Python::Python` target, which exists only when Python is found with the
`Development.Embed` component.
Requesting that component does not work here. It needs a libpython, and the
manylinux CPython has none, so the whole `find_package` fails instead:
```
Could NOT find Python (missing: Python_INCLUDE_DIRS Python_LIBRARIES ...)
```
pybind11 already handles this and asks for the component optionally on purpose:
```cmake
# pybind11NewTools.cmake
# Development.Module support (required for manylinux) started in 3.18
set(_pybind11_dev_component Development.Module OPTIONAL_COMPONENTS Development.Embed)
```
`codegen/tools` cannot be switched off on its own. It is gated only on
`EXECUTORCH_BUILD_PYBIND`, which has to stay on because this build requires
`portable_lib`.
## The change
Match pybind11's pattern, then fill the one gap it leaves:
- `Development.Module` required, `Development.Embed` optional. Finding Python
before pybind11 is the override pybind11 documents.
- Where `Development.Embed` is genuinely absent, supply `Python::Python` as an
empty `INTERFACE`, guarded on `NOT TARGET` so a real libpython wins when the
image has one. It forwards `Python::Module` for include directories and links no
libpython, which is correct for an extension module: its Python symbols resolve
from the interpreter that loads it.
`PYTHON_EXECUTABLE` is bridged to `Python_EXECUTABLE` first, because the build
passes the interpreter under the former name while FindPython reads the latter.
ExecuTorch does the same bridge but only later, so without it this call could pick
a different interpreter than the rest of the build.
A test pins the invariant, in the file that already pins eight properties of this
same CMakeLists as text: the component must stay optional, and the stand-in must
stay guarded.
## The root fix belongs upstream
One keyword: `selective_build`, `portable_lib` and `data_loader` should be
`MODULE`, not `SHARED`. CMake skips the check entirely for `MODULE`, no
`Python::Python` is needed and no libpython is linked, which is what an extension
module wants. That fixes it for every consumer. This change defers to the real
target if a later pin provides one.
## Testing
The configure failure only reproduces in the release image, so the mechanism was
verified separately with cmake 3.30 and 4.4:
- Embed absent, stand-in used: configures and links.
- Embed present, real target used: configures and links.
- Requesting Embed as REQUIRED on an image without libpython fails, so the
mechanism is not imagined.
The new test was verified in all three directions:
- fixed tree: 1 passed
- `Development.Embed` made required: 1 failed
- stand-in removed: 1 failed
Not yet confirmed on all ten matrix rows: a pull request push resolves the fast
lane, which schedules one row.
@shoumikhin
shoumikhinforce-pushed the fix-executorch-runtime-pybind-configure branch from 7d01513 to 2e7552fCompareAugust 19, 2026 16:11

@lanluo-nvidialanluo-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@lanluo-nvidia
lanluo-nvidia merged commit b800c5d into pytorch:mainAug 19, 2026
147 of 170 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@shoumikhin@lanluo-nvidia