Bring CI back to all-working condition by rwgk · Pull Request #5822 · pybind/pybind11

rwgk · 2025-09-02T17:08:44Z

Description

Fix "🐍 3 • windows-latest • mingw64" CI job
Apparently msys2/setup-msys2@v2 cannot be run twice anymore:
- https://github.com/pybind/pybind11/actions/runs/17394902023/job/49417376616?pr=5796

Run msys2/setup-msys2@v2
with:
msystem: mingw64
install: mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-scipy mingw-w64-x86_64-eigen3
path-type: minimal
update: false
pacboy: false
release: true
location: RUNNER_TEMP
platform-check-severity: fatal
cache: true
env:
PYTHONDEVMODE: 1
PIP_BREAK_SYSTEM_PACKAGES: 1
PIP_ONLY_BINARY: numpy
FORCE_COLOR: 3
PYTEST_TIMEOUT: 300
VERBOSE: 1
CMAKE_COLOR_DIAGNOSTICS: 1
MSYSTEM: MINGW64
Error: Trying to install MSYS2 to D:\a\_temp\msys64 but that already exists, cannot continue.

test_gil_scoped.py xfail macOS free-threading related
Disable build-ios job in tests-cibw.yml (commit 0f5a29d)
Remove macos_brew_install_llvm job because it started failing, to reduce our maintenance overhead:
Failures are tracked here: Bring CI back to all-working condition #5822 (comment)

Suggested changelog entry:

Placeholder.

…2@v2 cannot be run twice anymore): https://github.com/pybind/pybind11/actions/runs/17394902023/job/49417376616?pr=5796 ``` Run msys2/setup-msys2@v2 with: msystem: mingw64 install: mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-scipy mingw-w64-x86_64-eigen3 path-type: minimal update: false pacboy: false release: true location: RUNNER_TEMP platform-check-severity: fatal cache: true env: PYTHONDEVMODE: 1 PIP_BREAK_SYSTEM_PACKAGES: 1 PIP_ONLY_BINARY: numpy FORCE_COLOR: 3 PYTEST_TIMEOUT: 300 VERBOSE: 1 CMAKE_COLOR_DIAGNOSTICS: 1 MSYSTEM: MINGW64 Error: Trying to install MSYS2 to D:\a\_temp\msys64 but that already exists, cannot continue. ```

…tion

rwgk · 2025-09-02T22:19:21Z

Hi @henryiii, could you please review? This should stop the frequent CI / 🐍 (macos-13, 3.13t, -DCMAKE_CXX_STANDARD=11) / 🧪 (pull_request) failures.

henryiii · 2025-09-02T22:47:02Z

+ if env.WIN and env.PYPY:
+ pytest.xfail("[TEST-GIL-SCOPED] Windows PyPy: " + msg)
+ elif env.MACOS and not env.sys_is_gil_enabled():
+ pytest.xfail("[TEST-GIL-SCOPED] macOS free-threading: " + msg)


This doesn't always fail, does it? So strict=False?

The strict flag only applies to @pytest.mark.xfail(...) (the decorator/mark form). Inline pytest.xfail() doesn't take strict at all — it immediately aborts the test and reports XFAIL. There’s no concept of XPASS in this case, so strict isn't relevant here.
To be sure I looked it up the good-old way:
https://docs.pytest.org/en/stable/how-to/skipping.html#xfail-mark-test-functions-as-expected-to-fail
Note that no other code is executed after the pytest.xfail() call, differently from the marker.

henryiii · 2025-09-02T22:47:18Z

- if env.PYPY and env.WIN:
- pytest.skip(msg)
+ if env.WIN and env.PYPY:
+ pytest.xfail("[TEST-GIL-SCOPED] Windows PyPy: " + msg)


This didn't run before, now it will. Guessing that's fine?

See other comment: Note that no other code is executed after the pytest.xfail() call, differently from the marker.

henryiii · 2025-09-02T22:53:53Z

+ install_mingw64_only: ""
+ - sys: mingw64
+ env: x86_64
+ install_mingw64_only: |


I'd name this "extra_install", I think, since it's there to install extra stuff. And you can leave it off on the other job, variables default to empty.

Done: commit bb8a794

rwgk · 2025-09-02T23:30:35Z

Thanks @henryiii!

WDYT about disabling the two failing CIBW tests in this PR, too, until someone gets a chance to fix those?

CIBW / iOS wheel macos-13 (pull_request)
CIBW / iOS wheel macos-13 (pull_request)Failing after 19s

CIBW / iOS wheel macos-14 (pull_request)
CIBW / iOS wheel macos-14 (pull_request)Failing after 11s

rwgk · 2025-09-03T00:11:29Z

Ugh...

It is not working as intended:

🐍 (macos-13, 3.13t, -DCMAKE_CXX_STANDARD=11) / 🧪

 if process.exitcode is None:
assert t_delta > 0.9 * timeout
msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
if env.WIN and env.PYPY:
pytest.xfail("[TEST-GIL-SCOPED] Windows PyPy: " + msg)
elif env.MACOS and not env.sys_is_gil_enabled():
pytest.xfail("[TEST-GIL-SCOPED] macOS free-threading: " + msg)
> raise RuntimeError(msg)
E RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

@henryiii I'm a bit lost, should I maybe checking for env.PY_GIL_DISABLED here?

IIUC, it's failing even though the GIL is enabled? This has been happening with macOS ever since I added this test, but only fairly infrequently.

rwgk · 2025-09-03T06:17:13Z

For easy future reference, the errors after commit bd3900e were:

CIBW / iOS wheel macos-13 (pull_request) Failing after 1m

CIBW / iOS wheel macos-14 (pull_request) Failing after 35s

macos-13:

 + python -m build /Users/runner/work/pybind11/pybind11/tests --wheel --outdir=/private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/built_wheel
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- scikit-build-core
* Getting build dependencies for wheel...
* Building wheel...
2025-09-03 05:08:04,030 - scikit_build_core - INFO - RUN: /private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/build/venv/cibw_xbuild_tools/cmake -E capabilities
2025-09-03 05:08:04,053 - scikit_build_core - INFO - CMake version: 3.31.6
*** scikit-build-core 0.11.6 using CMake 3.31.6 (wheel)
2025-09-03 05:08:04,054 - scikit_build_core - INFO - Implementation: cpython ios on x86_64
2025-09-03 05:08:04,055 - scikit_build_core - INFO - Build directory: /private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmpx_swhd3d/build
*** Configuring CMake...
2025-09-03 05:08:04,089 - scikit_build_core - INFO - RUN: /private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/build/venv/cibw_xbuild_tools/ninja --version
2025-09-03 05:08:04,094 - scikit_build_core - INFO - Ninja version: 1.13.1
2025-09-03 05:08:04,097 - scikit_build_core - INFO - RUN: /private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/build/venv/cibw_xbuild_tools/cmake -S. -B/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmpx_swhd3d/build -DCMAKE_BUILD_TYPE:STRING=Release -C/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmpx_swhd3d/build/CMakeInit.txt -DCMAKE_INSTALL_PREFIX=/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmpx_swhd3d/wheel/platlib -DCMAKE_MAKE_PROGRAM=/private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/build/venv/cibw_xbuild_tools/ninja -DPYBIND11_FINDPYTHON=TRUE
CMake Error: Could not find CMAKE_ROOT !!!
CMake has most likely not been installed correctly.
Modules directory not found in
CMake Error: Error executing cmake::LoadCache(). Aborting.
*** CMake configuration failed
ERROR Backend subprocess exited when trying to invoke build_wheel

macos-14: very similar

This reverts commit bd3900e. See also: pybind#5822 (comment)

rwgk · 2025-09-03T07:20:11Z

This is flaking the second time (this PR):

CI / macos-13 • brew install llvm (pull_request) Failing after 10m (start time: Wed, 03 Sep 2025 06:38:28 GMT)

I'll try a rerun.

For completeness, the first flake was:

https://github.com/pybind/pybind11/actions/runs/17422859619/job/49464452403 (start time: Wed, 03 Sep 2025 04:21:04 GMT)

There was one success between the 1st and 2nd failure (https://github.com/pybind/pybind11/actions/runs/17423607165/job/49466476407, start time: Wed, 03 Sep 2025 05:07:40 GMT).

Edit: 3rd failure now: https://github.com/pybind/pybind11/actions/runs/17425182772/job/49473499749?pr=5822

Edit: I added that job with PR #4326 — initially to inform PR #4319, but then I decided to keep it, to be proactive about testing clang development versions. I'm not sure we still have the manpower to be proactive in this way.

Edit: 4th failure now: CI / macos-13 • brew install llvm (pull_request)

…uce our maintenance overhead: Failures tracked here: pybind#5822 (comment)

Replaced brew upgrade with brew install for cmake.

Uninstall cmake before installing the latest version due to GitHub's local tap changes.

henryiii · 2025-09-03T14:02:22Z

GitHub is unpinning cmake in the next runner release: actions/runner-images#12935, so hopefully this entire step will be not needed. The target date is in around a week, I think.

rwgk · 2025-09-03T16:06:24Z

Thanks @henryiii, awesome that you could get the CIBW jobs working again with a one-line change!

I'll mark my calendar to harvest the TEST-GIL-SCOPEDxfail tags from the CI logs in a month or so, to see where we stand.

rwgk requested a review from henryiii as a code owner September 2, 2025 17:08

rwgk added 2 commits September 2, 2025 13:41

Add pytest.xfail("[TEST-GIL-SCOPED] macOS free-threading...)
61d7577

Change env.SYS_IS_GIL_ENABLED constant to env.sys_is_gil_enabled func…
25b4c33
…tion

rwgk changed the title ~~Fix "🐍 3 • windows-latest • mingw64" CI job~~Fix "🐍 3 • windows-latest • mingw64" CI job; test_gil_scoped.py xfail macOS free-threadingSep 2, 2025

henryiii reviewed Sep 2, 2025
View reviewed changes

Change install_mingw64_only → extra_install
bb8a794

rwgk added 2 commits September 2, 2025 21:16

Also xfail if macOS and PY_GIL_DISABLED, show SOABI
13c1c14

build-ios: brew upgrade|install cmake
bd3900e

rwgk added 2 commits September 2, 2025 23:18

Revert "build-ios: brew upgrade|install cmake"
bd75761
This reverts commit bd3900e. See also: pybind#5822 (comment)

Disable build-ios job in tests-cibw.yml
0f5a29d

Remove macos_brew_install_llvm job because it started failing, to red…
e70ac7f
…uce our maintenance overhead: Failures tracked here: pybind#5822 (comment)

rwgk changed the title ~~Fix "🐍 3 • windows-latest • mingw64" CI job; test_gil_scoped.py xfail macOS free-threading~~Get CI back in all-working conditionSep 3, 2025

rwgk changed the title ~~Get CI back in all-working condition~~Bring CI back to all-working conditionSep 3, 2025

henryiii added 2 commits September 3, 2025 09:54

Fix iOS build step for cmake installation
4de21d8
Replaced brew upgrade with brew install for cmake.

Update cmake installation steps in CI workflow
38e9fa7
Uninstall cmake before installing the latest version due to GitHub's local tap changes.

henryiii reviewed Sep 3, 2025
View reviewed changes

Comment thread.github/workflows/tests-cibw.yml Outdated

Update .github/workflows/tests-cibw.yml
26496bb

rwgk merged commit cd56888 into pybind:masterSep 3, 2025
85 checks passed

github-actionsBot added the needs changelog Possibly needs a changelog entry label Sep 3, 2025

rwgk removed the needs changelog Possibly needs a changelog entry label Sep 3, 2025

rwgk deleted the fix-windows-latest-mingw64-ci-job branch September 3, 2025 17:21

This was referenced Sep 4, 2025
NOT FOR MERGE -- testing to see if windows-latest jobs are broken #5825
Closed
Fix crash in gil_scoped_acquire#5828
Merged

Uh oh!

Conversation

rwgk commented Sep 2, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Suggested changelog entry:

Uh oh!

rwgk commented Sep 2, 2025

Uh oh!

henryiiiSep 2, 2025

Choose a reason for hiding this comment

Uh oh!

rwgkSep 2, 2025

Choose a reason for hiding this comment

Uh oh!

henryiiiSep 2, 2025

Choose a reason for hiding this comment

Uh oh!

rwgkSep 2, 2025

Choose a reason for hiding this comment

Uh oh!

henryiiiSep 2, 2025

Choose a reason for hiding this comment

Uh oh!

rwgkSep 2, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk commented Sep 2, 2025

Uh oh!

rwgk commented Sep 3, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rwgk commented Sep 3, 2025

Uh oh!

rwgk commented Sep 3, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

henryiii commented Sep 3, 2025

Uh oh!

rwgk commented Sep 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rwgk commented Sep 2, 2025•
edited
Loading

rwgk commented Sep 3, 2025•
edited
Loading

rwgk commented Sep 3, 2025•
edited
Loading