Update buildbot to use XCframework build script. by freakboy3742 · Pull Request #691 · python/buildmaster-config

freakboy3742 · 2026-03-25T15:17:39Z

The iOS XCframework build script will shortly be moved to Platforms/iOS.

This updates the Buildbot config to use the new XCframework buildbot in "slow" mode for 3.15+ builds (ensuring a full, multi-architecture XCframework is built and tested.

~~I haven't pushed the corresponding CPython PR yet; this is to explore options for branch-based configuration of the build script.~~

As I currently understand it, this will work for buildbot runs that are triggered on a merge of a PR on main, 3.13, 3.14, or the 3.15 branch (when it is forked). A PR build will succeed on main and 3.15 (when it is forked). But a PR build based off a 3.13/3.14 branch will fail (because there's currently no way to identify the base branch of a PR job).

python/cpython#146497 has the changes necessary for this build script to work.

It introduces a symlink for 3.14 compatibility. It falls back to the old build script for builds explicitly on the 3.13 branch. PR builds based off a 3.13 branch will fail as the XCframework script wasn't introduced until 3.14, and there's not currently any way to identify the base branch for a "custom" PR build.

vstinner · 2026-03-26T00:55:57Z

But a PR build based off a 3.13/3.14 branch will fail (because there's currently no way to identify the base branch of a PR job).

Correct. IMO it's an acceptable trade-off. I don't see any simple solution using buildbot API to use different steps depending on an actual check run on the worker (ex: check if Platforms/Apple exists).

SetPropertyFromCommand() and util.Interpolate() can only be used to pass a string to a command, not really to use very different steps and commands (as you do in this PR).

encukou · 2026-03-26T12:47:21Z

It's probably not worth worrying about 3.13.

For 3.14, maybe do something like [ -e Platforms/Apple ] || (mkdir -p Platforms; ln -s ../Apple Platforms/Apple) for the custom?

freakboy3742 · 2026-03-26T21:51:23Z

It's probably not worth worrying about 3.13.

👍

For 3.14, maybe do something like [ -e Platforms/Apple ] || (mkdir -p Platforms; ln -s ../Apple Platforms/Apple) for the custom?

Good idea - I'll push an update shortly.

freakboy3742 · 2026-03-27T01:09:23Z

+ [
+ ShellCommand(
+ name="Set up compatibility symlink (will fail on <= 3.13 branches)",
+ command="[ -e Platforms/Apple ] || ln -s ../Apple Platforms/Apple",


The docs say single-string format should work...
The Platforms folder exists on 3.14; it doesn't on 3.13 - so this command will fail early on a 3.13-based build.

vstinner

LGTM. python/cpython#146497 landed in the Python main branch, so this PR is ready to be merged, right? Just one coding style suggestion :-)

vstinner · 2026-03-27T10:55:51Z

+ self.addSteps(
+ [


Coding style suggestion: you may move [ after addSteps( to remove one indentation level.
Suggested change
self.addSteps(
[
self.addSteps([

freakboy3742 · 2026-03-29T23:30:34Z

LGTM. python/cpython#146497 landed in the Python main branch, so this PR is ready to be merged, right?

Correct. I'll wait until I know there's someone around who can poke the buildbot in case something goes wrong.

Just one coding style suggestion :-)

Fixed.

vstinner · 2026-03-30T08:40:06Z

I merged the change and deployed manually the change. I forced a new build on the iOS 3.x worker: https://buildbot.python.org/#/builders/1380/builds/6193

Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/buildbot/buildarea/3.x.rkm-arm64-ios-simulator.iOS-simulator.arm64/build/Platforms/Apple/__main__.py", line 57, in <module>
ArgsT = Sequence[str | Path]
TypeError: unsupported operand type(s) for |: 'type' and 'type'

Ooops, the worker uses Python 3.9 and Platforms/Apple/ uses Python 3.10 type annotations.

I wrote python/cpython#146624 to use Python 3.9 type annotations: use typing.Union.

freakboy3742 · 2026-03-30T09:32:05Z

Oops indeed... 🤦

Python 3.9 is the version that comes with Xcode; I'll make a note that when I update the buildbot to macOS 26, I also need to fix the Python setup.

vstinner · 2026-03-30T09:42:37Z

New build, new error: https://buildbot.python.org/#/builders/1380/builds/6195

Traceback (mostrecentcalllast):
File"/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line197, in_run_module_as_mainreturn_run_code(code, main_globals, None,
File"/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line87, in_run_codeexec(code, run_globals)
File"/Users/buildbot/buildarea/3.x.rkm-arm64-ios-simulator.iOS-simulator.arm64/build/Platforms/Apple/testbed/__main__.py", line436, in<module>main()
File"/Users/buildbot/buildarea/3.x.rkm-arm64-ios-simulator.iOS-simulator.arm64/build/Platforms/Apple/testbed/__main__.py", line387, inmainclone_testbed(
File"/Users/buildbot/buildarea/3.x.rkm-arm64-ios-simulator.iOS-simulator.arm64/build/Platforms/Apple/testbed/__main__.py", line182, inclone_testbedframework.relative_to(xc_framework_path.parent, walk_up=True)
TypeError: relative_to() gotanunexpectedkeywordargument 'walk_up

walk_up parameter was added to Python 3.12.

freakboy3742 · 2026-03-30T10:16:01Z

I've now updated the iOS buildbot to use Python 3.12; that has unstuck the build, but it's now hitting a separate problem in the build.

vstinner · 2026-03-30T11:15:09Z

Follow-up fix: python/cpython#146628.

vstinner · 2026-03-30T19:24:40Z

3.13, 3.14 and 3.x buildbot workers are back to green.

vstinner · 2026-04-01T15:53:07Z

In the server logs, I started to see Got duplication connection from 'rkm-arm64-ios-simulator' errors (I replaced the IPv4 address with WORKER_IP for privacy):

2026-04-01 15:49:36+0000 [Broker,51,WORKER_IP] worker 'rkm-arm64-ios-simulator' attaching from IPv4Address(type='TCP', host='WORKER_IP', port=61515)
2026-04-01 15:49:36+0000 [Broker,51,WORKER_IP] Got duplication connection from 'rkm-arm64-ios-simulator' starting arbitration procedure
2026-04-01 15:49:36+0000 [Broker,9,WORKER_IP] Peer will receive following PB traceback:
2026-04-01 15:49:36+0000 [Broker,9,WORKER_IP] Unhandled Error
Traceback (most recent call last):
File "/srv/buildbot/venv/lib/python3.13/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
result = context.run(
File "/srv/buildbot/venv/lib/python3.13/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
return g.throw(self.value.with_traceback(self.tb))
File "/srv/buildbot/venv/lib/python3.13/site-packages/buildbot/worker/protocols/manager/pb.py", line 67, in requestAvatar
persp = yield afactory(mind, avatarIdStr)
File "/srv/buildbot/venv/lib/python3.13/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
result = context.run(
File "/srv/buildbot/venv/lib/python3.13/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
return g.throw(self.value.with_traceback(self.tb))
File "/srv/buildbot/venv/lib/python3.13/site-packages/buildbot/worker/protocols/base.py", line 67, in _create_connection
accepted = yield self.master.workers.newConnection(conn, workerName)
File "/srv/buildbot/venv/lib/python3.13/site-packages/twisted/internet/defer.py", line 1857, in _inlineCallbacks
result = context.run(gen.send, result)
File "/srv/buildbot/venv/lib/python3.13/site-packages/buildbot/worker/manager.py", line 131, in newConnection
raise RuntimeError("rejecting duplicate worker")
builtins.RuntimeError: rejecting duplicate worker

@freakboy3742: Any idea what's going on?

freakboy3742 · 2026-04-02T00:18:31Z

In the server logs, I started to see Got duplication connection from 'rkm-arm64-ios-simulator' errors (I replaced the IPv4 address with WORKER_IP for privacy):
...
@freakboy3742: Any idea what's going on?

I'm not exactly sure how it happened, but it looks like a second iOS buildbot was started (or tried to start). This might have been related to switching over to the 3.12 venv... but I don't remember seeing those errors at the time.

I think I've cleaned it up now (at least, I'm not seeing any more message in the logs about duplicate workers). I'll keep an eye on it; let me know if you spot any more problems.

vstinner · 2026-04-02T00:33:10Z

Thanks. The "rejecting duplicate worker" error is gone :-)

vstinner · 2026-04-03T12:57:38Z

@freakboy3742: And the "rejecting duplicate worker" (rkm-arm64-ios-simulator) error is back. Maybe you have again two clients running in parallel?

freakboy3742 · 2026-04-04T03:17:50Z

@vstinner I'm really confused about how this is happening...

Looking at the logs, something happened at around 06:17 UTC on Thursday:

2026-04-02 06:17:58+0000 [-] sending app-level keepalive
2026-04-02 06:17:58+0000 [Broker,client] Master replied to keepalive, everything's fine
2026-04-02 06:27:58+0000 [-] sending app-level keepalive
2026-04-02 06:27:58+0000 [Broker,client] Master replied to keepalive, everything's fine
2026-04-02 06:37:58+0000 [Broker,client] message from master: keepalive
2026-04-02 06:37:58+0000 [-] sending app-level keepalive
2026-04-02 06:37:58+0000 [Broker,client] Master replied to keepalive, everything's fine
2026-04-02 06:46:31+0000 [-] Scheduling retry 1 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x103d14530> in 2.1002453177953577 seconds.
2026-04-02 06:46:31+0000 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:33+0000 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:33+0000 [-] Scheduling retry 2 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x103d14530> in 2.8555260446733755 seconds.
2026-04-02 06:46:33+0000 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:36+0000 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:36+0000 [-] Scheduling retry 3 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x103d14530> in 3.782957902816462 seconds.
2026-04-02 06:46:36+0000 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:40+0000 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:40+0000 [-] Scheduling retry 4 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x103d14530> in 5.296599807564303 seconds.
2026-04-02 06:46:40+0000 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:45+0000 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:45+0000 [-] Scheduling retry 5 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x103d14530> in 8.493914382457785 seconds.
2026-04-02 06:46:45+0000 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:53+0000 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:46:53+0000 [-] Scheduling retry 6 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x103d14530> in 11.433260018340917 seconds.
2026-04-02 06:46:53+0000 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:47:05+0000 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:47:05+0000 [-] Scheduling retry 7 to connect <twisted.internet.endpoints.TCP4ClientEndpoint object at 0x103d14530> in 17.60820861487915 seconds.
2026-04-02 06:47:05+0000 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:47:22+0000 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x103d14350>
2026-04-02 06:47:23+0000 [Broker,client] message from master: attached
2026-04-02 06:47:23+0000 [Broker,client] message from master: attached
2026-04-02 06:47:23+0000 [Broker,client] message from master: attached
2026-04-02 06:47:23+0000 [Broker,client] message from master: attached
2026-04-02 06:47:23+0000 [Broker,client] message from master: attached
2026-04-02 06:47:23+0000 [Broker,client] Connected to buildmaster; worker is ready
2026-04-02 06:47:23+0000 [Broker,client] sending application-level keepalives every 600 seconds
2026-04-02 06:47:24+0000 [Broker,client] message from master: master got a duplicate connection
2026-04-02 06:47:25+0000 [Broker,client] message from master: master got a duplicate connection
2026-04-02 06:47:34+0000 [Broker,client] message from master: master got a duplicate connection
2026-04-02 06:47:45+0000 [Broker,client] message from master: master got a duplicate connection
...

I had to manually restart the buildbot at around 04:37 to correct a different error that started happening (some really odd file permission errors), but the buildbot came back online without any problems (as far as I can tell). It seemed to run happily until this 06:47 timestamp.

That looks like it might coincide with the master server restart mentioned in Discord... but I can't explain why that would have caused a problem (or, more specifically, why it was only a problem with the iOS buildbot).

When I logged in today and tried a buildbot restart, it told me that no buildbot was running... but there clearly was. There wasn't a PID file, though; it wasn't until I manually killed the server and it auto-started using the macOS service that a PID file was written.

So... it looks like it's restarted cleanly again... but I'm no wiser as to why this is happening.

Update buildbot to use XCframework build script.
bc5a3ee

freakboy3742 mentioned this pull request Mar 26, 2026
gh-146444: Move the Apple folder to the Platforms directory python/cpython#146497
Merged

Add a symlink for 3.14 compatibility.
1ee9d59

freakboy3742 force-pushed the iOS-platforms-refactor branch from 39e0ec3 to 1ee9d59Compare March 27, 2026 01:06

freakboy3742 commented Mar 27, 2026
View reviewed changes

freakboy3742 marked this pull request as ready for review March 27, 2026 03:46

freakboy3742 requested review from encukou and vstinner March 27, 2026 03:46

Shorten label for linting.
2de77fd

freakboy3742 force-pushed the iOS-platforms-refactor branch from 87b6799 to 2de77fdCompare March 27, 2026 03:47

vstinner reviewed Mar 27, 2026
View reviewed changes

Comment threadmaster/custom/factories.py Outdated

vstinner reviewed Mar 27, 2026
View reviewed changes

Comment threadmaster/custom/factories.py Outdated

freakboy3742 added 2 commits March 27, 2026 16:59

Refactor to make the 3.13 and 3.14+ branches more distinct.
4fef73c

Remove some ruff formatting.
e0423a8

freakboy3742 requested a review from vstinner March 27, 2026 09:09

vstinner reviewed Mar 27, 2026
View reviewed changes

encukou approved these changes Mar 27, 2026
View reviewed changes

Code formatting tweak.
600e004

vstinner merged commit ac55571 into python:mainMar 30, 2026
1 check passed

vstinner mentioned this pull request Mar 30, 2026
gh-146444: Make Platforms/Apple/ compatible with Python 3.9 python/cpython#146624
Merged

freakboy3742 deleted the iOS-platforms-refactor branch March 30, 2026 09:29

Uh oh!

Uh oh!

Conversation

freakboy3742 commented Mar 25, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Mar 26, 2026

Uh oh!

encukou commented Mar 26, 2026

Uh oh!

freakboy3742 commented Mar 26, 2026

Uh oh!

freakboy3742Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinnerMar 27, 2026

Choose a reason for hiding this comment

Uh oh!

freakboy3742 commented Mar 29, 2026

Uh oh!

Uh oh!

vstinner commented Mar 30, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

freakboy3742 commented Mar 30, 2026

Uh oh!

vstinner commented Mar 30, 2026

Uh oh!

freakboy3742 commented Mar 30, 2026

Uh oh!

vstinner commented Mar 30, 2026

Uh oh!

vstinner commented Mar 30, 2026

Uh oh!

vstinner commented Apr 1, 2026

Uh oh!

freakboy3742 commented Apr 2, 2026

Uh oh!

vstinner commented Apr 2, 2026

Uh oh!

vstinner commented Apr 3, 2026

Uh oh!

freakboy3742 commented Apr 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

freakboy3742 commented Mar 25, 2026•
edited
Loading

vstinner commented Mar 30, 2026•
edited
Loading