GH-136895: Update JIT builds to use LLVM 20 by savannahostrowski · Pull Request #140329 · python/cpython

savannahostrowski · 2025-10-19T10:30:56Z

Alright, this took me longer than expected for two reasons:

Bumping to LLVM 20 required some changes to the infrastructure we use to grab LLVM as a dependency on Windows. As of 20, LLVM now contains files that exceed GitHub's allowable size, so checking these directly into cpython-bin-deps' tree was not an option. Instead, this PR adds the ability to pull from release artifacts (see https://github.com/python/cpython-bin-deps/releases/tag/llvm-20.1.8.0). This also makes the process of bumping LLVM for Windows a little less hairy. Thank you, @emmatyping, for much of this code, and thank you, @zware, for publishing the release artifact for me 🙏 !
For macOS x86_64 debug builds, external symbol references generated by LLVM 20 can exceed the ±2GB PC-relative addressing range that patch_32r requires. This manifested as assertion failures in free-threading tests (see https://github.com/savannahostrowski/cpython/actions/runs/18438327725/job/52537027963?pr=10 as an example). After going down a rabbit hole, I believe the correct fix here is to implement x86_64 trampolines (similar to our existing aarch64 implementation) to handle out-of-range symbols...and so I've done that. AFAICT, trampolines are only needed on macOS right now, as the other target platforms have different relocation mechanisms.

Issue: Update JIT builds to use LLVM 20 #136895

With LLVM 20, individual files are greater than the 100MiB single file limit for items checked into git. Therefore, this PR pulls down binaries from GitHub releases, as `.tar.xz` files to additionally maximize compression ratio. Currently this is somewhat of a first draft, as there are things like hash checking needed to be done.

This reverts commit e6450de.

markshannon

A bit of C pedantry. Otherwise looks good.

bedevere-app · 2025-10-30T12:31:15Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

brandtbucher

Awesome! Here's a partial review of everything except the stuff in Tools (I'd like to dig a bit more into the changes to the flags and relocations and see how they affect the code quality).

brandtbucher · 2025-10-30T15:18:17Z

+ /* Generate the trampoline (14 bytes, padded to 16):
+ 0: ff 25 00 00 00 00 jmp *(%rip)
+ 6: XX XX XX XX XX XX XX XX (64-bit target address)
+
+ Reference: https://wiki.osdev.org/X86-64_Instruction_Encoding#FF (JMP r/m64)
+ */
+ trampoline[0] = 0xFF;
+ trampoline[1] = 0x25;
+ *(uint32_t *)(trampoline + 2) = 0;
+ *(uint64_t *)(trampoline + 6) = value;


@savannahostrowski... just want to make sure it's not lost on you how badass you are hand-writing x86-64 machine code byte-by-byte like this. 🔥

markshannon

Looks good

emmatyping

This looks great! Very excited for this

savannahostrowski · 2025-11-03T18:01:28Z

I am going to merge this since it has multiple reviews and is blocking some other work that we'd like to do. @brandtbucher let's chat offline about any other changes we'd like to make here re. flags/relocations, happy to follow up!

Co-authored-by: Emma Harper Smith <emma@emmatyping.dev>

mcepl · 2026-03-25T09:59:19Z

Could this be backported to 3.14? We would really like to get rid of LLVM 19 in openSUSE/Factory.

…0329)

bedevere-app · 2026-03-25T13:44:35Z

GH-146420 is a backport of this pull request to the 3.14 branch.

…0329)

savannahostrowskiand others added 27 commits September 18, 2025 15:56

LLVM 20 bump
d145747

Merge main
d86b66e

Add flags for testing
9324b14

Merge branch 'main' into llvm-20
f3bb6b9

Merge main
b6e7981

Fix windows
e867644

Merge main
13e9f5b

Merge branch 'main' into llvm-20
84781b4

Add hash checking
cc98d30

Apply Emma's commits for grabbing binaries from release artifacts
76842eb

Merge branch 'main' into llvm-20
a732cec

Fix up LLVM via release artifacts
e8395ce

Merge branch 'main' into llvm-20
d1e4363

Remove model flags for x86_64 darwin causing GOT relocation issues
01aed67

Clean up
94f1a89

Only patch x86_64 GOT relocations when relaxation succeeds
e6450de

Revert "Only patch x86_64 GOT relocations when relaxation succeeds"
1adf827
This reverts commit e6450de.

mcmodel=large
b9bfacf

Add macro to handle debug
57c44ee

fno-pic
081ee86

remove fno-pic
0b773f9

remove hack
c78af6f

Trampoline attempt
d715cf2

Touch up and add better comments
38e11b9

Merge main
74eb9a4

More clean up
ca95652

savannahostrowski requested review from a team, brandtbucher and diegorusso as code owners October 19, 2025 10:30

markshannon requested changes Oct 30, 2025
View reviewed changes

Comment threadPython/jit.c Outdated
Comment threadPython/jit.c Outdated

bedevere-appBot removed the awaiting merge label Oct 30, 2025

bedevere-appBot added the awaiting changes label Oct 30, 2025

brandtbucher reviewed Oct 30, 2025
View reviewed changes

savannahostrowski added 6 commits October 30, 2025 09:10

Merge branch 'main' into llvm-20
0540ba3

Revert CI simplication and address comments for memcpy
618a4a6

Address comments about get_externals and trampoline helper
2e9ba92

Merge branch 'main' into llvm-20
a21cb31

Restore get_externals
795f466

Fix spacing
b880f26

markshannon self-requested a review October 31, 2025 10:25

markshannon approved these changes Oct 31, 2025
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting changes labels Oct 31, 2025

emmatyping approved these changes Oct 31, 2025
View reviewed changes

savannahostrowski merged commit 4e2ff4a into python:mainNov 3, 2025
74 of 75 checks passed

bedevere-appBot removed the awaiting merge label Nov 3, 2025

This was referenced Nov 3, 2025
[3.14] gh-140889: Test tailcall and JIT in CI (GH-140891) #140962
Merged
GH-140479: Update JIT builds to use LLVM 21 #140973
Merged

savannahostrowski deleted the llvm-20 branch December 4, 2025 22:59

StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
pythonGH-136895: Update JIT builds to use LLVM 20 (python#140329)
63a26cd
Co-authored-by: Emma Harper Smith <emma@emmatyping.dev>

mcepl added a commit to openSUSE-Python/cpython that referenced this pull request Mar 25, 2026
[3.14]pythonGH-136895: Update JIT builds to use LLVM 20 (pythonGH-14…
efbeaa5
…0329)

mcepl mentioned this pull request Mar 25, 2026
[3.14] GH-136895: Update JIT builds to use LLVM 21 (GH-140329) #146419
Closed

This comment was marked as outdated.
Sign in to view

mcepl mentioned this pull request Mar 25, 2026
[3.14] GH-136895: Update JIT builds to use LLVM 21 (GH-140329) #146420
Closed

mcepl added a commit to openSUSE-Python/cpython that referenced this pull request Mar 25, 2026
[3.14]pythonGH-136895: Update JIT builds to use LLVM 20 (pythonGH-14…
3db83aa
…0329)

Uh oh!

Uh oh!

Conversation

savannahostrowski commented Oct 19, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bedevere-appBot commented Oct 30, 2025

Uh oh!

brandtbucher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brandtbucherOct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

emmatyping left a comment

Choose a reason for hiding this comment

Uh oh!

savannahostrowski commented Nov 3, 2025

Uh oh!

Uh oh!

mcepl commented Mar 25, 2026

Uh oh!

This comment was marked as outdated.

bedevere-appBot commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

savannahostrowski commented Oct 19, 2025•
edited
Loading