Skip to content

gh-116322: Enable the GIL while loading C extension modules - #118560

Merged
colesbury merged 9 commits into
python:mainfrom
swtaarrs:cpython-enable-gil
May 7, 2024
Merged

gh-116322: Enable the GIL while loading C extension modules#118560
colesbury merged 9 commits into
python:mainfrom
swtaarrs:cpython-enable-gil

Conversation

@swtaarrs

@swtaarrsswtaarrs commented May 3, 2024•
edited
Loading

Copy link
Copy Markdown
Member

Building on gh-116882, this PR adds the ability to enable/disable the GIL at runtime, and uses that in the C module loading code.

We can't know before running a module init function if it supports free-threading, so the GIL is temporarily enabled before doing so. If the module declares support for running without the GIL, the GIL is later disabled. Otherwise, the GIL is permanently enabled, and will never be disabled again for the life of the current interpreter.

@swtaarrsswtaarrs added interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading labels May 3, 2024
@swtaarrs
swtaarrs requested a review from colesburyMay 3, 2024 17:48
@bedevere-appbedevere-appBot mentioned this pull request May 3, 2024
Closed
@swtaarrs

swtaarrs commented May 3, 2024

Copy link
Copy Markdown
MemberAuthor

!buildbot nogil

@bedevere-bot

bedevere-bot commented May 3, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit 637c3e5 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

  • x86-64 MacOS Intel ASAN NoGIL PR
  • AMD64 Ubuntu NoGIL Refleaks PR
  • ARM64 MacOS M1 NoGIL PR
  • AMD64 Windows Server 2022 NoGIL PR
  • AMD64 Ubuntu NoGIL PR
  • ARM64 MacOS M1 Refleaks NoGIL PR
  • x86-64 MacOS Intel NoGIL PR

@swtaarrs
swtaarrs marked this pull request as ready for review May 3, 2024 19:21
@swtaarrs

swtaarrs commented May 3, 2024

Copy link
Copy Markdown
MemberAuthor

I have a local commit to update sys._is_gil_enabled() to use the new function I added (_PyEval_IsGILEnabled()) but I'll wait until the buildbots finish to push it.

colesbury
colesbury reviewed May 3, 2024
View reviewed changes

@colesburycolesbury left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty good to me. Some comments below.

We should update sys._is_gil_enabled as part of this PR.

Comment threadPython/pystate.c Outdated
Comment threadPython/ceval_gil.c Outdated
Comment threadPython/importdl.c Outdated
colesbury
colesbury approved these changes May 4, 2024
View reviewed changes

@colesburycolesbury left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ericsnowcurrently
ericsnowcurrently reviewed May 4, 2024
View reviewed changes

@ericsnowcurrentlyericsnowcurrently left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some questions.

Comment threadInclude/internal/pycore_import.h Outdated
Comment threadPython/import.c Outdated
Comment threadPython/import.c
Comment threadPython/importdl.c Outdated
swtaarrs
swtaarrs commented May 6, 2024
View reviewed changes
Comment threadPython/import.c Outdated
ericsnowcurrently
ericsnowcurrently approved these changes May 6, 2024
View reviewed changes

@ericsnowcurrentlyericsnowcurrently left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one small comment. Otherwise LGTM.

(There is stuff to follow up on though.)

Comment threadPython/import.c
@colesbury
colesbury merged commit 853163d into python:mainMay 7, 2024
SonicField pushed a commit to SonicField/cpython that referenced this pull request May 8, 2024
@swtaarrs@SonicField
pythongh-116322: Enable the GIL while loading C extension modules (py…
0d76d9c
…thon#118560)
Add the ability to enable/disable the GIL at runtime, and use that in
the C module loading code.
We can't know before running a module init function if it supports
free-threading, so the GIL is temporarily enabled before doing so. If
the module declares support for running without the GIL, the GIL is
later disabled. Otherwise, the GIL is permanently enabled, and will
never be disabled again for the life of the current interpreter.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrentlyericsnowcurrentlyericsnowcurrently approved these changes
@colesburycolesburycolesbury approved these changes
@kumaraditya303kumaraditya303Awaiting requested review from kumaraditya303kumaraditya303 is a code owner
@brettcannonbrettcannonAwaiting requested review from brettcannonbrettcannon is a code owner
@ncoghlanncoghlanAwaiting requested review from ncoghlanncoghlan is a code owner
@warsawwarsawAwaiting requested review from warsawwarsaw is a code owner

Assignees

No one assigned

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@swtaarrs@bedevere-bot@colesbury@ericsnowcurrently