gh-116167: Allow disabling the GIL with PYTHON_GIL=0 or -X gil=0

[swtaarrs]

In free-threaded builds, running with PYTHON_GIL=0 or -X gil=0 will now disable the GIL. #116322 and #116329 track follow-up work to re-enable the GIL when loading an incompatible extension, and to disable the GIL by default, respectively.

In order to support re-enabling the GIL at runtime, all GIL-related data structures are initialized as usual, and disabling the GIL simply sets a flag at startup that causes take_gil() and drop_gil() to return early.

With PYTHON_GIL=0 set, I spot-checked a few tests and small programs that don't use threads. They all seem to run fine, and very basic threaded programs work, sometimes. Trying to run the full test suite crashes pretty quickly, in test_asyncio.

• Issue: Add a mechanism to disable the GIL #116167

[swtaarrs]

!buildbot nogil

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit ac24233 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

• x86-64 MacOS Intel ASAN NoGIL PR
• x86-64 MacOS Intel NoGIL PR
• ARM64 MacOS M1 Refleaks NoGIL PR
• ARM64 MacOS M1 NoGIL PR
• AMD64 Ubuntu NoGIL Refleaks PR
• AMD64 Ubuntu NoGIL PR
• AMD64 Windows Server 2022 NoGIL PR

[swtaarrs]

!buildbot nogil

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit 9e03999 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

• x86-64 MacOS Intel ASAN NoGIL PR
• ARM64 MacOS M1 Refleaks NoGIL PR
• AMD64 Windows Server 2022 NoGIL PR
• AMD64 Ubuntu NoGIL PR
• x86-64 MacOS Intel NoGIL PR
• AMD64 Ubuntu NoGIL Refleaks PR
• ARM64 MacOS M1 NoGIL PR

[corona10]

Good to add test case in this file too.

cpython/Lib/test/test_cmd_line.py

Line 896 in 72714c0

deftest_int_max_str_digits(self):

[corona10]

Ah ignore above comments, this PR only referencing the environment variable so not proper for this case.

You can just add the testcase for the environment variable, still worth to adding it.

[corona10]

At not for this PR, since this feature is not officially announced, but in some moment, you should write about the environment variable at https://github.com/python/cpython/blob/72714c0266ce6d39c7c7fb63f617573b8f5a3cb2/Doc/using/cmdline.rst to get user feedback.

[swtaarrs]

!buildbot nogil

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit 4f43a87 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

• x86-64 MacOS Intel ASAN NoGIL PR
• ARM64 MacOS M1 NoGIL PR
• AMD64 Ubuntu NoGIL PR
• x86-64 MacOS Intel NoGIL PR
• ARM64 MacOS M1 Refleaks NoGIL PR
• AMD64 Ubuntu NoGIL Refleaks PR
• AMD64 Windows Server 2022 NoGIL PR

[swtaarrs]

you should write about the environment variable ... to get user feedback

What would be a good time to do that? Maybe once the GIL is disabled by default, so the description doesn't have to change?

[colesbury]

I would document the environment variable now, when it's added. For reference, we already document the --disable-gil configure flag.

I don't think it needs to be extensive, but should cover:

• Only available in free-threaded builds
• 0 forces the GIL to be disabled
• 1 forces the GIL to be enabled
• New in version 3.13

The behavior when then environment variable is not set is what's going to change in the future. We can document that later.

[swtaarrs]

After discussion in Discord, I'm also going to add a -X option to go with the environment variable.

[ericsnowcurrently]

LGTM

I've left one minor comment about an unclear test. It really isn't a blocker, as the test is passing, but it would still be nice to have a response.

[vstinner]

This is both exciting and very scary. Congrats @swtaarrs and @colesbury :-)