gh-128691: Use deferred reference counting on _thread._local

[colesbury]

This change, along with the LOAD_ATTR specializations, makes the "thread_local_read" micro benchmark in Tools/ftscalingbench/ftscalingbench.py scale well to multiple threads.

• Issue: threading.local() access should scale well from multiple threads #128691

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x has failed when building commit c141748.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/345/builds/9906) and take a look at the build logs.
4. Check if the failure is related to this commit (c141748) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/345/builds/9906

Failed tests:

• test.test_multiprocessing_spawn.test_misc

Failed subtests:

• test_large_pool - test.test_multiprocessing_forkserver.test_misc.MiscTestCase.test_large_pool
• test_large_pool - test.test_multiprocessing_spawn.test_misc.MiscTestCase.test_large_pool

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
File "<string>", line 1, in <module>from multiprocessing.forkserver import main; main(10, 11, ['__main__'], **{'sys_path': ['/root/buildarea/3.x.angelico-debian-amd64/build/target/lib/python314.zip', '/root/buildarea/3.x.angelico-debian-amd64/build/Lib', '/root/buildarea/3.x.angelico-debian-amd64/build/build/lib.linux-x86_64-3.14'], 'authkey_r': 13})
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 324, in main
pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/build/test_python_2151385æ/@test_2151385_tmpæ", line 4, in <module>with multiprocessing.Pool(200) as p:
~~~~~~~~~~~~~~~~~~~~^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 119, in Poolreturn Pool(processes, initializer, initargs, maxtasksperchild,
context=self.get_context())
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 215, in __init__self._repopulate_pool()
~~~~~~~~~~~~~~~~~~~~~^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 306, in _repopulate_poolreturnself._repopulate_pool_static(self._ctx, self.Process,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^self._processes,
^^^^^^^^^^^^^^^^...<3 lines>...self._maxtasksperchild,
^^^^^^^^^^^^^^^^^^^^^^^self._wrap_exception)
^^^^^^^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 329, in _repopulate_pool_static
w.start()
~~~~~~~^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line 121, in startself._popen =self._Popen(self)
~~~~~~~~~~~^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 300, in _Popenreturn Popen(process_obj)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 35, in __init__super().__init__(process_obj)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 20, in __init__self._launch(process_obj)
~~~~~~~~~~~~^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 59, in _launchself.pid = forkserver.read_signed(self.sentinel)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 390, in read_signedraiseEOFError('unexpected EOF')
EOFError: unexpected EOF
---
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 6607, in test_large_pool
rc, out, err = script_helper.assert_python_ok(testfn)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 182, in assert_python_okreturn _assert_python(True, *args, **env_vars)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 167, in _assert_python
res.fail(cmd_line)
~~~~~~~~^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 80, in failraiseAssertionError(f"Process return code is {exitcode}\n"...<10 lines>...f"---")
AssertionError: Process return code is 1
command line: ['/root/buildarea/3.x.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '@test_2151385_tmpæ']
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 6607, in test_large_pool
rc, out, err = script_helper.assert_python_ok(testfn)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 182, in assert_python_okreturn _assert_python(True, *args, **env_vars)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 167, in _assert_python
res.fail(cmd_line)
~~~~~~~~^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 80, in failraiseAssertionError(f"Process return code is {exitcode}\n"...<10 lines>...f"---")
AssertionError: Process return code is 1
command line: ['/root/buildarea/3.x.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '@test_2126709_tmpæ']
Traceback (most recent call last):
File "<string>", line 1, in <module>from multiprocessing.forkserver import main; main(10, 11, ['__main__'], **{'sys_path': ['/root/buildarea/3.x.angelico-debian-amd64/build/target/lib/python314.zip', '/root/buildarea/3.x.angelico-debian-amd64/build/Lib', '/root/buildarea/3.x.angelico-debian-amd64/build/build/lib.linux-x86_64-3.14'], 'authkey_r': 13})
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 324, in main
pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/build/test_python_2126709æ/@test_2126709_tmpæ", line 4, in <module>with multiprocessing.Pool(200) as p:
~~~~~~~~~~~~~~~~~~~~^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 119, in Poolreturn Pool(processes, initializer, initargs, maxtasksperchild,
context=self.get_context())
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 215, in __init__self._repopulate_pool()
~~~~~~~~~~~~~~~~~~~~~^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 306, in _repopulate_poolreturnself._repopulate_pool_static(self._ctx, self.Process,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^self._processes,
^^^^^^^^^^^^^^^^...<3 lines>...self._maxtasksperchild,
^^^^^^^^^^^^^^^^^^^^^^^self._wrap_exception)
^^^^^^^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 329, in _repopulate_pool_static
w.start()
~~~~~~~^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line 121, in startself._popen =self._Popen(self)
~~~~~~~~~~~^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 300, in _Popenreturn Popen(process_obj)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 35, in __init__super().__init__(process_obj)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 20, in __init__self._launch(process_obj)
~~~~~~~~~~~~^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 59, in _launchself.pid = forkserver.read_signed(self.sentinel)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 390, in read_signedraiseEOFError('unexpected EOF')
EOFError: unexpected EOF
---

[encukou]

The buildbot failed again with the same error -- OSError: [Errno 12] Cannot allocate memory.

[colesbury]

This change doesn't affect the default build. The added call is a no-op in that case. I'm pretty sure the buildbot is coincidentally in a bad state. Maybe too many zombie processes from previous runs or something like that.

[hugovk]

Perhaps a reboot of the buildbot machine might help? It's angelico-debian-amd64:

https://buildbot.python.org/#/builders/345
https://buildbot.python.org/#/workers/8

cc @Rosuav

[hugovk]

I made a draft revert PR (#128753) just to check the buildbots, and the "AMD64 Debian root" that failed here, passed there.

Reminder that the next 3.14 release is due on Tuesday: https://peps.python.org/pep-0745/

[colesbury]

And it also passed on main: https://buildbot.python.org/#/builders/345/builds/9920

[hugovk]

... but the newest build on main also passed :)

https://buildbot.python.org/#/builders/345

https://buildbot.python.org/#/builders/345/builds/9920

[hugovk]

Yes, all looks good now :)