Statically allocate interpreter states as much as possible.

[ericsnowcurrently]

BPO	45953
Nosy	@gvanrossum, @vstinner, @markshannon, @ericsnowcurrently, @JulienPalard, @miss-islington, @brandtbucher
PRs	bpo-45953: Statically allocate the main interpreter (and initial thread state). #29883 bpo-45953: Statically initialize the small ints. #30092 bpo-45953: Statically allocate and initialize global bytes objects. #30096 bpo-45953: Statically initialize all the _PyRuntimeState fields we can. #30588 bpo-45953: Statically initialize all the non-object PyInterpreterState fields we can. #30589 bpo-45953: Statically initialize all the PyThreadState fields we can. #30590 bpo-45953: Preserve backward compatibility on some PyThreadState field names. #31038

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = None
created_at = <Date 2021-12-01.18:50:39.120>
labels = ['interpreter-core', '3.11']
title = 'Statically allocate interpreter states as much as possible.'
updated_at = <Date 2022-03-29.07:04:09.685>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2022-03-29.07:04:09.685>
actor = 'mdk'
assignee = 'eric.snow'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-12-01.18:50:39.120>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 45953
keywords = ['patch']
message_count = 15.0
messages = ['407476', '408507', '410315', '410441', '410536', '410537', '412046', '412213', '412291', '416159', '416161', '416170', '416208', '416210', '416245']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'vstinner', 'Mark.Shannon', 'eric.snow', 'mdk', 'miss-islington', 'brandtbucher']
pr_nums = ['29883', '30092', '30096', '30588', '30589', '30590', '31038']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue45953'
versions = ['Python 3.11']

Linked PRs

• gh-90111: Minor Cleanup for Runtime-Global Objects #100254

[ericsnowcurrently]

Currently we allocate each new PyInterpreterState in PyInterpreterState_New(). Furthermore, PyInterpreterState is full of pointers which are almost all allocated on the heap during runtime init. We can statically allocate (and initialize?) most of what goes into PyInterpreterState, as well as the main interpreter itself (as part of _PyRuntimeState). This includes each interpreter's initial PyThreadState.

TODO:

• add PyInterpreterState main; to _PyRuntimeState.interpreters and use it
• change references from the pointer to the value
• add PyThreadState _main; to PyInterpreterState.threads and use it
• change references from the pointer to the value
• change PyInterpreterState pointer fields to values (as much as possible)
• change PyThreadState pointer fields to values (as much as possible)

benefits:

• fewer possible failures (no memory) during runtime/interpreter/thread init
• improved memory locality for pointers looked up relative to interpreter/thread state

There is one non-trivial bit: embedding the various PyObject values in PyInterpreterState and PyThreadState means hard-coding the various pieces of the object there (e.g. for dict, its keys/values; for list, its array), as well as adding necessary init code to PyInterpreterState_New() and PyThreadState_New(). The resulting added complexity can be mitigated somewhat with macros or even code generation. (In fact, there is probably significant overlap with Guido's deepfreeze tool.) Regardless, we'll probably need to factor out init funcs for a number of object types, where currently there are only "Py*_New()" funcs that combine allocation and init.

[ericsnowcurrently]

New changeset 121f1f8 by Eric Snow in branch 'main':
bpo-45953: Statically initialize the small ints. (gh-30092)
121f1f8

[ericsnowcurrently]

New changeset cf496d6 by Eric Snow in branch 'main':
bpo-45953: Statically allocate and initialize global bytes objects. (gh-30096)
cf496d6

[ericsnowcurrently]

New changeset ed57b36 by Eric Snow in branch 'main':
bpo-45953: Statically allocate the main interpreter (and initial thread state). (gh-29883)
ed57b36

[ericsnowcurrently]

New changeset 324908b by Eric Snow in branch 'main':
bpo-45953: Statically initialize all the PyThreadState fields we can. (gh-30590)
324908b

[ericsnowcurrently]

New changeset 322f962 by Eric Snow in branch 'main':
bpo-45953: Statically initialize all the non-object PyInterpreterState fields we can. (gh-30589)
322f962

[brandtbucher]

Any chance we could revert the recent renaming of tstate.exc_state and tstate.root_cframe in #30590? It broke Greenlet again:

python-greenlet/greenlet#288

If it's only a name change (and the members themselves are the same), I think reverting it is preferable to burying Greenlet in more compatibility macros and bugging them to put out another new release.

[ericsnowcurrently]

Any chance we could revert the recent renaming of tstate.exc_state and tstate.root_cframe

Yeah, I'll sort this out. Sorry for that.

[miss-islington]

New changeset f78be59 by Eric Snow in branch 'main':
bpo-45953: Preserve backward compatibility on some PyThreadState field names. (GH-31038)
f78be59