Share global string identifiers in deepfreeze

[kumaraditya303]

BPO	46712
Nosy	@gvanrossum, @ericsnowcurrently, @sweeneyde, @kumaraditya303
PRs	bpo-46712: Share global string identifiers in deepfreeze #31261 bpo-46430: fix error-handling in _Py_Deepfreeze_Init #31596 bpo-46712: Let generate_global_objects.py Run on Earlier Python Versions #31637 bpo-46712: Do not Regen Deep-Frozen Modules before Generating Global Objects #32061 bpo-46712: share more global strings in deepfreeze #32152

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-02-25.18:08:29.439>
created_at = <Date 2022-02-10.14:18:03.087>
labels = ['3.11', 'performance']
title = 'Share global string identifiers in deepfreeze'
updated_at = <Date 2022-03-28.09:14:38.402>
user = 'https://github.com/kumaraditya303'

bugs.python.org fields:

activity = <Date 2022-03-28.09:14:38.402>
actor = 'kumaraditya'
assignee = 'none'
closed = True
closed_date = <Date 2022-02-25.18:08:29.439>
closer = 'gvanrossum'
components = []
creation = <Date 2022-02-10.14:18:03.087>
creator = 'kumaraditya'
dependencies = []
files = []
hgrepos = []
issue_num = 46712
keywords = ['patch']
message_count = 7.0
messages = ['413003', '413005', '414028', '414295', '415665', '415811', '415882']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'eric.snow', 'Dennis Sweeney', 'kumaraditya']
pr_nums = ['31261', '31596', '31637', '32061', '32152']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46712'
versions = ['Python 3.11']

• PR: gh-90868: Adjust the Generated Objects #99223

• PR: gh-90868: Add _PyStaticObject_CheckRefcnt() function #99261

[kumaraditya303]

Since bpo-46541, the global strings are statically allocated so they can now be referenced by deep-frozen modules just like any other singleton. Sharing identifiers with deepfreeze will reduce the duplicated strings hence it would save space.

See faster-cpython/ideas#218
See faster-cpython/ideas#230

[kumaraditya303]

I have refactored generate_global_objects.py, and now instead of hard-coding every identifier manually, it now scans *.c files extracts the identifiers used in it and then generate the header file. This has multiple advantages:

• No need to manually add identifiers, as soon as it is used in a c file it is added to the global identifiers struct.
• It simplifies the codegen a lot.
• Remove the need of special casing certain file for checking now it is just a set of identifiers and auto removes unused global strings.

[gvanrossum]

New changeset eb002db by Kumar Aditya in branch 'main':
bpo-46712: Share global string identifiers in deepfreeze (GH-31261)
eb002db

[ericsnowcurrently]

New changeset 21099fc by Eric Snow in branch 'main':
bpo-46712: Let generate_global_objects.py Run on Earlier Python Versions (gh-31637)
21099fc

[sweeneyde]

In bpo-47067, there was concern about the addition of the makefile target from PR 31637:

regen-global-objects: regen-deepfreeze

After a new &_Py_ID(__orig_class__) is added to Objects/genericaliasobject.c, running make regen-global-objects starts

gcc -pthread -c [snipped] -DPy_BUILD_CORE -o [Objects/genericaliasobject.o](https://github.com/python/cpython/blob/main/Objects/genericaliasobject.o) [Objects/genericaliasobject.c](https://github.com/python/cpython/blob/main/Objects/genericaliasobject.c)

which fails with a compilation error because that identifier is not yet defined. Is there a good way to convince make to regenerate the global objects without this sort of circular dependency? Am I missing a step?

[ericsnowcurrently]

After a new &_Py_ID(__orig_class__) is added to Objects/genericaliasobject.c, running make regen-global-objects starts

gcc -pthread -c [snipped] -DPy_BUILD_CORE -o [Objects/genericaliasobject.o](https://github.com/python/cpython/blob/main/Objects/genericaliasobject.o) [Objects/genericaliasobject.c](https://github.com/python/cpython/blob/main/Objects/genericaliasobject.c)

which fails with a compilation error because that identifier is not yet defined. Is there a good way to convince make to regenerate the global objects without this sort of circular dependency? Am I missing a step?

I'm looking into this. A temporary workaround is to run Tools/scripts/generate-global-objects.py directly.

[ericsnowcurrently]

New changeset febf54b by Eric Snow in branch 'main':
bpo-46712: Do not Regen Deep-Frozen Modules before Generating Global Objects (gh-32061)
febf54b