[C API] Move the PyCodeObject structure to the internal C API (make the structure opaque)

[vstinner]

BPO	47241
Nosy	@vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-04-06.13:56:37.081>
labels = ['expert-C-API', '3.11']
title = '[C API] Move the PyCodeObject structure to the internal C API (make the structure opaque)'
updated_at = <Date 2022-04-06.13:56:37.081>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-04-06.13:56:37.081>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2022-04-06.13:56:37.081>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 47241
keywords = []
message_count = 1.0
messages = ['416869']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue47241'
versions = ['Python 3.11']

[vstinner]

The PyCodeObject structure is documented at:
https://docs.python.org/dev/c-api/code.html

The structured evolved a lot in Python 3.11 to optimize Python/ceval.c performance:

• read-only co_code (object) was replaced with modifiable co_code_adaptive (char[])
• co_varnames, co_freevars and co_cellvars were removed: merged into co_localsplusnames with co_localspluskinds
• co_cell2arg was removed
• co_zombieframe ("free list") was removed
• co_opcache, co_opcache_map, co_opcache_flag and co_opcache_size were removed: see PEP-659
  https://peps.python.org/pep-0659/

New members:

• co_exceptiontable
• co_warmup, co_code_adaptive
• co_nlocalsplus, co_nplaincellvars, co_ncellvars, co_nfreevars
• co_localsplusnames, co_localspluskinds
• co_qualname
• co_endlinetable, co_columntable

The PyCodeObject structure should be made opaque in the public C API and only accessed with function calls.

In Python 3.11, the PyFrameObject structure was made opaque (bpo-46836) and multiple getters were added (bpo-40421). The idea is similar, but it might be too late to do that in Python 3.11 (beta1 feature freeze is close).

---

By the way, it was proposed multiple times on python-dev to mark the PyCode_New() function as "unstable" since its API changed often. PEP-670 "Python Positional-Only Parameters" caused a lot of troubles in Cython when it added a new parameter to PyCode_New(). The change was reverted: instead, a new PyCode_NewWithPosOnlyArgs() function was added.

• PyCode_New() has 19 parameters!
• PyCode_NewWithPosOnlyArgs() has 20 parameters!!

On Python 3.11a1 and newer, Cython uses the code.replace() method (added to Python 3.8) to build new code objects. Otherwise, it just calls directly PyCode_New().
https://docs.python.org/dev/library/types.html#types.CodeType.replace

[vstinner]

See the semi-stable API:

• https://peps.python.org/pep-0689/
• gh-91744: Add semi-stable C API tier #91789

[vstinner]

On coveragepy/coveragepy#1353 (comment), @Fidget-Spinner wrote:

Also, co_code has been gone since 2bde682. Which is 3.11a7 and up only. For that you'd need the internal _PyCode_GetCode (Huh, I just realised we should probably expose that in CPython's API).

It would be nice to add a public C API getter function, something like PyCode_GetCode() which would return a strong reference to a bytes object.

[vstinner]

It would be nice to add a public C API getter function, something like PyCode_GetCode() which would return a strong reference to a bytes object.

Ah, @Fidget-Spinner opened #92154 for that!

[vstinner]

Cython issue related to PyCodeObject structure size in Python 3.11: cython/cython#4827