[subinterpreters] PyObject statics exposed in the limited API break isolation.

[ericsnowcurrently]

BPO	43503
Nosy	@gvanrossum, @nascheme, @vstinner, @encukou, @markshannon, @ericsnowcurrently, @mattip
PRs	bpo-43503: Make limited API objects effectively immutable. #24828

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-15.18:53:43.710>
labels = ['interpreter-core', 'expert-subinterpreters', 'type-bug', '3.10', 'extension-modules', 'expert-C-API']
title = '[subinterpreters] PyObject statics exposed in the limited API break isolation.'
updated_at = <Date 2021-03-17.14:46:29.259>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2021-03-17.14:46:29.259>
actor = 'eric.snow'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'Interpreter Core', 'C API', 'Subinterpreters']
creation = <Date 2021-03-15.18:53:43.710>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 43503
keywords = ['patch']
message_count = 14.0
messages = ['388759', '388764', '388765', '388773', '388782', '388783', '388848', '388859', '388861', '388871', '388920', '388922', '388923', '388924']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'nascheme', 'vstinner', 'petr.viktorin', 'Mark.Shannon', 'eric.snow', 'mattip']
pr_nums = ['24828']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43503'
versions = ['Python 3.10']

[ericsnowcurrently]

In the limited C-API we expose the following static PyObject variables:

• 5 singletons
• ~70 exception types
• ~70 other types

Since they are part of the limited API, they have a direct effect on the stable ABI.

The problem is that these objects should not be shared between interpreters. There are a number of possible solutions for isolating the objects, but the big constraint is that the solution cannot break the stable ABI.

[ericsnowcurrently]

Here are some solutions that I've considered:

1. immutable objects
   a. make the objects truly immutable/const
   • not trivial, if possible
     b. make the objects effectively immutable
   • (see Segfault when using store-context AST node in a load context #69016) use a really high refcount to make races irrelevant
2. per-interpreter objects
   a. replace them with macros that do a per-interpreter lookup
   b. replace them with simple placeholders and do a per-interpreter lookup internally
   c. replace them with PyObject placeholders and do a per-interpreter lookup internally

As far as I'm aware, only (1b) and (2c) are realistic and won't break the stable ABI (i.e. preserve layout).

(FWIW, I think that even with (1b) we would still have per-interpreter objects.)

-- Regarding (1a) --

See see #69016 for an example implementation. This includes storing some state for the objects in PyInterpreterState and doing a lookup internally.

pros:

• relatively straightforward to implement
• overlaps with other interests (see bpo-40255)
• makes the objects shareable between interpreters (making them more efficient)

cons:

• we have to ensure the objects stay immutable (a tractable problem if the solution is constrained to only the limited API objects)

-- Regarding (2c) --

This involves adding an API to get the per-interpreter object for a given identity value (flag, global, etc.) and then mapping the limited API objects to the corresponding per-interpreter ones.

pros:

• avoids a one-off solution
• extensions can stop using the variables directly (in favor of the new lookup API)

cons:

• effectively couples the C-API to the design (as long as the objects are in the limited API)
• many touches all over the C-API
• any future changes/additions in the C-API must deal with the objects

[ericsnowcurrently]

If the stable ABI weren't an issue then we would probably:

• deprecate using the objects directly
• do something like (2a) in the meantime

It may make sense to do that for "#ifndef Py_LIMITED_API", regardless of how we handle the limited API.