[subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[graysky]

BPO	46070
Nosy	@vstinner, @encukou, @ericsnowcurrently, @ndjensen, @hroncok, @uckelman, @corona10, @miss-islington, @shihai1991, @erlend-aasland, @bsteffensmeier, @M-Reimer, @prahal
PRs	bpo-46070: Fix asyncio initialisation guard #30423 [3.10] bpo-46070: Fix asyncio initialisation guard (GH-30423) #30453 [3.9] bpo-46070: Fix asyncio initialisation guard (GH-30423) #30454 bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30564 [3.10] bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30565 [3.9] bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30566 bpo-46070: _PyGC_Fini() untracks objects #30577 [3.10] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30578 [3.9] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30579 [3.9] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30580
Files	win_py399_crash_reproducer.py: Reproducer that intermittently crashes on windows bug.py pyobject_ob_interp.patch sqlite3_crash.py: Reproducer that crashes 90% of the time bug.py_asyncio_cpustressed-crash.log: asyncio bug.py crash while running stress -c nproc --all

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-12-14.10:57:50.149>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11', 'expert-subinterpreters']
title = '[subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)'
updated_at = <Date 2022-03-24.21:16:20.771>
user = 'https://bugs.python.org/graysky'

bugs.python.org fields:

activity = <Date 2022-03-24.21:16:20.771>
actor = 'prahal'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Subinterpreters']
creation = <Date 2021-12-14.10:57:50.149>
creator = 'graysky'
dependencies = []
files = ['50494', '50497', '50560', '50699', '50700']
hgrepos = []
issue_num = 46070
keywords = ['patch']
message_count = 54.0
messages = ['408520', '408539', '408540', '408541', '408558', '408568', '408633', '408662', '408664', '408665', '409245', '409255', '409461', '409573', '409686', '409772', '409778', '409780', '409795', '409798', '409802', '409962', '409965', '409966', '409970', '409980', '409983', '409984', '409991', '410010', '410014', '410442', '410444', '410446', '410447', '410493', '410497', '410498', '410500', '410505', '410507', '410509', '410510', '410513', '410517', '410518', '410520', '415954', '415955', '415963', '415964', '415970', '415975', '415977']
nosy_count = 15.0
nosy_names = ['vstinner', 'petr.viktorin', 'eric.snow', 'ndjensen', 'hroncok', 'uckelman', 'corona10', 'miss-islington', 'shihai1991', 'erlendaasland', 'graysky', 'bsteffensmeier', 'M-Reimer', 'jokot3', 'prahal']
pr_nums = ['30423', '30453', '30454', '30564', '30565', '30566', '30577', '30578', '30579', '30580']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46070'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[graysky]

Seems as though cpython is broken when working with subinterpreters. The problematic change could be (d0d2965) affecting import.c.[1] Reverting this commit and rebuilding python fixes the issues on my system with some scripts that import sqlite, for example, the Kodi plugin YouTube[2] and the IMDB Trailers plugin. Others have reported similar breakage with other python code[3].

Example output when the bug manifests:

ERROR <general>: Traceback (most recent call last):
ERROR <general>: File "/usr/lib/python3.10/sqlite3/dbapi2.py", line 83, in register_adapters_and_converters
ERROR <general>: 
ERROR <general>: register_converter("timestamp", convert_timestamp)
ERROR <general>: 
ERROR <general>: KeyError
ERROR <general>: : 
ERROR <general>: 'timepart_full'
ERROR <general>: 
ERROR <general>: Exception ignored deletion of interned string failed
ERROR <general>: :

References:

1. d0d2965
2. Plugin incompatible with python 3.10.1 anxdpanic/plugin.video.youtube#255
3. https://bbs.archlinux.org/viewtopic.php?id=272121

[ericsnowcurrently]

(related: bpo-44059)

Presumably the problem relates to global state used in different interpreters leading to an inconsistent state in the crashing extension (or its dependencies).

@graysky, do you know if this was a problem before Python 3.8?

[encukou]

Interned strings were broken in #64257, see bpo-46006. Maybe that's also the issue here?