gh-112292 : Catch import error conditions with readline hooks by tonybaloney · Pull Request #112313 · python/cpython

tonybaloney · 2023-11-22T08:41:27Z

Prevents a segmentation fault when modules import (or configure) readline from inside a sub interpreter.

The readlinestate_global macro was error-prone to PyImport_FindModule returning NULL and crashing in about 18 places. I could reproduce 1 easily, but this PR replaces the macro with a function and adds error conditions to the other functions.

Issue: Check for sub interpreter support is not catching readline global state and crashing #112292

…hook

ericsnowcurrently · 2023-11-22T16:35:48Z

+1 to fixing the other hooks, whether in this PR or another (though I'd think this PR would be the better place).

ericsnowcurrently

Mostly LGTM. I've noted one thing to fix and a couple others to consider.

bedevere-app · 2023-11-22T17:02:10Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

… into robust_readline_hook

… to get global state

… checks (won't raise assertion in non-debug code)

tonybaloney · 2023-11-23T02:13:10Z

I have made the requested changes; please review again

ericsnowcurrently

Thanks for the extra effort here. There are a few things we may want to do differently here, for which I've left notes.

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

… into robust_readline_hook

tonybaloney · 2023-11-28T01:35:18Z

@ericsnowcurrently it wasn't possible to check module state inside setup_readline but this code is more robust now so it no longer crashes when imported twice in a sub interpreter. test_builtin and test_readline now correctly fail instead of segfaulting.

$ ./python.exe test_in_interp.py test_builtinRunning test test_builtintest_abs (test.test_builtin.BuiltinTest.test_abs) ... oktest_all (test.test_builtin.BuiltinTest.test_all) ... oktest_any (test.test_builtin.BuiltinTest.test_any) ... oktest_ascii (test.test_builtin.BuiltinTest.test_ascii) ... oktest_bin (test.test_builtin.BuiltinTest.test_bin) ... oktest_bug_27936 (test.test_builtin.BuiltinTest.test_bug_27936) ... oktest_bytearray_extend_error (test.test_builtin.BuiltinTest.test_bytearray_extend_error) ... oktest_bytearray_translate (test.test_builtin.BuiltinTest.test_bytearray_translate) ... oktest_callable (test.test_builtin.BuiltinTest.test_callable) ... oktest_chr (test.test_builtin.BuiltinTest.test_chr) ... oktest_cmp (test.test_builtin.BuiltinTest.test_cmp) ... oktest_compile (test.test_builtin.BuiltinTest.test_compile) ... oktest_compile_ast (test.test_builtin.BuiltinTest.test_compile_ast) ... oktest_compile_async_generator (test.test_builtin.BuiltinTest.test_compile_async_generator)With the PyCF_ALLOW_TOP_LEVEL_AWAIT flag added in 3.8, we want to ... oktest_compile_top_level_await (test.test_builtin.BuiltinTest.test_compile_top_level_await)Test whether code some top level await can be compiled. ... oktest_compile_top_level_await_invalid_cases (test.test_builtin.BuiltinTest.test_compile_top_level_await_invalid_cases) ... oktest_compile_top_level_await_no_coro (test.test_builtin.BuiltinTest.test_compile_top_level_await_no_coro)Make sure top level non-await codes get the correct coroutine flags ... oktest_construct_singletons (test.test_builtin.BuiltinTest.test_construct_singletons) ... oktest_delattr (test.test_builtin.BuiltinTest.test_delattr) ... oktest_dir (test.test_builtin.BuiltinTest.test_dir) ... oktest_divmod (test.test_builtin.BuiltinTest.test_divmod) ... oktest_eval (test.test_builtin.BuiltinTest.test_eval) ... oktest_exec (test.test_builtin.BuiltinTest.test_exec) ... oktest_exec_closure (test.test_builtin.BuiltinTest.test_exec_closure) ... oktest_exec_globals (test.test_builtin.BuiltinTest.test_exec_globals) ... oktest_exec_globals_dict_subclass (test.test_builtin.BuiltinTest.test_exec_globals_dict_subclass) ... oktest_exec_globals_error_on_get (test.test_builtin.BuiltinTest.test_exec_globals_error_on_get) ... oktest_exec_globals_frozen (test.test_builtin.BuiltinTest.test_exec_globals_frozen) ... oktest_exec_redirected (test.test_builtin.BuiltinTest.test_exec_redirected) ... oktest_filter (test.test_builtin.BuiltinTest.test_filter) ... oktest_filter_dealloc (test.test_builtin.BuiltinTest.test_filter_dealloc) ... skipped "resource 'cpu' is not enabled"test_filter_pickle (test.test_builtin.BuiltinTest.test_filter_pickle) ... oktest_format (test.test_builtin.BuiltinTest.test_format) ... oktest_general_eval (test.test_builtin.BuiltinTest.test_general_eval) ... oktest_getattr (test.test_builtin.BuiltinTest.test_getattr) ... oktest_hasattr (test.test_builtin.BuiltinTest.test_hasattr) ... oktest_hash (test.test_builtin.BuiltinTest.test_hash) ... oktest_hex (test.test_builtin.BuiltinTest.test_hex) ... oktest_id (test.test_builtin.BuiltinTest.test_id) ... oktest_import (test.test_builtin.BuiltinTest.test_import) ... oktest_input (test.test_builtin.BuiltinTest.test_input) ... oktest_isinstance (test.test_builtin.BuiltinTest.test_isinstance) ... oktest_issubclass (test.test_builtin.BuiltinTest.test_issubclass) ... oktest_iter (test.test_builtin.BuiltinTest.test_iter) ... oktest_len (test.test_builtin.BuiltinTest.test_len) ... oktest_map (test.test_builtin.BuiltinTest.test_map) ... oktest_map_pickle (test.test_builtin.BuiltinTest.test_map_pickle) ... oktest_max (test.test_builtin.BuiltinTest.test_max) ... oktest_min (test.test_builtin.BuiltinTest.test_min) ... oktest_neg (test.test_builtin.BuiltinTest.test_neg) ... oktest_next (test.test_builtin.BuiltinTest.test_next) ... oktest_oct (test.test_builtin.BuiltinTest.test_oct) ... oktest_open (test.test_builtin.BuiltinTest.test_open) ... oktest_open_default_encoding (test.test_builtin.BuiltinTest.test_open_default_encoding) ... oktest_open_non_inheritable (test.test_builtin.BuiltinTest.test_open_non_inheritable) ... oktest_ord (test.test_builtin.BuiltinTest.test_ord) ... oktest_pow (test.test_builtin.BuiltinTest.test_pow) ... oktest_repr (test.test_builtin.BuiltinTest.test_repr) ... oktest_round (test.test_builtin.BuiltinTest.test_round) ... oktest_round_large (test.test_builtin.BuiltinTest.test_round_large) ... oktest_setattr (test.test_builtin.BuiltinTest.test_setattr) ... oktest_sum (test.test_builtin.BuiltinTest.test_sum) ... oktest_sum_accuracy (test.test_builtin.BuiltinTest.test_sum_accuracy) ... oktest_type (test.test_builtin.BuiltinTest.test_type) ... oktest_vars (test.test_builtin.BuiltinTest.test_vars) ... oktest_warning_notimplemented (test.test_builtin.BuiltinTest.test_warning_notimplemented) ... oktest_zip (test.test_builtin.BuiltinTest.test_zip) ... oktest_zip_bad_iterable (test.test_builtin.BuiltinTest.test_zip_bad_iterable) ... oktest_zip_pickle (test.test_builtin.BuiltinTest.test_zip_pickle) ... oktest_zip_pickle_strict (test.test_builtin.BuiltinTest.test_zip_pickle_strict) ... oktest_zip_pickle_strict_fail (test.test_builtin.BuiltinTest.test_zip_pickle_strict_fail) ... oktest_zip_result_gc (test.test_builtin.BuiltinTest.test_zip_result_gc) ... oktest_zip_strict (test.test_builtin.BuiltinTest.test_zip_strict) ... oktest_zip_strict_error_handling (test.test_builtin.BuiltinTest.test_zip_strict_error_handling) ... oktest_zip_strict_error_handling_stopiteration (test.test_builtin.BuiltinTest.test_zip_strict_error_handling_stopiteration) ... oktest_zip_strict_iterators (test.test_builtin.BuiltinTest.test_zip_strict_iterators) ... oktest_immortals (test.test_builtin.ImmortalTests.test_immortals) ... oktest_list_repeat_respect_immortality (test.test_builtin.ImmortalTests.test_list_repeat_respect_immortality) ... oktest_tuple_repeat_respect_immortality (test.test_builtin.ImmortalTests.test_tuple_repeat_respect_immortality) ... oktest_input_no_stdout_fileno (test.test_builtin.PtyTests.test_input_no_stdout_fileno) ... ERRORtest_input_tty (test.test_builtin.PtyTests.test_input_tty) ... ERRORtest_input_tty_non_ascii (test.test_builtin.PtyTests.test_input_tty_non_ascii) ... ERRORtest_input_tty_non_ascii_unicode_errors (test.test_builtin.PtyTests.test_input_tty_non_ascii_unicode_errors) ... ERRORtest_cleanup (test.test_builtin.ShutdownTest.test_cleanup) ... oktest_breakpoint (test.test_builtin.TestBreakpoint.test_breakpoint) ... oktest_breakpoint_with_args_and_keywords (test.test_builtin.TestBreakpoint.test_breakpoint_with_args_and_keywords) ... oktest_breakpoint_with_breakpointhook_reset (test.test_builtin.TestBreakpoint.test_breakpoint_with_breakpointhook_reset) ... oktest_breakpoint_with_breakpointhook_set (test.test_builtin.TestBreakpoint.test_breakpoint_with_breakpointhook_set) ... oktest_breakpoint_with_passthru_error (test.test_builtin.TestBreakpoint.test_breakpoint_with_passthru_error) ... oktest_envar_good_path_builtin (test.test_builtin.TestBreakpoint.test_envar_good_path_builtin) ... oktest_envar_good_path_empty_string (test.test_builtin.TestBreakpoint.test_envar_good_path_empty_string) ... oktest_envar_good_path_noop_0 (test.test_builtin.TestBreakpoint.test_envar_good_path_noop_0) ... oktest_envar_good_path_other (test.test_builtin.TestBreakpoint.test_envar_good_path_other) ... oktest_envar_ignored_when_hook_is_set (test.test_builtin.TestBreakpoint.test_envar_ignored_when_hook_is_set) ... oktest_envar_unimportable (test.test_builtin.TestBreakpoint.test_envar_unimportable) ... oktest_runtime_error_when_hook_is_lost (test.test_builtin.TestBreakpoint.test_runtime_error_when_hook_is_lost) ... oktest_bad_arguments (test.test_builtin.TestSorted.test_bad_arguments) ... oktest_baddecorator (test.test_builtin.TestSorted.test_baddecorator) ... oktest_basic (test.test_builtin.TestSorted.test_basic) ... oktest_inputtypes (test.test_builtin.TestSorted.test_inputtypes) ... oktest_bad_args (test.test_builtin.TestType.test_bad_args) ... oktest_bad_slots (test.test_builtin.TestType.test_bad_slots) ... oktest_namespace_order (test.test_builtin.TestType.test_namespace_order) ... oktest_new_type (test.test_builtin.TestType.test_new_type) ... oktest_type_doc (test.test_builtin.TestType.test_type_doc) ... oktest_type_name (test.test_builtin.TestType.test_type_name) ... oktest_type_nokwargs (test.test_builtin.TestType.test_type_nokwargs) ... oktest_type_qualname (test.test_builtin.TestType.test_type_qualname) ... oktest_type_typeparams (test.test_builtin.TestType.test_type_typeparams) ... okbin (builtins)Doctest: builtins.bin ... okhex (builtins.bytearray)Doctest: builtins.bytearray.hex ... okhex (builtins.bytes)Doctest: builtins.bytes.hex ... okas_integer_ratio (builtins.float)Doctest: builtins.float.as_integer_ratio ... okfromhex (builtins.float)Doctest: builtins.float.fromhex ... okhex (builtins.float)Doctest: builtins.float.hex ... okhex (builtins)Doctest: builtins.hex ... okint (builtins)Doctest: builtins.int ... okas_integer_ratio (builtins.int)Doctest: builtins.int.as_integer_ratio ... okbit_count (builtins.int)Doctest: builtins.int.bit_count ... okbit_length (builtins.int)Doctest: builtins.int.bit_length ... okhex (builtins.memoryview)Doctest: builtins.memoryview.hex ... okoct (builtins)Doctest: builtins.oct ... okzip (builtins)Doctest: builtins.zip ... ok======================================================================ERROR: test_input_no_stdout_fileno (test.test_builtin.PtyTests.test_input_no_stdout_fileno)----------------------------------------------------------------------Traceback (most recent call last): File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2314, in test_input_no_stdout_fileno lines = self.run_child(child, b"quux\r") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2187, in run_child old_sighup = signal.signal(signal.SIGHUP, self.handle_sighup) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anthonyshaw/projects/cpython/Lib/signal.py", line 56, in signal handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError: signal only works in main thread of the main interpreter======================================================================ERROR: test_input_tty (test.test_builtin.PtyTests.test_input_tty)----------------------------------------------------------------------Traceback (most recent call last): File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2284, in test_input_tty self.check_input_tty("prompt", b"quux") File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2269, in check_input_tty lines = self.run_child(child, terminal_input + b"\r\n") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2187, in run_child old_sighup = signal.signal(signal.SIGHUP, self.handle_sighup) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anthonyshaw/projects/cpython/Lib/signal.py", line 56, in signal handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError: signal only works in main thread of the main interpreter======================================================================ERROR: test_input_tty_non_ascii (test.test_builtin.PtyTests.test_input_tty_non_ascii)----------------------------------------------------------------------Traceback (most recent call last): File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2299, in test_input_tty_non_ascii self.check_input_tty("prompté", b"quux\xe9", "utf-8") File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2269, in check_input_tty lines = self.run_child(child, terminal_input + b"\r\n") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2187, in run_child old_sighup = signal.signal(signal.SIGHUP, self.handle_sighup) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anthonyshaw/projects/cpython/Lib/signal.py", line 56, in signal handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError: signal only works in main thread of the main interpreter======================================================================ERROR: test_input_tty_non_ascii_unicode_errors (test.test_builtin.PtyTests.test_input_tty_non_ascii_unicode_errors)----------------------------------------------------------------------Traceback (most recent call last): File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2304, in test_input_tty_non_ascii_unicode_errors self.check_input_tty("prompté", b"quux\xe9", "ascii") File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2269, in check_input_tty lines = self.run_child(child, terminal_input + b"\r\n") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anthonyshaw/projects/cpython/Lib/test/test_builtin.py", line 2187, in run_child old_sighup = signal.signal(signal.SIGHUP, self.handle_sighup) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/anthonyshaw/projects/cpython/Lib/signal.py", line 56, in signal handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError: signal only works in main thread of the main interpreter----------------------------------------------------------------------Ran 122 tests in 0.298sFAILED (errors=4, skipped=1)test test_builtin failed
$ ./python.exe test_in_interp.py test_readlineRunning test test_readlinetest_readline skipped -- module readline does not support loading in subinterpreters

ericsnowcurrently · 2023-11-28T01:49:53Z

it wasn't possible to check module state inside setup_readline

I'm guessing it's because PyState_FindModule() will always return NULL until after the module init func has already finished. That's fine.

but this code is more robust now so it no longer crashes when imported twice in a sub interpreter.

Yeah, that's the higher priority thing. The module is definitely in a better place now.

ericsnowcurrently

LGTM

ericsnowcurrently · 2023-11-28T01:59:13Z

Thanks for taking care of this, @tonybaloney!

ericsnowcurrently · 2023-11-28T02:00:09Z

How much of this should be backported to 3.12?

tonybaloney · 2023-11-28T02:01:37Z

How much of this should be backported to 3.12?

All of it could be

ericsnowcurrently · 2023-11-28T02:12:38Z

@Yhg1s, what do you think about backporting this as-is?

…ythongh-112313) Prevents a segmentation fault in registered hooks for the readline library, but only when the readline module is loaded inside an isolated sub interpreter. The module is single-phase init so loading it fails, but not until the module init function has already run, where the readline hooks get registered. The readlinestate_global macro was error-prone to PyImport_FindModule returning NULL and crashing in about 18 places. I could reproduce 1 easily, but this PR replaces the macro with a function and adds error conditions to the other functions.

Catch import error conditions with readline callback hook on_startup_…
e8d2115
…hook

bedevere-appBot mentioned this pull request Nov 22, 2023
Check for sub interpreter support is not catching readline global state and crashing #112292
Closed

tonybaloney marked this pull request as ready for review November 22, 2023 08:41

bedevere-appBot added the awaiting review label Nov 22, 2023

Add news entry
d56fb47

corona10 reviewed Nov 22, 2023
View reviewed changes

Comment threadModules/readline.c Outdated

corona10 requested a review from ericsnowcurrently November 22, 2023 10:02

ericsnowcurrently requested changes Nov 22, 2023
View reviewed changes

Comment threadModules/readline.c Outdated
Comment threadModules/readline.c
Comment threadModules/readline.c Outdated
Comment threadModules/readline.c Outdated

bedevere-appBot added awaiting changes and removed awaiting review labels Nov 22, 2023

tonybaloneyand others added 7 commits November 23, 2023 10:29

Move the GIL release to after setting an error
6e9e66d
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

remove else
364d8d5

Merge branch 'robust_readline_hook' of github.com:tonybaloney/cpython…
4ddf510
… into robust_readline_hook

Harden the on_pre_input_hook
8890f5c

Replace uses of a global macro with no error checking with a function…
f4580c8
… to get global state

Don't set exceptions on non-Python calls.
b408004

Acquire GIL before checking module state since it will require tstate…
30a1a73
… checks (won't raise assertion in non-debug code)

tonybaloney changed the title ~~gh-112292 : Catch import error conditions with readline on_startup_hook~~gh-112292 : Catch import error conditions with readline hooksNov 23, 2023

Merge remote-tracking branch 'origin/main' into robust_readline_hook
64cfda5

ericsnowcurrently requested changes Nov 27, 2023
View reviewed changes

tonybaloneyand others added 7 commits November 28, 2023 10:04

Update Modules/readline.c
451513b
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

Update Modules/readline.c
59239f0
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

Apply suggestions from code review
cf72dc4
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

Update Modules/readline.c
aa75a62
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>

fixup variable name
30c79d7

Merge branch 'robust_readline_hook' of github.com:tonybaloney/cpython…
ae57bed
… into robust_readline_hook

Use simpler assertion
b1fa25b

tonybaloney added 2 commits November 28, 2023 11:21

Simpler hook check
ff038f0

Don't assert error check
43d609d

ericsnowcurrently approved these changes Nov 28, 2023
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting changes labels Nov 28, 2023

ericsnowcurrently merged commit 154f099 into python:mainNov 28, 2023

bedevere-appBot removed the awaiting merge label Nov 28, 2023

tonybaloney deleted the robust_readline_hook branch November 28, 2023 02:01

ericsnowcurrently mentioned this pull request Jul 8, 2024
[subinterpreters] Multiple interpreters and readline module hook functions. #48452
Closed

Uh oh!

Uh oh!

Conversation

tonybaloney commented Nov 22, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ericsnowcurrently commented Nov 22, 2023

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-appBot commented Nov 22, 2023

Uh oh!

tonybaloney commented Nov 23, 2023

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonybaloney commented Nov 28, 2023

Uh oh!

ericsnowcurrently commented Nov 28, 2023

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently commented Nov 28, 2023

Uh oh!

ericsnowcurrently commented Nov 28, 2023

Uh oh!

tonybaloney commented Nov 28, 2023

Uh oh!

ericsnowcurrently commented Nov 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tonybaloney commented Nov 22, 2023•
edited
Loading