gh-130804: Fix support of typing unicode chars in pyrepl by sergey-miryanov · Pull Request #130805 · python/cpython

sergey-miryanov · 2025-03-03T18:57:59Z

Fixed typing of unicode chars (I tested on Cyrillic chars on Windows)

Issue: [Windows] New REPL doesn't allow to input non-ASCII Unicode characters #130804

encukou

This does fix the issue on Windows. I've also tested on Linux and saw no breakage.

Co-authored-by: Petr Viktorin <encukou@gmail.com>

pablogsal · 2025-03-10T21:44:14Z

LGTM

Thanks a lot for the fix @sergey-miryanov

miss-islington-app · 2025-03-10T22:13:54Z

Thanks @sergey-miryanov for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2025-03-10T22:13:56Z

Sorry, @sergey-miryanov and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7c98b0674daa3e4eb3e8f35afb61a0dba61d1780 3.13

StanFromIreland · 2025-03-10T22:22:53Z

Tests have failed.
workflow

pablogsal · 2025-03-10T22:27:24Z

How was the PR landed then?

pablogsal · 2025-03-10T22:28:17Z

Test were ok in https://github.com/python/cpython/actions/runs/13775445246/job/38523588646

StanFromIreland · 2025-03-10T22:34:30Z

It's odd, it's the same set up yet somehow they are all failing now?

pablogsal · 2025-03-10T22:34:43Z

This PR also only modifies test_multiple_statements_fail_early so I don't see how is this related

pablogsal · 2025-03-10T22:35:54Z

Reverting doesn't fix anything:

~/github/python/main main ⇡
❯ git revert 7c98b0674daa3e4eb3e8f35afb61a0dba61d1780
[main 46a7449b8e9] Revert "gh-130804: Fix support of typing unicode chars in pyrepl (#130805)"
3 files changed, 3 insertions(+), 17 deletions(-)
delete mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2025-03-10-21-46-37.gh-issue-130804.0PpcTx.rst
~/github/python/main main ⇡
❯ make -j -s
Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
~/github/python/main main ⇡
❯ ./python -m test test_repl -vvv
== CPython 3.14.0a5+ (heads/main:7c98b0674da, Mar 10 2025, 22:34:19) [GCC 14.2.1 20250207]
== Linux-6.13.5-arch1-1-x86_64-with-glibc2.41 little-endian
== Python build: debug
== cwd: /home/pablogsal/github/python/main/build/test_python_worker_1544359æ
== CPU count: 36
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
Using random seed: 1616529274
0:00:00 load avg: 9.94 Run 1 test sequentially in a single process
0:00:00 load avg: 9.94 [1/1] test_repl
test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ... FAIL
test_toplevel_contextvars_async (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_async) ... ok
test_toplevel_contextvars_sync (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_sync) ... ok
test_asyncio_repl_is_ok (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok) ... ok
test_asyncio_repl_reaches_python_startup_script (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_reaches_python_startup_script) ... ok
test_close_stdin (test.test_repl.TestInteractiveInterpreter.test_close_stdin) ... ok
test_interactive_source_is_in_linecache (test.test_repl.TestInteractiveInterpreter.test_interactive_source_is_in_linecache) ... ok
test_interactive_traceback_reporting (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting) ... ok
test_interactive_traceback_reporting_multiple_input (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting_multiple_input) ... ok
test_multiline_string_parsing (test.test_repl.TestInteractiveInterpreter.test_multiline_string_parsing) ... ok
test_no_memory (test.test_repl.TestInteractiveInterpreter.test_no_memory) ... ok
test_runsource_show_syntax_error_location (test.test_repl.TestInteractiveInterpreter.test_runsource_show_syntax_error_location) ... ok
test_interactive_syntax_error_correct_line (test.test_repl.TestInteractiveModeSyntaxErrors.test_interactive_syntax_error_correct_line) ... ok
======================================================================
FAIL: test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablogsal/github/python/main/Lib/test/test_repl.py", line 304, in test_multiple_statements_fail_early
self.assertNotIn("afterwards", output)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'afterwards' unexpectedly found in 'asyncio REPL 3.14.0a5+ (heads/main:7c98b0674da, Mar 10 2025, 22:34:19) [GCC 14.2.1 20250207] on linux\nUse "await" directly instead of "asyncio.run()".\nType "help", "copyright", "credits" or "license" for more information.\n>>> import asyncio\n>>> Traceback (most recent call last):\n File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 448, in result\n return self.__get_result()\n ~~~~~~~~~~~~~~~~~^^\n File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 393, in __get_result\n raise self._exception\n File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 42, in callback\n coro = func()\n File "<stdin>", line 1, in <module>\n 1 / 0; print(\'afterwards\')\n ~~^~~\nZeroDivisionError: division by zero\n>>> \nexiting asyncio REPL...\nTraceback (most recent call last):\n File "/home/pablogsal/github/python/main/Lib/runpy.py", line 198, in _run_module_as_main\n return _run_code(code, main_globals, None,\n "__main__", mod_spec)\n File "/home/pablogsal/github/python/main/Lib/runpy.py", line 88, in _run_code\n exec(code, run_globals)\n ~~~~^^^^^^^^^^^^^^^^^^^\n File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 204, in <module>\n sys.exit(return_code)\n ~~~~~~~~^^^^^^^^^^^^^\nSystemExit: 0\n>>> \n'
----------------------------------------------------------------------
Ran 13 tests in 1.223s
FAILED (failures=1)
test test_repl failed
0:00:01 load avg: 9.94 [1/1/1] test_repl failed (1 failure)
== Tests result: FAILURE ==
1 test failed:
test_repl
Total duration: 1.3 sec
Total tests: run=13 failures=1
Total test files: run=1/1 failed=1
Result: FAILURE

StanFromIreland · 2025-03-10T22:37:22Z

Its this PR: 6ab5c4a

pablogsal · 2025-03-10T22:38:04Z

Doesn't seem like it:

~/github/python/main main ⇡
❯ git revert 6ab5c4aa05bf35832a3ccd1e71b28b8475fa30f4
[main 1428e541980] Revert "gh-124927: Fix conversion issue between coordinates and position in REPL (#125001)"
4 files changed, 1 insertion(+), 12 deletions(-)
delete mode 100644 Misc/NEWS.d/next/Library/2024-10-05-13-25-07.gh-issue-124927.uzNA32.rst
~/github/python/main main ⇡
❯ make
Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
./python -E ./Tools/build/generate-build-details.py `cat pybuilddir.txt`/build-details.json
~/github/python/main main ⇡
❯ ./python -m test test_repl -vvv
== CPython 3.14.0a5+ (bisect/good-ecdf6b15b0c0570c3c3302ab95bdbfd3007ea941-2-g6ab5c4aa05b:6ab5c4aa05b,) [GCC 14.2.1 20250207]
== Linux-6.13.5-arch1-1-x86_64-with-glibc2.41 little-endian
== Python build: debug
== cwd: /home/pablogsal/github/python/main/build/test_python_worker_1588881æ
== CPU count: 36
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
Using random seed: 4038269957
0:00:00 load avg: 3.85 Run 1 test sequentially in a single process
0:00:00 load avg: 3.85 [1/1] test_repl
test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ... FAIL
test_toplevel_contextvars_async (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_async) ... ok
test_toplevel_contextvars_sync (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_sync) ... ok
test_asyncio_repl_is_ok (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok) ... ok
test_asyncio_repl_reaches_python_startup_script (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_reaches_python_startup_script) ... ok
test_close_stdin (test.test_repl.TestInteractiveInterpreter.test_close_stdin) ... ok
test_interactive_source_is_in_linecache (test.test_repl.TestInteractiveInterpreter.test_interactive_source_is_in_linecache) ... ok
test_interactive_traceback_reporting (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting) ... ok
test_interactive_traceback_reporting_multiple_input (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting_multiple_input) ... ok
test_multiline_string_parsing (test.test_repl.TestInteractiveInterpreter.test_multiline_string_parsing) ... ok
test_no_memory (test.test_repl.TestInteractiveInterpreter.test_no_memory) ... ok
test_runsource_show_syntax_error_location (test.test_repl.TestInteractiveInterpreter.test_runsource_show_syntax_error_location) ... ok
test_interactive_syntax_error_correct_line (test.test_repl.TestInteractiveModeSyntaxErrors.test_interactive_syntax_error_correct_line) ... ok
======================================================================
FAIL: test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pablogsal/github/python/main/Lib/test/test_repl.py", line 304, in test_multiple_statements_fail_early
self.assertNotIn("afterwards", output)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'afterwards' unexpectedly found in 'asyncio REPL 3.14.0a5+ (bisect/good-ecdf6b15b0c0570c3c3302ab95bdbfd3007ea941-2-g6ab5c4aa05b:6ab5c4aa05b,) [GCC 14.2.1 20250207] on linux\nUse "await" directly instead of "asyncio.run()".\nType "help", "copyright", "credits" or "license" for more information.\n>>> import asyncio\n>>> Traceback (most recent call last):\n File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 448, in result\n return self.__get_result()\n ~~~~~~~~~~~~~~~~~^^\n File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 393, in __get_result\n raise self._exception\n File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 42, in callback\n coro = func()\n File "<stdin>", line 1, in <module>\n 1 / 0; print(\'afterwards\')\n ~~^~~\nZeroDivisionError: division by zero\n>>> \nexiting asyncio REPL...\nTraceback (most recent call last):\n File "/home/pablogsal/github/python/main/Lib/runpy.py", line 198, in _run_module_as_main\n return _run_code(code, main_globals, None,\n "__main__", mod_spec)\n File "/home/pablogsal/github/python/main/Lib/runpy.py", line 88, in _run_code\n exec(code, run_globals)\n ~~~~^^^^^^^^^^^^^^^^^^^\n File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 204, in <module>\n sys.exit(return_code)\n ~~~~~~~~^^^^^^^^^^^^^\nSystemExit: 0\n>>> \n'
----------------------------------------------------------------------
Ran 13 tests in 1.400s
FAILED (failures=1)
test test_repl failed
0:00:01 load avg: 3.85 [1/1/1] test_repl failed (1 failure)
== Tests result: FAILURE ==
1 test failed:
test_repl
Total duration: 1.5 sec
Total tests: run=13 failures=1
Total test files: run=1/1 failed=1
Result: FAILURE

StanFromIreland · 2025-03-10T22:41:04Z

Weird? Only one left then would be yours a931a8b maybe it has something to do with the traceback modifications?

pablogsal · 2025-03-10T22:42:25Z

I'm pretty sure it's a931a8b

pablogsal · 2025-03-10T22:46:40Z

#131065

pablogsal · 2025-03-10T22:51:11Z

@sergey-miryanov Can you check #130805 (comment) ?

Seems the backport needs some adaptations to 3.13. Could you take a look?

sergey-miryanov · 2025-03-11T06:57:11Z

Seems the backport needs some adaptations to 3.13. Could you take a look?

@pablogsal
There is a conflict because a65366e doesn't backported to 3.13 and this PR fixes regression from this commit. Should we backport a65366e first?

pablogsal · 2025-03-11T12:56:19Z

That's tricky. That may be considered a feature so I am not sure we can. @ambv what do you think?

python#130805)" This reverts commit 7c98b06.

…pythonGH-130805) (cherry picked from commit 7c98b06) Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>

bedevere-app · 2025-05-05T18:02:22Z

GH-133462 is a backport of this pull request to the 3.13 branch.

…30805) (GH-133462) (cherry picked from commit 7c98b06) Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>

Fix support of typing unicode chars in pyrepl
3d674cf

sergey-miryanov requested review from ambv, lysnikolaou and pablogsal as code owners March 3, 2025 18:57

bedevere-appBot added the awaiting review label Mar 3, 2025

bedevere-appBot mentioned this pull request Mar 3, 2025
[Windows] New REPL doesn't allow to input non-ASCII Unicode characters #130804
Closed

sergey-miryanov added 3 commits March 4, 2025 00:45

Make mypy happy
c1f3aa8

Add assert to check if passed char is str
ef0d0ee

Merge branch 'main' intopythongh-130804-pyrepl-unicode-chars
497a0bb

encukou reviewed Mar 10, 2025
View reviewed changes

Comment threadLib/_pyrepl/base_eventqueue.py Outdated

sergey-miryanovand others added 2 commits March 11, 2025 01:06

Update Lib/_pyrepl/base_eventqueue.py
bfa2e4e
Co-authored-by: Petr Viktorin <encukou@gmail.com>

Merge branch 'main' intopythongh-130804-pyrepl-unicode-chars
c667049

pablogsal approved these changes Mar 10, 2025
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting review labels Mar 10, 2025

pablogsal added needs backport to 3.13 bugs and security fixes and removed awaiting merge labels Mar 10, 2025

Add NEWS entry
7cc16ea

pablogsal enabled auto-merge (squash) March 10, 2025 21:48

pablogsal disabled auto-merge March 10, 2025 21:48

pablogsal enabled auto-merge (squash) March 10, 2025 21:48

pablogsal merged commit 7c98b06 into python:mainMar 10, 2025

miss-islington-appBot assigned pablogsal Mar 10, 2025

sergey-miryanov mentioned this pull request Mar 29, 2025
New REPL on Windows exits when accented character is pasted/typed #131878
Closed

chris-eibl added a commit to chris-eibl/cpython that referenced this pull request Apr 26, 2025
Revert "pythongh-130804: Fix support of typing unicode chars in pyrepl (
a06c42a
python#130805)" This reverts commit 7c98b06.

chris-eibl mentioned this pull request Apr 26, 2025
gh-131878: Fix input of unicode characters with two or more code points in the REPL on Windows in vt mode #133030
Closed

This was referenced May 5, 2025
gh-131878: Fix input of unicode characters with two or more code points in new pyrepl on Windows #131901
Merged
GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440
Merged

ambv pushed a commit to ambv/cpython that referenced this pull request May 5, 2025
[3.13]pythongh-130804: Fix support of typing unicode chars in pyrepl (…
e980f8b
…pythonGH-130805) (cherry picked from commit 7c98b06) Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>

bedevere-appBot removed the needs backport to 3.13 bugs and security fixes label May 5, 2025

ambv added a commit that referenced this pull request May 5, 2025
[3.13]gh-130804: Fix support of typing unicode chars in pyrepl (GH-1…
76f52c4
…30805) (GH-133462) (cherry picked from commit 7c98b06) Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>

Uh oh!

Uh oh!

Conversation

sergey-miryanov commented Mar 3, 2025• edited by bedevere-app BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

miss-islington-appBot commented Mar 10, 2025

Uh oh!

miss-islington-appBot commented Mar 10, 2025

Uh oh!

StanFromIreland commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

StanFromIreland commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

StanFromIreland commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

StanFromIreland commented Mar 10, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

sergey-miryanov commented Mar 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Mar 11, 2025

Uh oh!

bedevere-appBot commented May 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sergey-miryanov commented Mar 3, 2025•
edited by bedevere-app Bot
Loading

StanFromIreland commented Mar 10, 2025•
edited
Loading

sergey-miryanov commented Mar 11, 2025•
edited
Loading