Unicode characters ≥ 0x10000 cannot be inputted/behaves unusually at the REPL terminal.

[haydenwong7bm]

Bug report

Bug description:

When the machine locale is set to UTF-8, when inputting a Unicode character ≥ 0x10000:
In CPython 3.13.5:
https://github.com/user-attachments/assets/7777b063-76fe-4929-b854-cae7d61807d2
In Cpython 3.14.0b4:

>>> Traceback (most recent call last):
 File "*\Python\Python314\Lib\_pyrepl\readline.py", line 394, in multiline_input
 return reader.readline()
 ~~~~~~~~~~~~~~~^^
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 748, in readline
 self.handle1()
 ~~~~~~~~~~~~^^
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 731, in handle1
 self.do_cmd(cmd)
 ~~~~~~~~~~~^^^^^
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 661, in do_cmd
 self.refresh()
 ~~~~~~~~~~~~^^
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 638, in refresh
 self.screen = self.calc_screen()
 ~~~~~~~~~~~~~~~~^^
 File "*\Python\Python314\Lib\_pyrepl\completing_reader.py", line 261, in calc_screen
 screen = super().calc_screen()
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 315, in calc_screen
 colors = list(gen_colors(self.get_unicode()))
 File "*\Python\Python314\Lib\_pyrepl\utils.py", line 108, in gen_colors
 for color in gen_colors_from_token_stream(gen, line_lengths):
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
 File "*\Python\Python314\Lib\_pyrepl\utils.py", line 168, in gen_colors_from_token_stream
 for prev_token, token, next_token in token_window:
 ^^^^^^^^^^^^
 File "*\Python\Python314\Lib\_pyrepl\utils.py", line 363, in prev_next_window
 window = deque((None, next(iterator)), maxlen=3)
 ~~~~^^^^^^^^^^
 File "*\Python\Python314\Lib\tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
 for info in it:
 ^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
>>> Traceback (most recent call last):
 File "*\Python\Python314\Lib\_pyrepl\readline.py", line 394, in multiline_input
 return reader.readline()
 ~~~~~~~~~~~~~~~^^
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 748, in readline
 self.handle1()
 ~~~~~~~~~~~~^^
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 731, in handle1
 self.do_cmd(cmd)
 ~~~~~~~~~~~^^^^^
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 661, in do_cmd
 self.refresh()
 ~~~~~~~~~~~~^^
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 638, in refresh
 self.screen = self.calc_screen()
 ~~~~~~~~~~~~~~~~^^
 File "*\Python\Python314\Lib\_pyrepl\completing_reader.py", line 261, in calc_screen
 screen = super().calc_screen()
 File "*\Python\Python314\Lib\_pyrepl\reader.py", line 315, in calc_screen
 colors = list(gen_colors(self.get_unicode()))
 File "*\Python\Python314\Lib\_pyrepl\utils.py", line 108, in gen_colors
 for color in gen_colors_from_token_stream(gen, line_lengths):
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
 File "*\Python\Python314\Lib\_pyrepl\utils.py", line 168, in gen_colors_from_token_stream
 for prev_token, token, next_token in token_window:
 ^^^^^^^^^^^^
 File "*\Python\Python314\Lib\_pyrepl\utils.py", line 363, in prev_next_window
 window = deque((None, next(iterator)), maxlen=3)
 ~~~~^^^^^^^^^^
 File "*\Python\Python314\Lib\tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
 for info in it:
 ^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed

Two surrogates were "inputted" and so two UnicodeEncodeErrors.

CPython versions tested on:

3.13, 3.14

Operating systems tested on:

Windows

Linked PRs

• gh-136595: Normalize surrogate pairs in REPL input to fix UnicodeEnco… #136639

[WFLing-seaer]

I've met this problem too when trying to paste 𣳆㶔𤃫𣸎 to πython running on WinTerminal.
btw in CLI which python command starts pasting the word won't cause crash though it can't be fully pasted.
怎么哪里都有屏被轰坏.jpg

[WFLing-seaer]

I've met this problem too when trying to paste 𣳆㶔𤃫𣸎 to πython running on WinTerminal. btw in CLI which python command starts pasting the word won't cause crash though it can't be fully pasted. 怎么哪里都有屏被轰坏.jpg

this indicates there must be something wwrong with πython+WinTerminal. maybe compatibility problems.