gh-110913: Fix WindowsConsoleIO chunking of UTF-8 text by sorgloomer · Pull Request #111007 · python/cpython

sorgloomer · 2023-10-17T21:08:42Z

Fix the loop that searches for an UTF-8 sequence boundary

Issue: Windows: WindowsConsoleIO produces mojibake replacement characters #110913

Issue: Windows: WindowsConsoleIO produces mojibake replacement characters #110913

ghost · 2023-10-17T21:08:45Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2023-10-17T21:08:51Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2023-10-17T23:43:06Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2023-10-17T23:45:10Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2023-10-17T23:46:19Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bedevere-app · 2023-10-18T00:42:05Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

zooba · 2023-10-18T12:44:16Z

I think we should have a NEWS entry for this, just to acknowledge that we changed the fix. Something like WindowsConsoleIO now correctly chunks large buffers without splitting up UTF-8 sequences

The change looks fine to me, but I'd like one more ACK before merging in case my head still isn't clear.

sorgloomer · 2023-10-19T21:48:53Z

Thanks! Added the NEWS entry.

serhiy-storchaka

Is it even possible to write tests for this?

zooba · 2023-10-19T22:29:54Z

Is it even possible to write tests for this?

Not without refactoring the entire thing (e.g. to make the actual WriteConsoleW call encoding-unaware and create a specific TextIOWrapper just for this case that does the chunking). Or to make it possible to require a WindowsConsoleIO instance to talk to a file (which then could not use WriteConsoleW I believe, though it may be possible to trick it).

It's a harmless enough issue that I think we're okay. Misunderstanding the UTF-8 format is what the problem was this time, and that's been triple checked now.

serhiy-storchaka · 2023-10-19T22:42:30Z

I'm just wondering if it's possible to get invalid UTF-8 here (using "surrogateescape"?). If yes, then we should use more sophisticated way to find the longest valid prefix of invalid sequence to meet recomendations for handling invalid UTF-8. Otherwise LGTM.

zooba · 2023-10-20T11:52:20Z

If it's undecodable by MultiByteToWideChar at this point, we'll have raised an error and never entered into this branch (wlen will be 0). And even if it did, it wouldn't be rendered correctly anyway and we aren't really making things worse by rendering it slightly more incorrectly.

Thanks for the review!

miss-islington-app · 2023-10-20T11:52:35Z

Thanks @sorgloomer for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

…H-111007) (cherry picked from commit 11312ea) Co-authored-by: Tamás Hegedűs <sorgloomer@users.noreply.github.com>

bedevere-app · 2023-10-20T11:52:48Z

GH-111108 is a backport of this pull request to the 3.12 branch.

bedevere-app · 2023-10-20T11:52:53Z

GH-111109 is a backport of this pull request to the 3.11 branch.

(cherry picked from commit 11312ea) Co-authored-by: Tamás Hegedűs <sorgloomer@users.noreply.github.com>

…H-111007)

bedevere-appBot mentioned this pull request Oct 17, 2023
Windows: WindowsConsoleIO produces mojibake replacement characters #110913
Closed

bedevere-appBot added the awaiting review label Oct 17, 2023

sorgloomer force-pushed the issue-110913-windows-console-utf8-boundary branch from b8a5110 to b66a4e4Compare October 17, 2023 23:43

sorgloomer force-pushed the issue-110913-windows-console-utf8-boundary branch from b66a4e4 to 709fb94Compare October 17, 2023 23:45

sorgloomer force-pushed the issue-110913-windows-console-utf8-boundary branch from 709fb94 to 2be4b92Compare October 17, 2023 23:46

Fix UTF-8 sequence boundary search
2b0bf5b

sorgloomer force-pushed the issue-110913-windows-console-utf8-boundary branch from 2be4b92 to 2b0bf5bCompare October 18, 2023 00:42

zooba added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Oct 18, 2023

serhiy-storchaka self-requested a review October 19, 2023 21:29

📜🤖 Added by blurb_it.
fae5147

serhiy-storchaka reviewed Oct 19, 2023
View reviewed changes

Comment threadModules/_io/winconsoleio.c Outdated

Style fix.
cd641bf

serhiy-storchaka reviewed Oct 19, 2023
View reviewed changes

Comment threadMisc/NEWS.d/next/Windows/2023-10-19-21-46-18.gh-issue-110913.CWlPfg.rst Outdated

Period.
0c8ca33

serhiy-storchaka reviewed Oct 19, 2023
View reviewed changes

serhiy-storchaka approved these changes Oct 19, 2023
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting review labels Oct 19, 2023

zooba merged commit 11312ea into python:mainOct 20, 2023

bedevere-appBot removed the awaiting merge label Oct 20, 2023

bedevere-appBot removed the needs backport to 3.12 only security fixes label Oct 20, 2023

bedevere-appBot removed the needs backport to 3.11 only security fixes label Oct 20, 2023

zooba pushed a commit that referenced this pull request Oct 20, 2023
gh-110913: Fix WindowsConsoleIO chunking of UTF-8 text (GH-111007)
69bcaf7
(cherry picked from commit 11312ea) Co-authored-by: Tamás Hegedűs <sorgloomer@users.noreply.github.com>

zooba pushed a commit that referenced this pull request Oct 20, 2023
gh-110913: Fix WindowsConsoleIO chunking of UTF-8 text (GH-111007)
dfc8d7a
(cherry picked from commit 11312ea) Co-authored-by: Tamás Hegedűs <sorgloomer@users.noreply.github.com>

aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
pythongh-110913: Fix WindowsConsoleIO chunking of UTF-8 text (pythonG…
18f2add
…H-111007)

Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
pythongh-110913: Fix WindowsConsoleIO chunking of UTF-8 text (pythonG…
7dec182
…H-111007)

Uh oh!

Uh oh!

Conversation

sorgloomer commented Oct 17, 2023• edited by bedevere-app BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Oct 17, 2023• edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-appBot commented Oct 17, 2023

Uh oh!

bedevere-appBot commented Oct 17, 2023

Uh oh!

bedevere-appBot commented Oct 17, 2023

Uh oh!

bedevere-appBot commented Oct 17, 2023

Uh oh!

bedevere-appBot commented Oct 18, 2023

Uh oh!

zooba commented Oct 18, 2023

Uh oh!

sorgloomer commented Oct 19, 2023

Uh oh!

Uh oh!

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

zooba commented Oct 19, 2023

Uh oh!

serhiy-storchaka commented Oct 19, 2023

Uh oh!

zooba commented Oct 20, 2023

Uh oh!

miss-islington-appBot commented Oct 20, 2023

Uh oh!

bedevere-appBot commented Oct 20, 2023

Uh oh!

bedevere-appBot commented Oct 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sorgloomer commented Oct 17, 2023•
edited by bedevere-app Bot
Loading

ghost commented Oct 17, 2023•
edited by ghost
Loading