gh-103578: Fix pdb reading code with non-utf8 encoding by gaogaotiantian · Pull Request #103581 · python/cpython

gaogaotiantian · 2023-04-16T18:26:59Z

pdb should use io.open_code to open code to avoid encoding issue.

Issue: python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578

hauntsaninja

Thanks for this PR!

We should probably mention in the news entry that because we use io.open_code, we'll now trigger auditing ~~events~~ hooks

gaogaotiantian · 2023-04-24T05:06:19Z

Thanks for this PR!
We should probably mention in the news entry that because we use io.open_code, we'll now trigger auditing events

io.open() also raise auditing events. According to the docs,

However, assuming that path is a str and an absolute path, open_code(path) should always behave the same as open(path, 'rb')

io.open_code() should behave the same as io.open(path, 'rb'), so is there an auditing event difference?

hauntsaninja · 2023-04-24T05:10:26Z

Ah, I meant the PyFile_SetOpenCodeHook() hook from PEP 578. I'm not sure what the best phrase is, but it was described as auditing event in #17127. Maybe "auditing hook" as per the PEP?
(that PR got regressed in #26992, this PR fixes :-) )

gaogaotiantian · 2023-04-24T05:27:59Z

So it seems like changing it to io.open_code() makes it possible to call into the hook set by PyFile_SetOpenCodeHook()? Is that something expected in the news entry?

hauntsaninja · 2023-04-24T05:32:51Z

Yup! I think can't hurt to mention it in the news entry, since it is a (good) change in behaviour and one that seems potentially noteworthy.

gaogaotiantian · 2023-04-24T05:38:09Z

Yup! I think can't hurt to mention it in the news entry, since it is a (good) change in behaviour and one that seems potentially noteworthy.

I updated the news entry. Let me know if you think that requires extra polish!

hauntsaninja

Looks great, thanks!

miss-islington · 2023-04-26T05:04:55Z

Thanks @gaogaotiantian for the PR, and @hauntsaninja for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

…H-103581) `pdb` should use `io.open_code` to open code to avoid encoding issue. (cherry picked from commit 31acfd7) Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>

bedevere-bot · 2023-04-26T05:05:06Z

GH-103867 is a backport of this pull request to the 3.11 branch.

) (#103867) `pdb` should use `io.open_code` to open code to avoid encoding issue. (cherry picked from commit 31acfd7) Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>

…103581) `pdb` should use `io.open_code` to open code to avoid encoding issue.

miss-islington · 2023-04-27T19:30:31Z

Thanks @gaogaotiantian for the PR, and @hauntsaninja for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2023-04-27T19:30:31Z

Thanks @gaogaotiantian for the PR, and @hauntsaninja for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

miss-islington · 2023-04-27T19:30:31Z

Thanks @gaogaotiantian for the PR, and @hauntsaninja for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2023-04-27T19:30:32Z

Sorry @gaogaotiantian and @hauntsaninja, I had trouble checking out the 3.8 backport branch.
Please retry by removing and re-adding the "needs backport to 3.8" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 31acfd78a0810f84898d36a8289e407d3754b823 3.8

miss-islington · 2023-04-27T19:30:48Z

Sorry, @gaogaotiantian and @hauntsaninja, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 31acfd78a0810f84898d36a8289e407d3754b823 3.9

miss-islington · 2023-04-27T19:30:49Z

Sorry @gaogaotiantian and @hauntsaninja, I had trouble checking out the 3.10 backport branch.
Please retry by removing and re-adding the "needs backport to 3.10" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 31acfd78a0810f84898d36a8289e407d3754b823 3.10

miss-islington · 2023-04-27T19:31:45Z

Thanks @gaogaotiantian for the PR, and @hauntsaninja for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2023-04-27T19:31:47Z

Sorry, @gaogaotiantian and @hauntsaninja, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 31acfd78a0810f84898d36a8289e407d3754b823 3.10

zooba · 2023-04-27T19:34:26Z

No backport needed - this was fine in 3.8-3.10 and was regressed in 3.11.

Fix pdb reading code with non-utf8 encoding issue
abbc7d5

bedevere-bot mentioned this pull request Apr 16, 2023
python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578
Closed

bedevere-bot added the awaiting review label Apr 16, 2023

blurb-itBotand others added 2 commits April 16, 2023 18:29

📜🤖 Added by blurb_it.
cad5b05

Add python ext check
aba11f9

hauntsaninja reviewed Apr 24, 2023
View reviewed changes

hauntsaninja added the needs backport to 3.11 only security fixes label Apr 24, 2023

Update 2023-04-16-18-29-04.gh-issue-103578.fly1wc.rst
1d8fc05

hauntsaninja approved these changes Apr 24, 2023
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Apr 24, 2023

Merge branch 'main' into pdb-encoding-issue
f2a96fc

hauntsaninja merged commit 31acfd7 into python:mainApr 26, 2023

bedevere-bot removed the awaiting merge label Apr 26, 2023

bedevere-bot removed the needs backport to 3.11 only security fixes label Apr 26, 2023

gaogaotiantian deleted the pdb-encoding-issue branch April 26, 2023 05:13

itamaro pushed a commit to itamaro/cpython that referenced this pull request Apr 26, 2023
pythongh-103578: Fix pdb reading code with non-utf8 encoding (python#…
8c02f95
…103581) `pdb` should use `io.open_code` to open code to avoid encoding issue.

gaogaotiantian mentioned this pull request Apr 27, 2023
trace.__main__ does not use io.open_code #103935
Closed

zooba added needs backport to 3.8 needs backport to 3.10 only security fixes labels Apr 27, 2023

miss-islington assigned hauntsaninja Apr 27, 2023

zooba added needs backport to 3.10 only security fixes and removed needs backport to 3.10 only security fixes labels Apr 27, 2023

zooba removed needs backport to 3.8 needs backport to 3.10 only security fixes labels Apr 27, 2023

Uh oh!

Uh oh!

Conversation

gaogaotiantian commented Apr 16, 2023• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hauntsaninja left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaogaotiantian commented Apr 24, 2023

Uh oh!

hauntsaninja commented Apr 24, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaogaotiantian commented Apr 24, 2023

Uh oh!

hauntsaninja commented Apr 24, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaogaotiantian commented Apr 24, 2023

Uh oh!

hauntsaninja left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Apr 26, 2023

Uh oh!

bedevere-bot commented Apr 26, 2023

Uh oh!

miss-islington commented Apr 27, 2023

Uh oh!

miss-islington commented Apr 27, 2023

Uh oh!

miss-islington commented Apr 27, 2023

Uh oh!

miss-islington commented Apr 27, 2023

Uh oh!

miss-islington commented Apr 27, 2023

Uh oh!

miss-islington commented Apr 27, 2023

Uh oh!

miss-islington commented Apr 27, 2023

Uh oh!

miss-islington commented Apr 27, 2023

Uh oh!

zooba commented Apr 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gaogaotiantian commented Apr 16, 2023•
edited by bedevere-bot
Loading

hauntsaninja left a comment •
edited
Loading

hauntsaninja commented Apr 24, 2023•
edited
Loading

hauntsaninja commented Apr 24, 2023•
edited
Loading