Add check if `str(handle)` correctly converted the object, and throw py::error_already_set if not by YannickJadoul · Pull Request #2392 · pybind/pybind11

YannickJadoul · 2020-08-14T00:25:18Z

Only problematic on Python 2, I think. Discovered here: #2380 (comment)

YannickJadoul · 2020-08-14T00:30:10Z

Also note that by checking always checking the result after calling raw_ptr, we can more the if (!str_value) throw error_already_set(); inside the Python 2-only preprocessor block, and get rid of one full pointer comparison (!) on Python 3, when calling the constructor from py::object! ;-)

YannickJadoul · 2020-08-14T00:36:01Z

Of course. I tried this locally based on #2380, but detail::PyUnicode_Check_Permissive first still needs to die, such that py::str cannot contain a bytes-object anymore!

YannickJadoul · 2020-08-14T00:40:07Z

Rebased onto #2380, but waiting for #2380 to get merged before this can get rebased and merged.

bstaletic

Looks good, apart from that sys.version.major thing.

henryiii · 2020-08-16T17:44:12Z

@YannickJadoul if you used pre-commit install this couldn't happen. 😜

henryiii · 2020-08-16T17:46:39Z

This might benefit from being rebased after #2397

YannickJadoul · 2020-08-16T21:06:28Z

@YannickJadoul if you used pre-commit install this couldn't happen. stuck_out_tongue_winking_eye

OK, you win this one ;-)

…py::error_already_set if not

YannickJadoul · 2020-08-16T21:35:40Z

Fixed this. It's quite a minor PR, so a few more quick checks (@bstaletic & @henryiii, the things you mentioned should be fixed).
@rwgk, maybe you should give this a quick final check (since it involves py::str), and merge into str-bytes-cleanup at will if you're happy with it?

wjakob · 2020-08-17T08:25:23Z

LGTM.

rwgk

Looks great, thanks Yannick!
I'll hold off merging only because I'm still unclear why the current str-bytes-cleanup branch fails to pass some tests in the Google environment (I didn't get a chance to work on it at all yesterday).
(For completeness: the failing tests are NOT pybind11 unit tests.)

rwgk · 2020-08-17T17:53:05Z

Looks great, thanks Yannick!
I'll hold off merging only because I'm still unclear why the current str-bytes-cleanup branch fails to pass some tests in the Google environment (I didn't get a chance to work on it at all yesterday).
(For completeness: the failing tests are NOT pybind11 unit tests.)

For future reference: the test failure was due to accidental str vs bytes mixup, masked by the double-nature of pybind11::str.

… py::error_already_set if not. Similar to pybind#2392, but does not depend on pybind#2409. Splitting out this PR from pybind#2409 to make that PR as simple as possible. Net effects of this PR: * Adds missing test coverage. * Changes TypeError to UnicodeDecodeError for Python 2. This PR has two commits. Please do not squash, to make the behavior change obvious in the commit history.

@rwgk

…py::error_already_set if not (bis) (#2477) * Add check if `str(handle)` correctly converted the object, and throw py::error_already_set if not * Fix tests on Python 3 * Apply @rwgk's fixes to cherry-picked commits from #2392

This was referenced Aug 14, 2020
Changing pybind11::str to only hold PyUnicodeObject (NOT also bytes). #2380
Merged
Tracking PR: intermediate state of completed str/bytes cleaning up PRs #2348
Closed

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from 235168a to d58307dCompare August 14, 2020 00:39

YannickJadoul commented Aug 14, 2020
View reviewed changes

Comment threadtests/test_pytypes.py Outdated

rwgk force-pushed the str-bytes-cleanup branch 2 times, most recently from f38fafc to 6cd4552Compare August 14, 2020 22:03

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch 3 times, most recently from 7eee06e to 366f713Compare August 15, 2020 21:51

YannickJadoul self-assigned this Aug 15, 2020

bstaletic reviewed Aug 16, 2020
View reviewed changes

rwgk force-pushed the str-bytes-cleanup branch from 1d553e4 to 0b49467Compare August 16, 2020 17:24

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from 366f713 to 6452317Compare August 16, 2020 17:41

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from 6452317 to b25eaceCompare August 16, 2020 17:47

henryiii reviewed Aug 16, 2020
View reviewed changes

Comment threadtests/test_pytypes.py Outdated

henryiii reviewed Aug 16, 2020
View reviewed changes

Comment threadinclude/pybind11/cast.h

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from b25eace to f7d376cCompare August 16, 2020 21:13

YannickJadoul force-pushed the str-bytes-cleanup branch from 0b49467 to 0ab4b97Compare August 16, 2020 21:14

Add check if str(handle) correctly converted the object, and throw …
9e900fa
…py::error_already_set if not

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from f7d376c to 8e53b83Compare August 16, 2020 21:29

Fix tests on Python 3
3dbafa6

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from 8e53b83 to 3dbafa6Compare August 16, 2020 21:30

bstaletic approved these changes Aug 17, 2020
View reviewed changes

rwgk approved these changes Aug 17, 2020
View reviewed changes

rwgk merged commit 7ae6bb2 into pybind:str-bytes-cleanupAug 17, 2020

YannickJadoul deleted the fix-python2-str-malformed_utf8 branch August 17, 2020 22:08

henryiii added the bug label Aug 19, 2020

rwgk mentioned this pull request Sep 8, 2020
Adds check if str(handle) correctly converted the object, and throw py::error_already_set if not. #2473
Closed

rwgk added a commit to rwgk/pybind11 that referenced this pull request Sep 10, 2020
Backing out changes originating frompybind#2392.
6f270a3

YannickJadoul added a commit to YannickJadoul/pybind11 that referenced this pull request Sep 10, 2020
Apply@rwgk's fixes to cherry-picked commits frompybind#2392
76f3ebe

YannickJadoul mentioned this pull request Sep 10, 2020
Add check if str(handle) correctly converted the object, and throw py::error_already_set if not (bis) #2477
Merged

YannickJadoul added a commit to YannickJadoul/pybind11 that referenced this pull request Sep 11, 2020
Apply@rwgk's fixes to cherry-picked commits frompybind#2392
055f86f

rwgk added a commit to rwgk/pybind11 that referenced this pull request Sep 16, 2020
Backing out changes originating frompybind#2392.
bd1d77b

rwgk added a commit to rwgk/pybind11 that referenced this pull request Sep 16, 2020
Backing out changes originating frompybind#2392.
21376dd

rwgk added a commit to rwgk/pybind11 that referenced this pull request Sep 27, 2020
Backing out changes originating frompybind#2392.
1bdb07d

rwgk added a commit to rwgk/pybind11 that referenced this pull request Oct 13, 2020
Backing out changes originating frompybind#2392.
1152249

rwgk added a commit to rwgk/pybind11 that referenced this pull request Oct 18, 2020
Backing out changes originating frompybind#2392.
924502a

rwgk added a commit to rwgk/pybind11 that referenced this pull request Oct 29, 2020
Backing out changes originating frompybind#2392.
df6e916

rwgk mentioned this pull request Feb 10, 2023
FWD pybind11 google/pybind11clif#2392
Closed

Navigation Menu

Uh oh!

Add check if `str(handle)` correctly converted the object, and throw py::error_already_set if not - #2392

Add check if `str(handle)` correctly converted the object, and throw py::error_already_set if not#2392
rwgk merged 2 commits into
pybind:str-bytes-cleanupfrom
YannickJadoul:fix-python2-str-malformed_utf8

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

Uh oh!

bstaletic left a comment

Uh oh!

henryiii commented Aug 16, 2020•
edited
Loading

Uh oh!

henryiii commented Aug 16, 2020•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

YannickJadoul commented Aug 16, 2020

Uh oh!

YannickJadoul commented Aug 16, 2020

Uh oh!

wjakob commented Aug 17, 2020

Uh oh!

rwgk left a comment

Uh oh!

rwgk commented Aug 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

Uh oh!

bstaletic left a comment

Choose a reason for hiding this comment

Uh oh!

henryiii commented Aug 16, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Aug 16, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YannickJadoul commented Aug 16, 2020

Uh oh!

YannickJadoul commented Aug 16, 2020

Uh oh!

wjakob commented Aug 17, 2020

Uh oh!

rwgk left a comment

Choose a reason for hiding this comment

Uh oh!

rwgk commented Aug 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

henryiii commented Aug 16, 2020•
edited
Loading

henryiii commented Aug 16, 2020•
edited
Loading