bpo-46615: Don't crash when set operations mutate the sets by sweeneyde · Pull Request #31120 · python/cpython

sweeneyde · 2022-02-04T04:24:41Z

Users of set_next should call Py_INCREF on the resulting set keys to avoid use-after-free.

https://bugs.python.org/issue46615

sweeneyde · 2022-02-04T04:41:30Z

Checked for refleaks:

PS C:\Users\sween\Source\Repos\cpython2\cpython> .\python.bat -m test test_set -R3:3 Running Debug|x64 interpreter...
0:00:00 Run tests sequentially
0:00:00 [1/1] test_set
beginning 6 repetitions
123456
......
== Tests result: SUCCESS ==
1 test OK.
Total duration: 26.3 sec
Tests result: SUCCESS

tim-one

Looks like the right approach to me, but Raymond is much more familiar with this code. Added a comment about something that confused me in the new tests.

…esent (no TypeErrors)

sweeneyde · 2022-02-06T01:16:07Z

It's unclear to me why this doesn't crash with the added test cases:

cpython/Objects/setobject.c

Lines 905 to 908 in 96b344c

while (_PyDict_Next(other, &pos, &key, &value, &hash)) { 
if (set_add_entry(so, key, hash)) 
return-1; 
 } 

tim-one · 2022-02-06T01:24:44Z

It's unclear to me why this doesn't crash

None of these ever "crashed" on my box (Windows). Instead they weirded out silently in various ways (seemingly infinite loops, or the process just stopped without an error exit code). Just about anything can happen when mucking with freed memory - including no visible symptoms at all. Try running in a mode that checks runtime memory use (like valgrind)?

sweeneyde · 2022-02-06T05:16:24Z

Aha: set_add_entry already does Py_INCREF(key) internally.

JelleZijlstra · 2022-02-11T01:57:50Z

+ def __eq__(self, other):
+ if not enabled:
+ return False
+ if randrange(20) == 0:


Might be better to use something like a counter, so that these tests behave consistently. The current behavior might make the test pass or fail inconsistently depending on what the RNG produces.

I'm personally okay with randomized tests, though I know others might feel differently. The test should pass for anything the RNG can give, and I'm personally more made more confident in code tested in millions of random situations, which ideally hit all of the corner cases I couldn't think of.

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

serhiy-storchaka

LGTM.

miss-islington · 2022-02-11T20:08:43Z

Thanks @sweeneyde for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2022-02-11T20:08:43Z

Thanks @sweeneyde for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

miss-islington · 2022-02-11T20:09:04Z

Sorry, @sweeneyde, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 4a66615ba736f84eadf9456bfd5d32a94cccf117 3.9

bedevere-bot · 2022-02-11T20:09:06Z

GH-31284 is a backport of this pull request to the 3.10 branch.

…31120) Ensure strong references are acquired whenever using `set_next()`. Added randomized test cases for `__eq__` methods that sometimes mutate sets when called. (cherry picked from commit 4a66615) Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>

Ensure strong references are acquired whenever using `set_next()`. Added randomized test cases for `__eq__` methods that sometimes mutate sets when called. (cherry picked from commit 4a66615) Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>

…31120) Ensure strong references are acquired whenever using `set_next()`. Added randomized test cases for `__eq__` methods that sometimes mutate sets when called. (cherry picked from commit 4a66615)

bedevere-bot · 2022-02-13T02:47:13Z

GH-31312 is a backport of this pull request to the 3.9 branch.

…GH-31312) Ensure strong references are acquired whenever using `set_next()`. Added randomized test cases for `__eq__` methods that sometimes mutate sets when called. (cherry picked from commit 4a66615)

…31120) (pythonGH-31312) Ensure strong references are acquired whenever using `set_next()`. Added randomized test cases for `__eq__` methods that sometimes mutate sets when called. (cherry picked from commit 4a66615)

Don't crash when set operations mutate the sets
8f858b0

sweeneyde requested a review from rhettinger as a code owner February 4, 2022 04:24

the-knights-who-say-ni added the CLA signed label Feb 4, 2022

bedevere-bot added the awaiting review label Feb 4, 2022

blurb-itBotand others added 2 commits February 4, 2022 04:33

📜🤖 Added by blurb_it.
ba679c2

make new test, don't repeat ops on empty sets
780bca9

sweeneyde added 2 commits February 4, 2022 00:06

Paramerize tests by type
16e990d

test xor as well
0f42bb8

tim-one reviewed Feb 4, 2022
View reviewed changes

Comment threadLib/test/test_set.py Outdated

sweeneyde added 2 commits February 4, 2022 01:22

Catch RuntimeError explicitly. Only test methods that are actually pr…
c83c426
…esent (no TypeErrors)

More consistent naming of TestCase classes
a8951b0

erlend-aasland reviewed Feb 4, 2022
View reviewed changes

Comment threadObjects/setobject.c

erlend-aasland reviewed Feb 4, 2022
View reviewed changes

Comment threadObjects/setobject.c Outdated

erlend-aasland reviewed Feb 4, 2022
View reviewed changes

Comment threadObjects/setobject.c

sweeneyde added 2 commits February 4, 2022 05:56

Factor out a common DECREF
44e40c2

Factor out another common DECREF
b4091fd

sweeneyde closed this Feb 9, 2022

sweeneyde reopened this Feb 9, 2022

JelleZijlstra reviewed Feb 11, 2022
View reviewed changes

sweeneydeand others added 3 commits February 10, 2022 21:22

Update Objects/setobject.c
0858513
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

Re-use key variable
2ab5f4e

newline in NEWS entry
c16cba3

serhiy-storchaka approved these changes Feb 11, 2022
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Feb 11, 2022

sweeneyde merged commit 4a66615 into python:mainFeb 11, 2022

bedevere-bot removed the awaiting merge label Feb 11, 2022

serhiy-storchaka added needs backport to 3.9 needs backport to 3.10 only security fixes type-bug An unexpected behavior, bug, or error labels Feb 11, 2022

miss-islington assigned sweeneyde Feb 11, 2022

bedevere-bot removed the needs backport to 3.10 only security fixes label Feb 11, 2022

bedevere-bot removed the needs backport to 3.9 label Feb 13, 2022

sweeneyde deleted the set_crashers branch February 14, 2022 22:20

sweeneyde mentioned this pull request May 18, 2022
Use-after-free by mutating set during set operations #90773
Closed

Uh oh!

Uh oh!

Conversation

sweeneyde commented Feb 4, 2022• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sweeneyde commented Feb 4, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tim-one left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sweeneyde commented Feb 6, 2022

Uh oh!

tim-one commented Feb 6, 2022

Uh oh!

sweeneyde commented Feb 6, 2022

Uh oh!

Uh oh!

JelleZijlstraFeb 11, 2022

Choose a reason for hiding this comment

Uh oh!

sweeneydeFeb 11, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Feb 11, 2022

Uh oh!

miss-islington commented Feb 11, 2022

Uh oh!

miss-islington commented Feb 11, 2022

Uh oh!

bedevere-bot commented Feb 11, 2022

Uh oh!

bedevere-bot commented Feb 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

sweeneyde commented Feb 4, 2022•
edited by bedevere-bot
Loading

sweeneyde commented Feb 4, 2022•
edited
Loading