gh-104252: Immortalize Py_EMPTY_KEYS by ericsnowcurrently · Pull Request #104253 · python/cpython

ericsnowcurrently · 2023-05-06T23:59:09Z

This was missed in gh-19474. It matters for with a per-interpreter GIL since PyDictKeysObject.dk_refcnt breaks isolation and leads to races.

cc @eduardo-elizondo

Issue: Immortalize Py_EMPTY_KEYS #104252

sunmy2019

There are other places with direct changes to dk_refcnt. They are not guarded.

cpython/Objects/dictobject.c

Line 639 in 4729383

dk->dk_refcnt=1;

cpython/Objects/dictobject.c

Lines 822 to 828 in 4729383

/* Since we copied the keys table we now have an extra reference
 in the system. Manually call increment _Py_RefTotal to signal that
 we have it now; calling dictkeys_incref would be an error as
 keys->dk_refcnt is already set to 1 (after memcpy). */
#ifdefPy_REF_DEBUG
_Py_IncRefTotal(_PyInterpreterState_GET()); 
#endif

cpython/Objects/dictobject.c

Lines 1530 to 1538 in 4729383

// We can not use free_keys_object here because key's reference
// are moved already.
#ifdefPy_REF_DEBUG
_Py_DecRefTotal(_PyInterpreterState_GET()); 
#endif
if (oldkeys==Py_EMPTY_KEYS) { 
oldkeys->dk_refcnt--; 
assert(oldkeys->dk_refcnt>0); 
 } 

ericsnowcurrently · 2023-05-08T13:47:12Z

@sunmy2019, thanks for pointing out the other uses of dk_refcnt. I'll fix that.

[done]

methane · 2023-05-09T01:56:53Z

Please remove these lines too:

cpython/Objects/dictobject.c

Line 849 in 58585a9

dictkeys_incref(Py_EMPTY_KEYS);

cpython/Objects/dictobject.c

Line 1347 in 58585a9

dictkeys_decref(interp, Py_EMPTY_KEYS);

And Move _Py_DecRefTotal() into if (oldkeys != Py_EMPTY_KEYS) block.

cpython/Objects/dictobject.c

Lines 1545 to 1548 in 58585a9

#ifdefPy_REF_DEBUG
_Py_DecRefTotal(_PyInterpreterState_GET()); 
#endif
if (oldkeys!=Py_EMPTY_KEYS) { 

ericsnowcurrently · 2023-05-09T03:35:51Z

Thanks for the feedback. I'll clean things up tomorrow.

bedevere-bot · 2023-05-09T18:51:10Z

🤖 New build scheduled with the buildbot fleet by @sunmy2019 for commit 5aabfd7 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

sunmy2019

LGTM! now it's much cleaner

ericsnowcurrently · 2023-05-10T13:42:49Z

Thanks for the reviews!

* main: pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182) pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364) pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539) pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292) pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202) pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013) pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355) pythongh-103960: Dark mode: invert image brightness (python#103983) pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253) pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354) pythongh-101819: Harden _io init (python#104352) pythongh-103247: clear the module cache in a test in test_importlib/extensions/test_loader.py (pythonGH-104226) pythongh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (python#103849) pythongh-74895: adjust tests to work on Solaris (python#104326) pythongh-101819: Refactor _io in preparation for module isolation (python#104334) pythongh-90953: Don't use deprecated AST nodes in clinic.py (python#104322) pythongh-102327: Extend docs for "url" and "headers" parameters to HTTPConnection.request() pythongh-104328: Fix typo in ``typing.Generic`` multiple inheritance error message (python#104335)

* main: (27 commits) pythongh-87849: fix SEND specialization family definition (pythonGH-104268) pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384) pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367) pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369) pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383) pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287) pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185) pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368) pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141) pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182) pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364) pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539) pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292) pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202) pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013) pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355) pythongh-103960: Dark mode: invert image brightness (python#103983) pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253) pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354) pythongh-101819: Harden _io init (python#104352) ...

eduardo-elizondo · 2023-06-19T15:32:56Z

Nice fix! I just got around to looking at this PR, it was weird thinking about the PyDictKeysObject since they were not PyObjects, so I just let them be. That being said, this LGTM specially if we are trying to share as much state as possible.

ericsnowcurrently added 2 commits May 6, 2023 17:55

Factor out _dict_state_INIT.
5bc2469

Immortalize Py_EMPTY_KEYS.
e472d94

ericsnowcurrently added the skip news label May 6, 2023

ericsnowcurrently requested a review from markshannon May 6, 2023 23:59

ericsnowcurrently requested a review from methane as a code owner May 6, 2023 23:59

bedevere-bot mentioned this pull request May 6, 2023
Immortalize Py_EMPTY_KEYS #104252
Closed

bedevere-bot added the awaiting core review label May 6, 2023

ericsnowcurrently requested review from methane and removed request for methane May 6, 2023 23:59

sunmy2019 requested changes May 7, 2023
View reviewed changes

Comment threadObjects/dictobject.c Outdated

ericsnowcurrently added 2 commits May 8, 2023 07:57

Handle other uses of dk_refcnt.
87351e0

Add a note.
f5da088

methane reviewed May 8, 2023
View reviewed changes

Comment threadObjects/dictobject.c Outdated

sunmy2019 reviewed May 8, 2023
View reviewed changes

Comment threadInclude/internal/pycore_dict_state.h Outdated

ericsnowcurrently added 2 commits May 8, 2023 13:25

Drop an old line.
db8612f

Drop an unnecessary condition.
58585a9

methane reviewed May 9, 2023
View reviewed changes

Comment threadObjects/dictobject.c Outdated

methane reviewed May 9, 2023
View reviewed changes

Comment threadObjects/dictobject.c

ericsnowcurrently added 4 commits May 9, 2023 10:11

Drop an unnecessary assert.
6b5c952

Do not bother inc/dec-refing Py_EMPTY_KEYS.
bdbce5f

Do not update Py_Reftotal for Py_EMPTY_KEYS.
aab493a

Clarify the comment about dictkeys_incref() and dictkeys_decref().
5aabfd7

ericsnowcurrently requested a review from sunmy2019 May 9, 2023 17:45

sunmy2019 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 9, 2023

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 9, 2023

methane approved these changes May 10, 2023
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels May 10, 2023

This comment was marked as duplicate.
Sign in to view

sunmy2019 approved these changes May 10, 2023
View reviewed changes

ericsnowcurrently merged commit b8f7ab5 into python:mainMay 10, 2023

bedevere-bot removed the awaiting merge label May 10, 2023

ericsnowcurrently deleted the immortalize-empty-keys branch May 10, 2023 13:28

Navigation Menu

Uh oh!

Uh oh!

gh-104252: Immortalize Py_EMPTY_KEYS - #104253

gh-104252: Immortalize Py_EMPTY_KEYS#104253
ericsnowcurrently merged 10 commits into
python:mainfrom
ericsnowcurrently:immortalize-empty-keys

ericsnowcurrently commented May 6, 2023•
edited by bedevere-bot
Loading

Uh oh!

sunmy2019 left a comment •
edited
Loading

Uh oh!

Uh oh!

ericsnowcurrently commented May 8, 2023•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

methane commented May 9, 2023

Uh oh!

ericsnowcurrently commented May 9, 2023

Uh oh!

bedevere-bot commented May 9, 2023

Uh oh!

This comment was marked as duplicate.

Uh oh!

sunmy2019 left a comment

Uh oh!

ericsnowcurrently commented May 10, 2023

Uh oh!

eduardo-elizondo commented Jun 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	/* Since we copied the keys table we now have an extra reference
	in the system. Manually call increment _Py_RefTotal to signal that
	we have it now; calling dictkeys_incref would be an error as
	keys->dk_refcnt is already set to 1 (after memcpy). */
	#ifdefPy_REF_DEBUG
	_Py_IncRefTotal(_PyInterpreterState_GET());
	#endif

	// We can not use free_keys_object here because key's reference
	// are moved already.
	#ifdefPy_REF_DEBUG
	_Py_DecRefTotal(_PyInterpreterState_GET());
	#endif
	if (oldkeys==Py_EMPTY_KEYS) {
	oldkeys->dk_refcnt--;
	assert(oldkeys->dk_refcnt>0);
	}

Uh oh!

Uh oh!

Conversation

ericsnowcurrently commented May 6, 2023• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sunmy2019 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ericsnowcurrently commented May 8, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

methane commented May 9, 2023

Uh oh!

ericsnowcurrently commented May 9, 2023

Uh oh!

bedevere-bot commented May 9, 2023

Uh oh!

This comment was marked as duplicate.

Uh oh!

sunmy2019 left a comment

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently commented May 10, 2023

Uh oh!

eduardo-elizondo commented Jun 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ericsnowcurrently commented May 6, 2023•
edited by bedevere-bot
Loading

sunmy2019 left a comment •
edited
Loading

ericsnowcurrently commented May 8, 2023•
edited
Loading