bpo-42972: Fully implement GC protocol for functools keywrapper and partial types by erlend-aasland · Pull Request #26363 · python/cpython

erlend-aasland · 2021-05-25T20:14:03Z

https://bugs.python.org/issue42972

shihai1991

Thanks for your update :)

erlend-aasland · 2021-05-27T05:44:16Z

Thanks for your update :)

Thank you so much for reviewing and pointing out the weakness in the LRU cache :)

vstinner · 2021-05-27T13:51:51Z

+ Py_VISIT(pto->args);
+ Py_VISIT(pto->kw);
+ Py_VISIT(pto->dict);
+ Py_VISIT(Py_TYPE(pto));


I would prefer to always start by visiting the type in all traverse functions. So same remark for other traverse functions.
I like to look into partialobject structure and visit them by their definition order. Technically, ob_type is the first one :-D

Sure, I'll apply to all remaining PR's :)

vstinner · 2021-05-27T14:04:30Z

 {Py_tp_members, partial_memberlist},
 {Py_tp_getset, partial_getsetlist},
 {Py_tp_new, partial_new},
- {Py_tp_free, PyObject_GC_Del},


Would it be possible to keep it and maybe only change that in 3.11 (in a separated PR)? Type inheritance is more complex than what it looks. See https://bugs.python.org/issue43770 for examples of bad surprised that I got when trying to use the default implementation of tp_getattro and tp_setattro.

vstinner · 2021-05-27T14:07:24Z

- PyObject_Free(ko);
+ PyObject_GC_UnTrack(ko);
+ (void)keyobject_clear(ko);
+ PyObject_GC_Del(ko);


IMO here it's ok to hardcode PyObject_GC_Del(), but in general I would suggest to call tp->tp_free(ko); in case tp_alloc/tp_free is overriden in a subclass. To make the code more consistent, I would suggest to always call tp_free in dealloc functions. What do you think?

I agree :) Consistency FTW

This reverts commit 6d7cdaf.

erlend-aasland · 2021-05-27T18:31:56Z

@pablogsal, @shihai1991, @methane: I've reverted the LRU cache optimisations as per @vstinner's request.

vstinner · 2021-05-27T21:04:17Z

+ Py_VISIT(link->result);
+ Py_VISIT(Py_TYPE(link));
+ return 0;
+}


A lru_list_elem_clear() function would also be needed to implement the GC protocol, no?

A lru_list_elem_clear() function would also be needed to implement the GC protocol, no?
I was under the impression traverse was enough. I may be wrong.

Without clear, the ref count cannot each zero, and the GC cannot break cycles: https://devguide.python.org/garbage_collector/#destroying-unreachable-objects
cc @pablogsal

_lru_list_elem is not a standalone type. It is an internal of _lru_cache_wrapper.
lru_cache_tp_traverse() visits lru_list_elem.
lru_cache_tp_clear() clears lru_list_elem.
That's why we can keep _lru_list_elem non-GC type.

IMO, Inada's #26416 is acceptable.

vstinner · 2021-05-28T07:21:44Z

If you want to hide the functools._lru_list_elem type, the lru_cache_tp_traverse() function must not traverse its type. Otherwise, there is a risk that the type is discovered using gc.get_objects() or a similar function.

Right now, the functools._lru_list_elem type is hidden in the main branch and it doesn't implement the GC protocol. In this case, the current code for it in lru_cache_tp_traverse() in ok:

 lru_list_elem *link = self->root.next;
while (link != &self->root) {
lru_list_elem *next = link->next;
Py_VISIT(link->key);
Py_VISIT(link->result);
link = next;
}

And the current code for it in lru_cache_cache_clear() is also ok:

 lru_cache_clear_list(list);

But you still need to add Py_VISIT(Py_TYPE(self)); in lru_cache_tp_traverse().

Since the lru_cache is way more complicated, I suggest to restrict this PR to functools.KeyWrapper and partial type, and write a second PR just for the lru_cache type.

This reverts commit fc39917.

erlend-aasland · 2021-05-28T07:48:06Z

Since the lru_cache is way more complicated, I suggest to restrict this PR to functools.KeyWrapper and partial type, and write a second PR just for the lru_cache type.

Reverting fc39917 should be sufficient; no need to open a new PR.

UPDATE: The PR now adds GC support to the KeyWrapper and partial types, and modifies lru_cache_tp_traverse to visit the LRU cache type and the LRU cache element type.

methane · 2021-05-28T07:52:36Z

If you want to hide the functools._lru_list_elem type, the lru_cache_tp_traverse() function must not traverse its type. Otherwise, there is a risk that the type is discovered using gc.get_objects() or a similar function.

Since _lru_list_elem is not GC-tracked type, gc.get_objects() don't leak _lru_list_elem regardless lru_cache_tp_traverse() visit _lru_list_elem or not.

Right now, the functools._lru_list_elem type is hidden in the main branch and it doesn't implement the GC protocol. In this case, the current code for it in lru_cache_tp_traverse() in ok:

Current PR breaks circular reference correctly. See #26363 (comment)

Since _lru_list_elem is hidden and immutable now, it is OK to remove Py_VISIT(Py_TYPE(link)); if we are sure about _lru_list_elem_type don't create circular reference.
But I am not sure about it. So I feel current PR is the best. It breaks circular reference correctly anyway.

erlend-aasland · 2021-05-28T07:55:05Z

Since _lru_list_elem is hidden and immutable now [...]

It's not immutable; only static types and heap types with the Py_TPFLAGS_IMMUTABLETYPE flag are immutable.

vstinner · 2021-05-28T08:13:09Z

@methane: "Since _lru_list_elem is not GC-tracked type, gc.get_objects() don't leak _lru_list_elem regardless lru_cache_tp_traverse() visit _lru_list_elem or not."

It doesn't matter if the type implements the protocol or not. As soon as there is a Py_VISIT() call on it, it is exposed in gc.get_objects().

I tested the current PR (commit 6313247, Revert "Revert LRU cache element optimisation"):

static int
lru_cache_tp_traverse(lru_cache_object *self, visitproc visit, void *arg)
{
...
lru_list_elem *link = self->root.next;
while (link != &self->root) {
lru_list_elem *next = link->next;
Py_VISIT(link->key);
Py_VISIT(link->result);
Py_VISIT(Py_TYPE(link)); // <==== HERE
link = next;
}
...
}

Example:

$ cat x.py import functools
class Value:
pass
value = Value()
@functools.lru_cache
def func():
return value
func()
$ ./python -i x.py >>> import gc
>>> isinstance(int, type)
True
>>> types=[obj for obj in gc.get_objects() if isinstance(obj, type)]
>>> lru=[type for type in types if 'lru' in type.__qualname__]
>>> lru
[<class 'functools._lru_cache_wrapper'>, <class 'functools._lru_list_elem'>]

Here is the hidden functools._lru_list_elem type.

vstinner · 2021-05-28T08:13:45Z

@erlend-aasland: Can you please leave LRU unchanged in this PR and create a new PR just for LRU? The LRU problem sounds complicated and IMO it deserves its own change.

erlend-aasland · 2021-05-28T08:15:22Z

@erlend-aasland: Can you please leave LRU unchanged in this PR and create a new PR just for LRU? The LRU problem sounds complicated and IMO it deserves its own change.

Of course. Give me 2 minutes. UPDATE: Done.

vstinner

LGTM, thanks.

erlend-aasland · 2021-05-28T08:27:57Z

@vstinner @methane, see GH-26423

erlend-aasland · 2021-05-28T08:29:21Z

Thanks for your insights and reviews, everyone!

miss-islington · 2021-05-28T08:41:18Z

Thanks @erlend-aasland for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

bedevere-bot · 2021-05-28T08:41:29Z

GH-26424 is a backport of this pull request to the 3.10 branch.

…artial types (pythonGH-26363) (cherry picked from commit 8994e9c) Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>

…artial types (GH-26363) (GH-26424) (cherry picked from commit 8994e9c) Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>

erlend-aasland requested a review from rhettinger as a code owner May 25, 2021 20:14

the-knights-who-say-ni added the CLA signed label May 25, 2021

bedevere-bot added the awaiting review label May 25, 2021

erlend-aasland requested a review from pablogsal May 25, 2021 20:14

erlend-aasland added skip news needs backport to 3.10 only security fixes labels May 25, 2021

shihai1991 reviewed May 26, 2021
View reviewed changes

Comment threadModules/_functoolsmodule.c Outdated

shihai1991 requested a review from methane May 26, 2021 10:19

methane reviewed May 26, 2021
View reviewed changes

Comment threadModules/_functoolsmodule.c Outdated

erlend-aasland force-pushed the bpo-42972/functools branch 2 times, most recently from ce657de to f6e6676Compare May 26, 2021 20:48

erlend-aasland marked this pull request as draft May 26, 2021 20:51

Erlend E. Aasland added 2 commits May 27, 2021 01:58

bpo-42972: Fully implement GC protocol for functools types
8ef6e98

Address review
6d7cdaf

erlend-aasland force-pushed the bpo-42972/functools branch from f6e6676 to 6d7cdafCompare May 26, 2021 23:58

erlend-aasland marked this pull request as ready for review May 27, 2021 00:01

shihai1991 approved these changes May 27, 2021
View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels May 27, 2021

Address review: visit node type for each strong ref
a15b49f

vstinner reviewed May 27, 2021
View reviewed changes

Erlend E. Aasland added 6 commits May 27, 2021 19:37

Address review: visit members in correct order
0c8df1f

Address review: revert Py_CLEAR change in lru_list_elem_dealloc
f9ac4c5

Revert LRU cache element optimisation
fc39917
This reverts commit 6d7cdaf.

Address review: revert redundant slot removal
07f2182

Address review: use tp_free
0861788

Revert lru list elem clear fn
7c2f750

vstinner reviewed May 27, 2021
View reviewed changes

Revert "Revert LRU cache element optimisation"
6313247
This reverts commit fc39917.

Restore lru_cache_tp_traverse
1539195

erlend-aasland requested a review from vstinner May 28, 2021 08:20

erlend-aasland changed the title ~~bpo-42972: Fully implement GC protocol for functools types~~bpo-42972: Fully implement GC protocol for functools keywrapper and partial typesMay 28, 2021

vstinner approved these changes May 28, 2021
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels May 28, 2021

vstinner merged commit 8994e9c into python:mainMay 28, 2021

bedevere-bot removed the awaiting merge label May 28, 2021

bedevere-bot removed the needs backport to 3.10 only security fixes label May 28, 2021

vstinner pushed a commit that referenced this pull request May 28, 2021
bpo-42972: Fully implement GC protocol for functools keywrapper and p…
eb8ab04
…artial types (GH-26363) (GH-26424) (cherry picked from commit 8994e9c) Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>

erlend-aasland deleted the bpo-42972/functools branch May 28, 2021 09:59

pablogsal mentioned this pull request Jul 18, 2021
[3.10] Correct the order of regen-abidump #27228
Closed

vstinner mentioned this pull request Jun 9, 2022
[C API] Heap types (PyType_FromSpec) must fully implement the GC protocol #87138
Closed

Uh oh!

Uh oh!

Conversation

erlend-aasland commented May 25, 2021• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shihai1991 left a comment

Choose a reason for hiding this comment

Uh oh!

erlend-aasland commented May 27, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erlend-aasland commented May 27, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner commented May 28, 2021

Uh oh!

erlend-aasland commented May 28, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

methane commented May 28, 2021

Uh oh!

erlend-aasland commented May 28, 2021

Uh oh!

vstinner commented May 28, 2021

Uh oh!

vstinner commented May 28, 2021

Uh oh!

erlend-aasland commented May 28, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

erlend-aasland commented May 28, 2021

Uh oh!

erlend-aasland commented May 28, 2021

Uh oh!

miss-islington commented May 28, 2021

Uh oh!

bedevere-bot commented May 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

erlend-aasland commented May 25, 2021•
edited by bedevere-bot
Loading

erlend-aasland commented May 28, 2021•
edited
Loading

erlend-aasland commented May 28, 2021•
edited
Loading