gh-75459: Doc: C API: Improve object life cycle documentation

[rhansen]

• Add "cyclic isolate" to the glossary.
• Add a new "Object Life Cycle" page.
  • Illustrate the order of life cycle functions.
  • Document PyObject_CallFinalizer and PyObject_CallFinalizerFromDealloc.
• PyObject_Init does not call tp_init.
• PyObject_New:
  • also initializes the memory
  • does not call tp_alloc, tp_new, or tp_init
  • should not be used for GC-enabled objects
  • memory must be freed by PyObject_Free
• PyObject_GC_New memory must be freed by PyObject_GC_Del.
• Warn that garbage collector functions can be called from any thread.
• tp_finalize and tp_clear:
  • Only called when there's a cyclic isolate.
  • Only one object in the cyclic isolate is finalized/cleared at a time.
  • Clearly warn that they might not be called.
  • They can optionally be manually called from tp_dealloc (via PyObject_CallFinalizerFromDealloc in the case of tp_finalize).
• tp_finalize:
  • Reference object.__del__.
  • The finalizer can resurrect the object.
  • Suggest PyErr_GetRaisedException and PyErr_SetRaisedException instead of the deprecated PyErr_Fetch and PyErr_Restore functions.
  • Add links to PyErr_GetRaisedException and PyErr_SetRaisedException.
  • Suggest using PyErr_WriteUnraisable if an exception is raised during finalization.
  • Rename the example function from local_finalize to foo_finalize for consistency with the tp_dealloc documentation and as a hint that the name isn't special.
  • Minor wording and sylistic tweaks.
  • Warn that tp_finalize can be called during shutdown.

• Issue: PyObject_CallFinalizerFromDealloc is undocumented #75459

---

📚 Documentation preview 📚: https://cpython-previews--125962.org.readthedocs.build/

[rhansen]

I'm not familiar enough with CPython's internals to be super confident about these changes. I would appreciate it if a GC expert would carefully review this.

Thanks!

[hugovk]

Hmm, I'm not sure if we can require graphviz for the docs.

We'd have to consider installing it on the main docs server in addition to Read the Docs, and also make sure the docs can still build without it, for downstream redistributors who might only want to build with "vanilla" Sphinx and no extra extensions. Plus other developers would need an easy way to build the docs on their machines.

cc @AA-Turner

[rhansen]

Hmm, I'm not sure if we can require graphviz for the docs.

Maybe I should just commit the generated .svg (and the input dot file so it can be revised easily). Would that be acceptable?

[rhansen]

I committed the generated .svg so that the substance of this PR can be reviewed while we figure out if it is acceptable to add graphviz as a dependency. (Note that sphinx.ext.graphviz is a built-in extension, so enabling it doesn't add any new sphinx dependencies.)

[AA-Turner]

I think that requiring graphviz should be fine -- Debian, Fedora, Gentoo, and OpenSUSE all package it. As Richard notes, it's a built-in extension, so should be fine from the "Vanilla" perspective.

I would want to include a NEWS entry to say that graphviz is now required to build the docs, though.

A

[ZeroIntensity]

My main concern with documenting nitty-gritty details of the lifecycle is that we're technically documenting implementation details, which are subject to change (and we've been bad at updating these kind of things from version-to-version in past). I suggest the SVG go into the InternalDocs folder instead.

It's also worth noting here that tp_finalize isn't 100% related to garbage collection, it's supposed to be used over tp_dealloc if complicated things are being done upon finalization, even for non-GC types. And while we're here, I think it would be a good idea to document the cases that tp_clear should exist for a tracked type.

[rhansen]

My main concern with documenting nitty-gritty details of the lifecycle is that we're technically documenting implementation details, which are subject to change (and we've been bad at updating these kind of things from version-to-version in past). I suggest the SVG go into the InternalDocs folder instead.

I don't want to document any implementation details here, so I'm happy to remove what isn't necessary. It's hard to tell what is and isn't necessary because the end of an object's life is especially fraught with peril. I think that it is better to err on the side of over-documenting this topic than under-documenting.

I wrote this PR because there were several things that I needed to know that the existing documentation didn't make clear:

• The order the tp_* functions might be called (to know what invariants are possible).
• Which threads might execute the functions (for locking correctness).
• Details about when a function is called (or not) that are necessary for locking correctness. (e.g., if multiple objects in the same cyclic isolate are never finalized concurrently then a lock-free design might be possible)
• Approximately how often a tp_* function might be called: maybe never, exactly once, at most once, at least once, very frequently, etc.
• Which other objects might be in an inconsistent state.

It's also worth noting here that tp_finalize isn't 100% related to garbage collection, it's supposed to be used over tp_dealloc if complicated things are being done upon finalization, even for non-GC types.

If I understand correctly, tp_finalize is never called for non-GC types unless the class designer calls it from tp_dealloc. In that case tp_finalize is just like any other helper function that might be called from tp_dealloc. (Maybe this is only true for static types and not heap types? I don't fully understand the difference.)

And while we're here, I think it would be a good idea to document the cases that tp_clear should exist for a tracked type.

I thought that was already sufficiently explained, even before this PR. Can you explain what you think is lacking?

[ZeroIntensity]

First, thanks for doing this!

I think that it is better to err on the side of over-documenting this topic than under-documenting.

I don't, that limits our ability to modify the lifecycle in the future (especially because there's no good way to deprecate things here). I'll point this out when doing a more in-depth review though, I don't see anything particularly bad right now.

If I understand correctly, tp_finalize is never called for non-GC types unless the class designer calls it from tp_dealloc.

You're right, it's not, but I don't think we should limit ourselves to that in the future. It might be possible someday to automatically do this for untracked types as well. We should just document that all types, even GC, require PyObject_CallFinalizerFromDealloc in the destructor if they want tp_finalize to get eventually called--we can note that it could happen automatically, though.

I thought that was already sufficiently explained, even before this PR. Can you explain what you think is lacking?

Basically, it's not documented which types need to have a tp_clear, because not all GC types have it. I'm not even sure which cases require it. I think it's only needed if the type can have a direct reference cycle to itself? (As in, running its finalizer will try to Py_DECREF itself.)

Also, I don't think it should be documented that tp_clear is related to tp_dealloc by making an "optional call", that's sort of incidental. They tend to do the same thing, and the destructor can utilize the clear function for convenience, but they're for different purposes.

---

A few other notes:

• It's fine to document what the specific allocators (e.g. PyObject_GC_New and PyObject_GC_Del) do, but we should point users to using tp_alloc and tp_free instead.
• That said, I don't see the need to mention that functions like PyObject_New don't call tp_init. Those are strictly allocators, and currently documented as such.
• This isn't exhaustive--static objects and some immortal objects don't follow this lifecycle (single-phase modules don't either, I think). Other objects might not follow this in the future either.
• If I read "called from any thread" in the docs, I would be worried about holding the GIL. Maybe mention that while it can be called from any thread, it will still hold the GIL.

[rhansen]

I did a major rewrite to hopefully address all of the feedback (thanks for reviewing!). Please take another look.

[ZeroIntensity]

I'm going to be taking this over. I've addressed my own comments and most of Petr's. Other reviews would be appreciated!

[encukou]

Thank you! I definitely want to take another look -- after Beta 1, next week or so.

[encukou]

Overall, this is an improvement. Thank you for the work!
Let's do one more pass, merge, and leave the rest for a future PR?

[ZeroIntensity]

Let's do one more pass, merge, and leave the rest for a future PR?

Sounds good to me.

[ZeroIntensity]

Thanks, Petr!

Do we want to backport this? On the one hand, this is a great improvement to the C API documentation that can definitely help users in 3.13. But on the other day, this is a huge change to backport. Maybe that doesn't matter? I'm not sure we should treat documentation changes the same way we treat code changes.

[hugovk]

Very few people read the pre-release docs. Over the past week, we had these many visitors for these homepages:

• /3/ - 94.1k
• /3.13/ - 33.6k (same as /3/)
• /3.14/ - 2.3k
• /3.15/ - 4.4k
• /dev/ - 358 (same as /3.15)

If we want people to benefit from this work before October 2026(!), we should backport.

[ZeroIntensity]

To 3.14, or all the way to 3.13? I can see arguments for both. Note that if we backport to 3.13, we also need to backport the attached thread state change that this PR uses. (That might be a good thing, I've noticed myself constantly pointing free-threading users to the 3.14 docs for the better documentation.)

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.x (tier-3) has failed when building commit 3246ea5.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1380/builds/3754) and take a look at the build logs.
4. Check if the failure is related to this commit (3246ea5) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1380/builds/3754

Failed tests:

• test.test_concurrent_futures.test_interpreter_pool

Failed subtests:

• test_map_buffersize_on_infinite_iterable - test.test_concurrent_futures.test_interpreter_pool.InterpreterPoolExecutorTest.test_map_buffersize_on_infinite_iterable

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
File "/Users/buildbot/Library/Developer/XCTestDevices/4832ED11-0C79-4767-B1A6-77FDBE8E03AE/data/Containers/Bundle/Application/95646D17-A682-4984-91B0-88B196595790/iOSTestbed.app/python/lib/python3.15/concurrent/futures/thread.py", line 99, in _worker
ctx.initialize()
~~~~~~~~~~~~~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/4832ED11-0C79-4767-B1A6-77FDBE8E03AE/data/Containers/Bundle/Application/95646D17-A682-4984-91B0-88B196595790/iOSTestbed.app/python/lib/python3.15/concurrent/futures/interpreter.py", line 132, in initializeself.interpid = _interpreters.create(reqrefs=True)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^interpreters.InterpreterError: interpreter creation failed
ERROR
Traceback (most recent call last):
File "/Users/buildbot/Library/Developer/XCTestDevices/4832ED11-0C79-4767-B1A6-77FDBE8E03AE/data/Containers/Bundle/Application/95646D17-A682-4984-91B0-88B196595790/iOSTestbed.app/python/lib/python3.15/test/test_concurrent_futures/executor.py", line 127, in test_map_buffersize_on_infinite_iterableself.assertEqual(next(res, None), "0")
~~~~^^^^^^^^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/4832ED11-0C79-4767-B1A6-77FDBE8E03AE/data/Containers/Bundle/Application/95646D17-A682-4984-91B0-88B196595790/iOSTestbed.app/python/lib/python3.15/concurrent/futures/_base.py", line 666, in result_iteratoryield _result_or_cancel(fs.pop())
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/4832ED11-0C79-4767-B1A6-77FDBE8E03AE/data/Containers/Bundle/Application/95646D17-A682-4984-91B0-88B196595790/iOSTestbed.app/python/lib/python3.15/concurrent/futures/_base.py", line 311, in _result_or_cancelreturn fut.result(timeout)
~~~~~~~~~~^^^^^^^^^
File "/Users/buildbot/Library/Developer/XCTestDevices/4832ED11-0C79-4767-B1A6-77FDBE8E03AE/data/Containers/Bundle/Application/95646D17-A682-4984-91B0-88B196595790/iOSTestbed.app/python/lib/python3.15/concurrent/futures/_base.py", line 450, in resultreturnself.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/Users/buildbot/Library/Developer/XCTestDevices/4832ED11-0C79-4767-B1A6-77FDBE8E03AE/data/Containers/Bundle/Application/95646D17-A682-4984-91B0-88B196595790/iOSTestbed.app/python/lib/python3.15/concurrent/futures/_base.py", line 395, in __get_resultraiseself._exception
concurrent.futures.interpreter.BrokenInterpreterPool: A thread initializer failed, the thread pool is not usable anymore

[encukou]

I'd go for 3.14; that'll become /3/ in this October.

[miss-islington-app]

Thanks @rhansen for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

[bedevere-app]

GH-134344 is a backport of this pull request to the 3.14 branch.