Crash in `DistributionFinder.context` because it is using `vars(self).update(kwargs)` under memory preasure

[sobolevn]

Crash report

Original report: https://gist.github.com/devdanzin/bf9fec4554c58c2a279b05b7ff8e6d9b (it produces the correct repro, but everything else is not correct, since it is AI generated and can be fuzzy).

Here's the real picture.

This class raises an error:

cpython/Lib/importlib/metadata/__init__.py

Lines 742 to 750 in 1785f4b

	class DistributionFinder(MetaPathFinder):
	"""
	A MetaPathFinder capable of discovering installed distributions.
	Custom providers should implement this interface in order to
	supply metadata.
	"""
	class Context:

Repro:

import faulthandler, importlib.metadata
faulthandler.enable()
from _testcapi import set_nomemory
for start in range(40):
 set_nomemory(start)
 try:
 importlib.metadata.metadata("f")
 except BaseException:
 pass
print("done, no crash")

Error:

Fatal Python error: _Py_Dealloc: Deallocator of type 'Context' cleared the current exception
Python runtime state: initialized

Stack (most recent call first):
 File "/Users/sobolev/Desktop/cpython/Lib/importlib/metadata/__init__.py", line 465 in from_name
 File "/Users/sobolev/Desktop/cpython/Lib/importlib/metadata/__init__.py", line 1091 in metadata
 File "/Users/sobolev/Desktop/cpython/ex.py", line 7 in <module>

Extension modules: _testcapi (total: 1)
[1] 14108 abort ./python.exe ex.py

I tested this hack, no crash:

diff --git Lib/importlib/metadata/__init__.py Lib/importlib/metadata/__init__.py
index 32f4b7d2d6e..43ec93db749 100644
--- Lib/importlib/metadata/__init__.py
+++ Lib/importlib/metadata/__init__.py
@@ -777,7 +777,8 @@ class Context:
 """
 
 def __init__(self, **kwargs):
- vars(self).update(kwargs)
+ for k, v in kwargs.items():
+ setattr(self, k, v)
 
 @property
 def path(self) -> list[str]:

But, this is clearly not a proper fix. Because for some reason the exception is cleared somewhere.

After:

» ./python.exe ex.py
object address : 0x20002128190
object refcount : 3
object type : 0x101267e70
object type name: MemoryError
object repr : 
lost sys.stderr
object address : 0x20002128110
object refcount : 3
object type : 0x101267e70
object type name: MemoryError
object repr : 
lost sys.stderr

Linked PRs

• gh-152083: Fix crash clearing a managed dict under low memory #152981

[sobolevn]

Config: make clean && ./configure --with-pydebug --disable-gil && make -j$(sysctl -n hw.ncpu)
Simplier repro:

import faulthandler, importlib.metadata
faulthandler.enable()
from _testcapi import set_nomemory
import abc


class DistributionFinder:
 class Context:
 name = None
 def __init__(self, **kwargs):
 vars(self).update(kwargs)

 @abc.abstractmethod
 def find_distributions(self, context=Context()):
 pass

for start in range(40):
 set_nomemory(start)
 try:
 DistributionFinder.Context(name='f')
 except BaseException:
 pass
print("done, no crash")

Traceback:

done, no crash
Assertion failed: (_Py_IsOwnedByCurrentThread((PyObject *)mp) || IS_DICT_SHARED(mp)), function set_keys, file dictobject.c, line 206.
Fatal Python error: Aborted

<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):

Current thread's C stack trace (most recent call first):
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at _Py_DumpStack+0x44 [0x1007b2234]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at faulthandler_dump_c_stack+0x68 [0x1007e19e0]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at faulthandler_fatal_error+0x1ac [0x1007e177c]
 Binary file "/usr/lib/system/libsystem_platform.dylib", at _sigtramp+0x38 [0x1a00d16a4]
 Binary file "/usr/lib/system/libsystem_pthread.dylib", at pthread_kill+0x128 [0x1a009788c]
 Binary file "/usr/lib/system/libsystem_c.dylib", at abort+0x7c [0x19ffa0a3c]
 Binary file "/usr/lib/system/libsystem_c.dylib", at err+0x0 [0x19ff9fc70]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at set_keys+0x70 [0x1004d5d3c]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at PyObject_ClearManagedDict+0x144 [0x1004d58ac]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at subtype_clear+0xd8 [0x1005630ec]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at delete_garbage+0x1c8 [0x10070ab20]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at gc_collect_internal+0x2ec [0x100709474]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at gc_collect_main+0x248 [0x100706708]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at _PyGC_CollectNoFail+0x20 [0x100706984]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at finalize_modules+0xc4 [0x1007658b0]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at _Py_Finalize+0x1c4 [0x100764bac]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at Py_FinalizeEx+0x14 [0x1007649e0]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at Py_RunMain+0x28 [0x1007cb5b8]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at pymain_main+0x64 [0x1007cba84]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at Py_BytesMain+0x38 [0x1007cbad4]
 Binary file "/Users/sobolev/Desktop/cpython/python.exe", at main+0x24 [0x1003588fc]
 Binary file "/usr/lib/dyld", at start+0x17bc [0x19fcf6b98]
[1] 14739 abort ./python.exe ex.py

The crash seems different, but it might have the similar cause.

[deadlovelll]

Hi @sobolevn ! I think the proper fix somewhere deep in the c-side, i can work on this issue if you dont mind =)

[sobolevn]

@deadlovelll please go ahead :)

[deadlovelll]

@deadlovelll please go ahead :)

Hello! Just pushed the fix, would appreciate a review =)