Behavioral differences between the with-statement and `ExitStack.enter_context`

[gottadiveintopython]

Bug report

Bug description:

I noticed a behavioral difference between the (async) with statement and (Async)ExitStack. The with statement correctly recognizes context manager methods (__enter__, __exit__, __aenter__, and __aexit__) even when they are defined via __slots__, whereas (Async)ExitStack does not.

class SlotsCM:
 __slots__ = ("__enter__", "__exit__", )

 def __init__(self):
 self.__enter__ = self.__exit__ = lambda *__: None


# This works
with SlotsCM():
 pass


# This doesn't.
from contextllib import ExitStack
with ExitStack() as stack:
 stack.enter_context(SlotsCM())

Traceback (most recent call last):
 File "xxxx.py", line 16, in <module>
 stack.enter_context(SlotsCM())
 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
 File "......../contextlib.py", line 530, in enter_context
 result = _enter(cm)
TypeError: 'member_descriptor' object is not callable

Furthermore, both (async) with-statement and (Async)ExitStack don't suppot those special dunder methods stored in __dict__. You can verify from this repository:
https://github.com/gottadiveintopython/invest-context-manager

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-144386: Add support for descriptors in ExitStack and AsyncExitStack #144420
• gh-144386: Update equivalent code for "with", "async with" and "async for" #144472
• [3.14] gh-144386: Update equivalent code for "with", "async with" and "async for" (GH-144472) #144945
• [3.13] gh-144386: Update equivalent code for "with", "async with" and "async for" (GH-144472) #144946

[tanloong]

contextlib.ExitStack.enter_context(cm) calls type(cm).__enter__(cm). The problem looks like that type(cm) does not recognize those __enter__ and __exit__ methods defined by __slots__. The document says type() generally returns the same object as returned by object.__class__, which throws the same TypeError:

class SlotsCM:
 __slots__ = (
 "__enter__",
 "__exit__",
 )

 def __init__(self):
 self.__enter__ = self.__exit__ = lambda *__: None

s = SlotsCM()
s.__enter__(s) # This works.
type(s).__enter__(s) # TypeError: 'member_descriptor' object is not callable
s.__class__.__enter__(s) # TypeError: 'member_descriptor' object is not callable

[gottadiveintopython]

I don't fully understand the error message, but yes, the cause of the issue seems to be that (Async)ExitStack.enter_(async)_context() looks up dunder methods on the class rather than on
the instance.

[tanloong]

I am not familiar with the member_descriptor neither, it looks like a concept in C. A printing shows it refers to the __enter__ object reserved by __slots__.
Also I found that calling __get__(s) on the member descriptor returns the same function object as s.__enter__.

s = SlotsCM()
print(type(s.__enter__)) # <class 'function'>
print(type(s.__class__.__enter__)) # <class 'member_descriptor'>
print(id(s.__class__.__enter__.__get__(s, SlotsCM)) == id(s.__enter__)) # True

I tried to modify the contextlib.ExitStack.enter_context() code to use s.__class__.__enter__.__get__(s). It works, the TypeError is gone:

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 5b646fabca0..2bb72bae536 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -524,6 +524,10 @@ def enter_context(self, cm):
 try:
 _enter = cls.__enter__
 _exit = cls.__exit__
+ if not callable(_enter):
+ _enter = _enter.__get__(cm, cls)
+ if not callable(_exit):
+ _exit = _exit.__get__(cm, cls)
 except AttributeError:
 raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
 f"not support the context manager protocol") from None

[tanloong]

I just found that there is also a difference between the actual with-statement behavior and its documented semantically equivalent code. The documentted equivalent code throws the same TypeError as contextlib.ExitStack.enter_context().

The following code:

with EXPRESSION as TARGET:
 SUITE

is semantically equivalent to:

manager = (EXPRESSION)
enter = type(manager).__enter__
exit = type(manager).__exit__
value = enter(manager)
hit_except = False

try:
 TARGET = value
 SUITE
except:
 hit_except = True
 if not exit(manager, *sys.exc_info()):
 raise
finally:
 if not hit_except:
 exit(manager, None, None, None)

PEP 343 explains with statement with generally the same code which also throws the TypeError in this usecase.

[picnixz]

Instead of using callable() I would suggest that maybe we first try to check if the enter/exit methods are descriptors or not by checking the presence of __get__. I think we have similar approaches with partial/partialmethod but I am not entirely sure

[tanloong]

@picnixz Thanks for the suggestion! This is the modified change I made following the partialmethod.

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 5b646fabca0..fd798796e7d 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -524,6 +524,10 @@ def enter_context(self, cm):
 try:
 _enter = cls.__enter__
 _exit = cls.__exit__
+ if (_enter_get := getattr(_enter, "__get__", None)) is not None:
+ _enter = _enter_get(cm, cls)
+ if (_exit_get := getattr(_exit, "__get__", None)) is not None:
+ _exit = _exit_get(cm, cls)
 except AttributeError:
 raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
 f"not support the context manager protocol") from None

[picnixz]

This looks correct but I am not very fond of having a huge try-catch here. Likewise it does not seem right to catch the attribute error that could be raised by __get__ so you should move the descriptor handling after the try-catch.

[tanloong]

OK here is the new change:

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 5b646fabca0..5b5dc48affa 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -527,6 +527,10 @@ def enter_context(self, cm):
 except AttributeError:
 raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
 f"not support the context manager protocol") from None
+ if (_enter_get := getattr(_enter, "__get__", None)) is not None:
+ _enter = _enter_get(cm, cls)
+ if (_exit_get := getattr(_exit, "__get__", None)) is not None:
+ _exit = _exit_get(cm, cls)
 result = _enter(cm)
 self._push_cm_exit(cm, _exit)
 return result

[picnixz]

Yes, something like that. Though I think you can write something more readable as:

if hasattr(_enter, "__get__"):
 _enter = _enter.__get__(cm, cls)

[tanloong]

Thank you for the suggestion! I updated the change. But I found that the change breaks some contextlib tests:

Running Debug|x64 interpreter...
..................................................EE......E..F..............................
======================================================================
ERROR: test_dont_reraise_RuntimeError (Lib.test.test_contextlib.TestExitStack.test_dont_reraise_RuntimeError)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 1151, in test_dont_reraise_RuntimeError
 es_ctx.enter_context(second())
 ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 538, in enter_context
 result = _enter(cm)
TypeError: _GeneratorContextManager.__enter__() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_enter_context (Lib.test.test_contextlib.TestExitStack.test_enter_context)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 786, in test_enter_context
 stack.enter_context(cm)
 ~~~~~~~~~~~~~~~~~~~^^^^
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 538, in enter_context
 result = _enter(cm)
TypeError: TestBaseExitStack.test_enter_context.<locals>.TestCM.__enter__() takes 1 positional argument but 2 were given

======================================================================
ERROR: test_exit_exception_explicit_none_context (Lib.test.test_contextlib.TestExitStack.test_exit_exception_explicit_none_context) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 1002, in test_exit_exception_explicit_none_context
 with cm():
 ~~^^
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 141, in __enter__
 return next(self.gen)
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 996, in my_cm_with_exit_stack
 stack.enter_context(my_cm())
 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 538, in enter_context
 result = _enter(cm)
TypeError: _GeneratorContextManager.__enter__() takes 1 positional argument but 2 were given

======================================================================
FAIL: test_exit_exception_with_correct_context (Lib.test.test_contextlib.TestExitStack.test_exit_exception_with_correct_context)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 1055, in test_exit_exception_with_correct_context
 stack.enter_context(gets_the_context_right(exc4))
 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\contextlib.py", line 538, in enter_context
 result = _enter(cm)
TypeError: _GeneratorContextManager.__enter__() takes 1 positional argument but 2 were given

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "C:\Users\Administrator\Desktop\cpython\main\Lib\test\test_contextlib.py", line 1060, in test_exit_exception_with_correct_context
 self.assertIs(exc, exc4)
 ~~~~~~~~~~~~~^^^^^^^^^^^
AssertionError: TypeError('_GeneratorContextManager.__enter__() takes 1 positional argument but 2 were given') is not Exception(4)

----------------------------------------------------------------------
Ran 92 tests in 0.493s

FAILED (failures=1, errors=3)

I found that the __enter__ object defined by a regular class method defination also has a __get__ attribute:

class CM:
 def __init__(self):
 pass
 def __enter__(self):
 pass
 def __exit__(self, *exc_info):
 pass

class SlotsCM:
 __slots__ = ("__enter__", "__exit__")
 def __init__(self):
 self.__enter__ = self.__exit__ = lambda *__: None

cm = CM()
cls = type(cm)
_enter = cls.__enter__
print(getattr(_enter, "__get__", None)) # <method-wrapper '__get__' of function object at 0x00000154BDFB4650>

s = SlotsCM()
cls = type(s)
_enter = cls.__enter__
print(getattr(_enter, "__get__", None)) # <method-wrapper '__get__' of member_descriptor object at 0x000002C78844D5B0>

I tried to add the callable() check and this passes existing tests:

diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 5b5dc48affa..fbc4de9277b 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -527,10 +527,14 @@ def enter_context(self, cm):
 except AttributeError:
 raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
 f"not support the context manager protocol") from None
+ if not callable(_enter) and hasattr(_enter, "__get__"):
+ _enter = _enter.__get__(cm, cls)
+ if not callable(_exit) and hasattr(_exit, "__get__"):
+ _exit = _exit.__get__(cm, cls)
 result = _enter(cm)
 self._push_cm_exit(cm, _exit)
 return result

[picnixz]

Urgh, I hate working with descriptors. I actually wonder what we did on the C side for __enter__ as a slot to work. This may require more subtle checks

[tanloong]

I am not good at C but dis.dis() shows there is a LOAD_SPECIAL to handle the special methods.

class SlotsCM:
 __slots__ = ("__enter__", "__exit__")
 def __init__(self):
 self.__enter__ = self.__exit__ = lambda *__: None

def func():
 with SlotsCM():
 pass

import dis
from pprint import pprint
pprint(dis.dis(func))

# 8 RESUME 0
#
# 9 LOAD_GLOBAL 1 (SlotsCM + NULL)
# CALL 0
# COPY 1
# LOAD_SPECIAL 1 (__exit__)
# SWAP 2
# SWAP 3
# LOAD_SPECIAL 0 (__enter__)
# CALL 0
# L1: POP_TOP
#
# 10 L2: NOP
#
# 9 LOAD_CONST 0 (None)
# LOAD_CONST 0 (None)
# LOAD_CONST 0 (None)
# CALL 3
# POP_TOP
# LOAD_CONST 0 (None)
# RETURN_VALUE
# L3: PUSH_EXC_INFO
# WITH_EXCEPT_START
# TO_BOOL
# POP_JUMP_IF_TRUE 2 (to L6)
# L4: NOT_TAKEN
# L5: RERAISE 2
# L6: POP_TOP
# L7: POP_EXCEPT
# POP_TOP
# POP_TOP
# POP_TOP
# LOAD_CONST 0 (None)
# RETURN_VALUE
#
# -- L8: COPY 3
# POP_EXCEPT
# RERAISE 1
# ExceptionTable:
# L1 to L2 -> L3 [2] lasti
# L3 to L4 -> L8 [4] lasti
# L5 to L7 -> L8 [4] lasti
# None