Finalization of non-exhausted asynchronous generators is deferred

[serhiy-storchaka]

BPO	44518
Nosy	@gvanrossum, @asvetlov, @markshannon, @serhiy-storchaka, @1st1, @gvanrossum

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-06-27.09:11:32.621>
labels = ['interpreter-core', '3.11', '3.9', '3.10', 'performance']
title = 'Finalization of non-exhausted asynchronous generators is deferred'
updated_at = <Date 2021-06-28.18:33:50.454>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-06-28.18:33:50.454>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-06-27.09:11:32.621>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 44518
keywords = []
message_count = 6.0
messages = ['396569', '396610', '396611', '396644', '396650', '396659']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'asvetlov', 'Mark.Shannon', 'serhiy.storchaka', 'yselivanov', 'Guido.van.Rossum']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue44518'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[serhiy-storchaka]

In the following example:

def gen():
 try:
 yield 1
 finally:
 print('finalize inner')

def func():
 try:
 for x in gen():
 break
 finally:
 print('finalize outer')

func()
print('END')

the output in CPython is:

finalize inner
finalize outer
END

But in similar example for asynchronous generator:

async def gen():
 try:
 yield 1
 finally:
 print('finalize inner')

async def func():
 try:
 async for x in gen():
 break
 finally:
 print('finalize outer')

import asyncio
asyncio.run(func())
print('END')

the output in CPython is:

finalize outer
finalize inner
END

There is a strong link somewhere which prevents finalization of the asynchronous generator object until leaving the outer function.

Tested on CPython 3.7-3.11. In PyPy "finalize inner" is not printed at all. Using closing() and aclosing() is the right way to get deterministic finalization, but in any case it would be better to get rid of strong link which prevents finalization of the asynchronous generator object.

[gvanrossum]

Can you repro this without asyncio?

[serhiy-storchaka]

Sorry, I do not understand what do you mean. The problem is that non-exhausted asynchronous generators are not finalized until asyncio.run() is finished. This differs from synchronous generators which are finalized as early as possible (in CPython).

[gvanrossum]

I am wondering whether the issue is in asyncio.run, or in the basic async generator implementation. If the latter, you should be able to demonstrate this with a manual "driver" (trampoline?) instead of asyncio.run.

[serhiy-storchaka]

Inlined asyncio.run(func()) is roughly equivalent to the following code:

loop = asyncio.new_event_loop()
loop.run_until_complete(func())
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()

If comment out loop.run_until_complete(loop.shutdown_asyncgens()) I get the following output:

finalize outer
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<<async_generator_athrow without __name__>()>>
END

No "finalize inner" but a warning about pending task instead.

On other hand, the following code

for _ in func().__await__(): pass

produces the output in expected order:

finalize inner
finalize outer
END