Skip to content

Fix in-process Execution API loop stopped while its transport is in use - #68855

Closed
potiuk wants to merge 2 commits into
apache:mainfrom
potiuk:fix-in-process-execution-api-loop-lifecycle
Closed

Fix in-process Execution API loop stopped while its transport is in use#68855
potiuk wants to merge 2 commits into
apache:mainfrom
potiuk:fix-in-process-execution-api-loop-lifecycle

Conversation

@potiuk

@potiukpotiuk commented Jun 22, 2026

Copy link
Copy Markdown
Member

The in-process Execution API owns a background asyncio event loop + daemon thread. Their cleanup was registered with weakref.finalize keyed on the InProcessExecutionAPI instance. But callers build a sync Client from InProcessExecutionAPI().transport and discard the factory object, so the instance was garbage-collected while the transport was still in use. The finalizer then stopped the loop, and every subsequent request hung on the dead loop -- surfacing as Timeout failures across the Dag-processor and triggerer in-process API tests on main.

Tie the loop + thread + lifespan cleanup to the lifetime of the WSGI transport (what callers actually retain) instead of the factory instance. This keeps the original thread-leak fix intact -- the loop is still stopped and the thread joined once nothing can use the transport -- while no longer tearing it down prematurely.

Follow up after #68840


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:task-sdk labels Jun 22, 2026
The in-process Execution API owns a background asyncio event loop + daemon
thread. Their cleanup was registered with `weakref.finalize` keyed on the
`InProcessExecutionAPI` instance. But callers build a sync `Client` from
`InProcessExecutionAPI().transport` and discard the factory object, so the
instance was garbage-collected while the transport was still in use. The
finalizer then stopped the loop, and every subsequent request hung on the
dead loop -- surfacing as `Timeout` failures across the Dag-processor and
triggerer in-process API tests on main.
Tie the loop + thread + lifespan cleanup to the lifetime of the WSGI
transport (what callers actually retain) instead of the factory instance.
This keeps the original thread-leak fix intact -- the loop is still stopped
and the thread joined once nothing can use the transport -- while no longer
tearing it down prematurely.
@potiuk
potiukforce-pushed the fix-in-process-execution-api-loop-lifecycle branch from fb08675 to 48b2427CompareJune 22, 2026 19:13
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/app.py Outdated
@seanghaeli

Copy link
Copy Markdown
Contributor

Hi @potiuk, thanks for the fix, sorry for jumping in, I assume you're asleep so I made another PR for the full fix so it can be merged now, needed it urgently for downstream PRs.

@potiuk

Copy link
Copy Markdown
MemberAuthor

Fixed by #68865 copied from this one

@potiukpotiuk closed this Jun 23, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:APIAirflow's REST/HTTP APIarea:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@seanghaeli