Fix `dispose_orm()` not disposing async engine on shutdown by kaxil · Pull Request #65274 · apache/airflow

kaxil · 2026-04-15T01:20:52Z

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

atexit.register(dispose_orm) -- process exit
Gunicorn worker_exit hook -- worker restarts
reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

kaxil · 2026-04-15T01:20:52Z

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

atexit.register(dispose_orm) -- process exit
Gunicorn worker_exit hook -- worker restarts
reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

kaxil · 2026-04-15T01:20:52Z

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

atexit.register(dispose_orm) -- process exit
Gunicorn worker_exit hook -- worker restarts
reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

kaxil · 2026-04-15T01:20:52Z

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

atexit.register(dispose_orm) -- process exit
Gunicorn worker_exit hook -- worker restarts
reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

kaxil · 2026-04-15T01:20:52Z

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

atexit.register(dispose_orm) -- process exit
Gunicorn worker_exit hook -- worker restarts
reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

kaxil · 2026-04-15T01:20:52Z

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

atexit.register(dispose_orm) -- process exit
Gunicorn worker_exit hook -- worker restarts
reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

kaxil · 2026-04-15T01:20:52Z

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

atexit.register(dispose_orm) -- process exit
Gunicorn worker_exit hook -- worker restarts
reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

kaxil · 2026-04-15T01:20:52Z

dispose_orm() now disposes async_engine and clears AsyncSession, preventing connection leaks on process exit, gunicorn worker restarts, and atexit

Background

When _configure_async_session() was extracted from configure_orm() the corresponding cleanup in dispose_orm() was not updated. The async engine's connection pool was never closed on:

atexit.register(dispose_orm) -- process exit
Gunicorn worker_exit hook -- worker restarts
reconfigure_orm() -- ORM reconfiguration

With the async engine now actively used by the Execution API (via get_team_name_dep in the dependency chain for connection/variable lookups), each worker restart leaked pool connections against PostgreSQL's max_connections.

Approach

Calls async_engine.sync_engine.dispose() (synchronous path) rather than await async_engine.dispose() since dispose_orm() runs in sync context. This matches the existing pattern in clean_in_fork().

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf)

…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

When _configure_async_session() was extracted from configure_orm() in PR apache#51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references.

kaxil added this to the Airflow 3.2.1 milestone Apr 15, 2026

kaxil requested review from ashb, dstandish and uranusjr April 15, 2026 01:20

kaxil force-pushed the fix/dispose-orm-async-engine branch from 8cd2b62 to 70bc934Compare April 15, 2026 01:23

kaxil force-pushed the fix/dispose-orm-async-engine branch from 70bc934 to f44d845Compare April 15, 2026 01:25

vatsrahul1001 approved these changes Apr 15, 2026
View reviewed changes

vatsrahul1001 merged commit 69a88bf into apache:mainApr 15, 2026
79 checks passed

vatsrahul1001 deleted the fix/dispose-orm-async-engine branch April 15, 2026 06:00

vatsrahul1001 mentioned this pull request Apr 15, 2026
[v3-2-test] Fix dispose_orm() not disposing async engine on shutdown (#65274) #65284
Merged