Add gunicorn support for API server with rolling worker restarts by kaxil · Pull Request #60940 · apache/airflow

kaxil · 2026-01-22T14:49:00Z

This PR adds an optional gunicorn server type for the API server, providing:

Memory sharing: Gunicorn uses preload + fork, so workers share memory via copy-on-write (unlike uvicorn's multiprocess mode where each worker loads everything independently)
Rolling worker restarts: Custom Arbiter performs zero-downtime worker recycling to prevent memory accumulation
Proper signal handling: SIGTTOU kills oldest worker (FIFO), enabling true rolling restarts

Memory Impact

With --preload, gunicorn loads the application once in the arbiter process, then forks workers. Workers share read-only memory pages via copy-on-write:

Configuration	Estimated Memory Usage
Uvicorn 4 workers	~600 MB (4 × 150 MB, each loads independently)
Gunicorn 4 workers	~300-350 MB (shared base + ~50 MB per worker for unique pages)

Savings: ~40-50% memory reduction with 4 workers. Benefits scale with worker count.

Usage

# Enable gunicorn modeexport AIRFLOW__API__SERVER_TYPE=gunicorn
export AIRFLOW__API__WORKER_REFRESH_INTERVAL=43200 # 12 hours
airflow api-server

Configuration

New [api] configuration options:

server_type: uvicorn (default) or gunicorn
worker_refresh_interval: Seconds between worker refresh cycles (0 = disabled)
worker_refresh_batch_size: Workers to refresh per cycle (default: 1)

Architecture

┌─────────────────────────────────────────────────────────┐
│ airflow api-server (gunicorn via Python API) │
│ └── AirflowArbiter (custom Arbiter with monitoring) │
│ ├── worker 1 (UvicornWorker) │
│ ├── worker 2 (UvicornWorker) │
│ └── worker N (UvicornWorker) │
└─────────────────────────────────────────────────────────┘

Uses gunicorn's recommended extension pattern: custom AirflowArbiter subclass integrates worker monitoring directly into the arbiter loop via manage_workers(). No separate thread or subprocess needed.

Rolling Restart Flow

Spawn batch_size new workers (spawn_worker())
Wait for workers to reach target count
Kill batch_size old workers (kill_worker() - kills oldest via FIFO)
Repeat until all original workers replaced

Zero-downtime is guaranteed because new workers are spawned before old workers are killed.

Why Gunicorn Over Uvicorn Multiprocess?

Aspect	Gunicorn	Uvicorn
Memory sharing	Yes (preload + fork COW)	No (independent workers)
Rolling restarts	Yes (SIGTTOU kills oldest - FIFO)	No (SIGTTOU kills newest - LIFO)
Worker management	Always has arbiter process	No arbiter with workers=1
macOS support	Limited (setproctitle issues)	Full

The key difference: Uvicorn's SIGTTOU kills the newest worker (LIFO), while Gunicorn kills the oldest (FIFO). Rolling restarts require killing old workers, not new ones.

Why Gunicorn is Optional

Gunicorn is an optional extra (apache-airflow-core[gunicorn]) because:

Windows incompatibility: Gunicorn is Unix-only
Most users don't need it: Default uvicorn is sufficient for development and simple production setups
Different trade-offs: Some users prefer uvicorn's simplicity

Copilot

Pull request overview

This PR adds optional gunicorn support to the API server, enabling zero-downtime worker recycling to prevent memory accumulation in long-running processes. The implementation uses a GunicornMonitor to perform rolling restarts while maintaining service availability.

Changes:

Added gunicorn as an optional server type alongside the default uvicorn
Implemented GunicornMonitor for zero-downtime worker recycling with configurable refresh intervals
Added configuration options for server type, worker refresh intervals, and batch sizes

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pyproject.toml	Added gunicorn extra to main package dependencies
airflow-core/pyproject.toml	Added gunicorn as an optional dependency with version constraint
docs/spelling_wordlist.txt	Added "multiprocess" to spelling dictionary
airflow-core/src/airflow/config_templates/config.yml	Added server_type, worker_refresh_interval, and worker_refresh_batch_size configuration options
airflow-core/src/airflow/settings.py	Added GUNICORN_WORKER_READY_PREFIX constant for process title tracking
airflow-core/src/airflow/cli/commands/gunicorn_monitor.py	Implemented GunicornMonitor class for rolling worker restarts
airflow-core/src/airflow/cli/commands/api_server_command.py	Added gunicorn command building and server type routing logic
airflow-core/src/airflow/api_fastapi/gunicorn_config.py	Added gunicorn hooks for worker readiness tracking and cleanup
airflow-core/tests/unit/cli/commands/test_gunicorn_monitor.py	Comprehensive test coverage for GunicornMonitor functionality
airflow-core/newsfragments/60921.significant.rst	Release notes documenting the new feature
airflow-core/docs/extra-packages-ref.rst	Documentation for gunicorn extra package
airflow-core/docs/administration-and-deployment/web-stack.rst	User guide for server types and rolling worker restarts
.pre-commit-config.yaml	Added new files to pre-commit exclusions

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/cli/commands/api_server_command.py:1

Using verify=False disables SSL certificate verification. While the comment notes this is for localhost health checks, consider only disabling verification when the scheme is https and host is localhost/127.0.0.1 to avoid accidentally disabling verification for remote health checks.

# Licensed to the Apache Software Foundation (ASF) under one

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pierrejeambrun

Nice I think this is a great addition.

I think it would be cool to have both https://github.com/apache/airflow/pull/60804/changes, in case a particular dagbag grows too big in a shorter time than the lifecycle. (just to make sure it doesn't explode and put a limit there)

Add optional gunicorn server type for the API server that provides: - Memory sharing via preload + fork copy-on-write - Rolling worker restarts through GunicornMonitor - Correct FIFO signal handling (SIGTTOU kills oldest worker) New configuration options in [api] section: - server_type: uvicorn (default) or gunicorn - worker_refresh_interval: seconds between refresh cycles (0=disabled) - worker_refresh_batch_size: workers to refresh per cycle - master_timeout: gunicorn master timeout - reload_on_plugin_change: reload on plugin file changes Requires apache-airflow-core[gunicorn] extra for gunicorn mode.

Matches Airflow 2's webserver pattern: monitor runs in main thread, so if it crashes, the whole process exits (fail-fast). No silent degradation where gunicorn keeps running without worker recycling. Also triggers monitor when reload_on_plugin_change is enabled, even if worker_refresh_interval is 0.

…onitor This refactor changes the gunicorn worker monitoring architecture from an external thread-based approach to using a custom Arbiter subclass, which is gunicorn's recommended extension pattern. Changes: - New gunicorn_app.py with AirflowArbiter and AirflowGunicornApp - AirflowArbiter integrates worker refresh into manage_workers() loop - Removed gunicorn_monitor.py (no longer needed) - Simplified api_server_command.py (no subprocess, direct gunicorn API) - Updated tests for new architecture Benefits: - Simpler architecture (no separate thread or subprocess) - Direct access to worker state via self.WORKERS - Uses gunicorn's internal spawn_worker/kill_worker methods - Follows gunicorn's documented extension pattern

jason810496

Nice! LGTM overall and the changes make sense to me.

github-actions · 2026-02-04T09:46:17Z

Backport failed to create: v3-1-test. View the failure log Run details

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker b906080 v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…che#60940) Add optional gunicorn server type for the API server that provides: - Memory sharing via preload + fork copy-on-write - Rolling worker restarts through GunicornMonitor - Correct FIFO signal handling (SIGTTOU kills oldest worker) New configuration options in [api] section: - server_type: uvicorn (default) or gunicorn - worker_refresh_interval: seconds between refresh cycles (0=disabled) - worker_refresh_batch_size: workers to refresh per cycle - master_timeout: gunicorn master timeout - reload_on_plugin_change: reload on plugin file changes Requires apache-airflow-core[gunicorn] extra for gunicorn mode. * Run GunicornMonitor in main thread instead of daemon thread Matches Airflow 2's webserver pattern: monitor runs in main thread, so if it crashes, the whole process exits (fail-fast). No silent degradation where gunicorn keeps running without worker recycling. Also triggers monitor when reload_on_plugin_change is enabled, even if worker_refresh_interval is 0. * Refactor gunicorn support to use custom Arbiter instead of external monitor This refactor changes the gunicorn worker monitoring architecture from an external thread-based approach to using a custom Arbiter subclass, which is gunicorn's recommended extension pattern. Changes: - New gunicorn_app.py with AirflowArbiter and AirflowGunicornApp - AirflowArbiter integrates worker refresh into manage_workers() loop - Removed gunicorn_monitor.py (no longer needed) - Simplified api_server_command.py (no subprocess, direct gunicorn API) - Updated tests for new architecture Benefits: - Simpler architecture (no separate thread or subprocess) - Direct access to worker state via self.WORKERS - Uses gunicorn's internal spawn_worker/kill_worker methods - Follows gunicorn's documented extension pattern

jscheffl · 2026-04-11T08:55:36Z

Just came across this - COOL!

Fixes memory growth in long-running API servers by adding bounded LRU+TTL caching to `DBDagBag`. Previously, the internal dict cache never expired and never evicted, causing memory to grow indefinitely as DAG versions accumulated (~500 MB/day with 100+ DAGs updating daily). Two new `[api]` config options control caching: | Config | Default | Description | |--------|---------|-------------| | `dag_cache_size` | `64` | Max cached DAG versions (0 = unbounded dict, no eviction) | | `dag_cache_ttl` | `3600` | TTL in seconds (0 = LRU only, no time-based expiry) | ## Design decisions **API server only.** The scheduler continues using a plain unbounded dict with zero lock overhead (`nullcontext` instead of `RLock`). The bounded cache + lock is only created when `cache_size > 0`. **Cache thrashing prevention.** `iter_all_latest_version_dags()` (used by the DAG listing endpoint) bypasses the cache entirely. Without this, every DAG listing request would flush the hot working set and replace it with a full scan of all DAGs. **Double-checked locking.** When multiple threads miss on the same `version_id` concurrently, only the first thread queries the DB. The rest find it cached after acquiring the lock. Metrics are emitted correctly: a single lookup never counts as both a hit and a miss. **Separate model cache.** `get_serialized_dag_model()` maintains its own dict cache. The triggerer needs the full `SerializedDagModel` (for `.data`), not the deserialized `SerializedDAG` stored in the LRU/TTL cache. **Cache keying.** The cache is keyed by DAG version ID. Lookups by `dag_id` (e.g., viewing a DAG's details) always query the DB for the latest version, but the deserialized result is cached for subsequent version-specific lookups (e.g., task instance views for a specific DAG run). **Staleness.** After a DAG is updated, the API server may serve the previous version until the cached entry expires (controlled by `dag_cache_ttl`). This is documented in the config description. **Why `cachetools`.** `cachetools` is a small, pure-Python library (~1K LOC) already present as a transitive dependency via `google-auth`. It provides battle-tested `LRUCache` and `TTLCache` implementations. Pinned at `>=6.0.0` to match the FAB provider. **Why `RLock`.** `cachetools` caches are NOT thread-safe -- `.get()` mutates internal doubly-linked lists (LRU reordering) and TTL access triggers cleanup. Without synchronization, concurrent access can corrupt the data structure. ## Metrics | Metric | Type | Description | |--------|------|-------------| | `api_server.dag_bag.cache_hit` | Counter | Cache hits (including double-checked locking hits) | | `api_server.dag_bag.cache_miss` | Counter | Confirmed misses (after double-check) | | `api_server.dag_bag.cache_clear` | Counter | Cache clears | | `api_server.dag_bag.cache_size` | Gauge | Current cache size (sampled at 10%) | ## Backward compatibility - Default behavior unchanged for scheduler and triggerer (unbounded dict, no lock) - API server gets caching by default (`dag_cache_size=64`, `dag_cache_ttl=3600`) - Use `dag_cache_size=0` to restore pre-change behavior (unbounded dict) - No breaking changes to public APIs; `get_serialized_dag_model()` and `get_dag()` signatures preserved ## Related - #64326 (closed) -- similar fix with OrderedDict-based LRU, no TTL - #60940 (merged) -- gunicorn support with rolling worker restarts (complementary, handles memory growth from any source)

#66862) Fixes memory growth in long-running API servers by adding bounded LRU+TTL caching to `DBDagBag`. Previously, the internal dict cache never expired and never evicted, causing memory to grow indefinitely as DAG versions accumulated (~500 MB/day with 100+ DAGs updating daily). Two new `[api]` config options control caching: | Config | Default | Description | |--------|---------|-------------| | `dag_cache_size` | `64` | Max cached DAG versions (0 = unbounded dict, no eviction) | | `dag_cache_ttl` | `3600` | TTL in seconds (0 = LRU only, no time-based expiry) | **API server only.** The scheduler continues using a plain unbounded dict with zero lock overhead (`nullcontext` instead of `RLock`). The bounded cache + lock is only created when `cache_size > 0`. **Cache thrashing prevention.** `iter_all_latest_version_dags()` (used by the DAG listing endpoint) bypasses the cache entirely. Without this, every DAG listing request would flush the hot working set and replace it with a full scan of all DAGs. **Double-checked locking.** When multiple threads miss on the same `version_id` concurrently, only the first thread queries the DB. The rest find it cached after acquiring the lock. Metrics are emitted correctly: a single lookup never counts as both a hit and a miss. **Separate model cache.** `get_serialized_dag_model()` maintains its own dict cache. The triggerer needs the full `SerializedDagModel` (for `.data`), not the deserialized `SerializedDAG` stored in the LRU/TTL cache. **Cache keying.** The cache is keyed by DAG version ID. Lookups by `dag_id` (e.g., viewing a DAG's details) always query the DB for the latest version, but the deserialized result is cached for subsequent version-specific lookups (e.g., task instance views for a specific DAG run). **Staleness.** After a DAG is updated, the API server may serve the previous version until the cached entry expires (controlled by `dag_cache_ttl`). This is documented in the config description. **Why `cachetools`.** `cachetools` is a small, pure-Python library (~1K LOC) already present as a transitive dependency via `google-auth`. It provides battle-tested `LRUCache` and `TTLCache` implementations. Pinned at `>=6.0.0` to match the FAB provider. **Why `RLock`.** `cachetools` caches are NOT thread-safe -- `.get()` mutates internal doubly-linked lists (LRU reordering) and TTL access triggers cleanup. Without synchronization, concurrent access can corrupt the data structure. | Metric | Type | Description | |--------|------|-------------| | `api_server.dag_bag.cache_hit` | Counter | Cache hits (including double-checked locking hits) | | `api_server.dag_bag.cache_miss` | Counter | Confirmed misses (after double-check) | | `api_server.dag_bag.cache_clear` | Counter | Cache clears | | `api_server.dag_bag.cache_size` | Gauge | Current cache size (sampled at 10%) | - Default behavior unchanged for scheduler and triggerer (unbounded dict, no lock) - API server gets caching by default (`dag_cache_size=64`, `dag_cache_ttl=3600`) - Use `dag_cache_size=0` to restore pre-change behavior (unbounded dict) - No breaking changes to public APIs; `get_serialized_dag_model()` and `get_dag()` signatures preserved - #64326 (closed) -- similar fix with OrderedDict-based LRU, no TTL - #60940 (merged) -- gunicorn support with rolling worker restarts (complementary, handles memory growth from any source) (cherry picked from commit 26cbdcb) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

boring-cyborgBot added area:CLI area:ConfigTemplates area:dev-tools backport-to-v3-1-test kind:documentation labels Jan 22, 2026

kaxil marked this pull request as ready for review January 22, 2026 14:49

kaxil requested review from bugraoz93, choo121600, ephraimbuddy, jason810496, pierrejeambrun, potiuk, rawwar and shubhamraj-git as code owners January 22, 2026 14:49

kaxil mentioned this pull request Jan 22, 2026
Add configurable LRU+TTL caching for API server DAG retrieval #60804
Merged

kaxil force-pushed the gunicorn-api-server branch 2 times, most recently from b13f0a5 to 802b8e3Compare January 22, 2026 18:08

kaxil mentioned this pull request Jan 28, 2026
Add UvicornMonitor for zero-downtime API server worker recycling #60919
Closed

kaxil force-pushed the gunicorn-api-server branch from 802b8e3 to 816507eCompare January 28, 2026 18:06

kaxil requested a review from Copilot January 28, 2026 18:07

CopilotAI reviewed Jan 28, 2026
View reviewed changes

Comment threadairflow-core/docs/extra-packages-ref.rst Outdated
Comment threadairflow-core/src/airflow/cli/commands/api_server_command.py

kaxil force-pushed the gunicorn-api-server branch 3 times, most recently from 884e5d3 to 4b1ca24Compare January 29, 2026 01:21

ashb reviewed Jan 29, 2026
View reviewed changes

Comment threadairflow-core/src/airflow/api_fastapi/gunicorn_monitor.py Outdated

pierrejeambrun reviewed Jan 29, 2026
View reviewed changes

kaxil force-pushed the gunicorn-api-server branch 3 times, most recently from ec264a8 to aa59022Compare January 30, 2026 17:42

kaxil added 2 commits January 30, 2026 18:39

kaxil force-pushed the gunicorn-api-server branch from 4e0812b to 048b919Compare January 30, 2026 18:39

kaxil added this to the Airflow 3.2.0 milestone Jan 30, 2026

jason810496 reviewed Feb 2, 2026
View reviewed changes

Comment threadairflow-core/newsfragments/60921.significant.rst

This was referenced Feb 4, 2026
Kubernetes best practices and Helm support for API server rollout restarts #61430
Open
Helm chart support for periodic API server rollout restarts on Kubernetes #61432
Open
Document best practices for using ‎uvicorn API server on Kubernetes #61431
Closed

jason810496 approved these changes Feb 4, 2026
View reviewed changes

kaxil merged commit b906080 into apache:mainFeb 4, 2026
435 of 437 checks passed

kaxil deleted the gunicorn-api-server branch February 4, 2026 09:45

henry3260 mentioned this pull request Feb 8, 2026
chart: add api-server rollout restart cronjob #61636
Closed
1 task

pierrejeambrun mentioned this pull request Feb 16, 2026
API - UI Optimization #62020
Closed

kaxil mentioned this pull request Feb 26, 2026
Possible deadlock in 25.1.0 when preload_app=True during worker boot benoitc/gunicorn#3529
Closed

ashb mentioned this pull request Feb 26, 2026
Make the gunicorn API server respect GUNICORN_CMD_ARGS again #62522
Merged

kaxil mentioned this pull request Apr 10, 2026
Add FAQ entry for API server memory growth with gunicorn worker recycling #65036
Merged

Navigation Menu

Uh oh!

Add gunicorn support for API server with rolling worker restarts - #60940

Add gunicorn support for API server with rolling worker restarts#60940
kaxil merged 3 commits into
apache:mainfrom
astronomer:gunicorn-api-server

kaxil commented Jan 22, 2026•
edited
Loading

Uh oh!

CopilotAI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Uh oh!

jason810496 left a comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Feb 4, 2026

Uh oh!

jscheffl commented Apr 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

kaxil commented Jan 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Memory Impact

Usage

Configuration

Architecture

Rolling Restart Flow

Why Gunicorn Over Uvicorn Multiprocess?

Why Gunicorn is Optional

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Feb 4, 2026

Backport failed to create: v3-1-test. View the failure log Run details

Uh oh!

jscheffl commented Apr 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kaxil commented Jan 22, 2026•
edited
Loading