Add configurable LRU+TTL caching for API server DAG retrieval

[kaxil]

Summary

Fixes memory growth in long-running API servers by adding bounded LRU+TTL caching to DBDagBag. Previously, the internal dict cache never expired and never evicted, causing memory to grow indefinitely as DAG versions accumulated (~500 MB/day with 100+ DAGs updating daily).

Two new [api] config options control caching:

Config	Default	Description
dag_cache_size	64	Max cached DAG versions (0 = unbounded dict, no eviction)
dag_cache_ttl	3600	TTL in seconds (0 = LRU only, no time-based expiry)

Design decisions

API server only. The scheduler continues using a plain unbounded dict with zero lock overhead (nullcontext instead of RLock). The bounded cache + lock is only created when cache_size > 0.

Cache thrashing prevention.iter_all_latest_version_dags() (used by the DAG listing endpoint) bypasses the cache entirely. Without this, every DAG listing request would flush the hot working set and replace it with a full scan of all DAGs.

Double-checked locking. When multiple threads miss on the same version_id concurrently, only the first thread queries the DB. The rest find it cached after acquiring the lock. Metrics are emitted correctly: a single lookup never counts as both a hit and a miss.

Separate model cache.get_serialized_dag_model() maintains its own dict cache. The triggerer needs the full SerializedDagModel (for .data), not the deserialized SerializedDAG stored in the LRU/TTL cache.

Cache keying. The cache is keyed by DAG version ID. Lookups by dag_id (e.g., viewing a DAG's details) always query the DB for the latest version, but the deserialized result is cached for subsequent version-specific lookups (e.g., task instance views for a specific DAG run).

Staleness. After a DAG is updated, the API server may serve the previous version until the cached entry expires (controlled by dag_cache_ttl). This is documented in the config description.

Why cachetools.cachetools is a small, pure-Python library (~1K LOC) already present as a transitive dependency via google-auth. It provides battle-tested LRUCache and TTLCache implementations. Pinned at >=6.0.0 to match the FAB provider.

Why RLock.cachetools caches are NOT thread-safe -- .get() mutates internal doubly-linked lists (LRU reordering) and TTL access triggers cleanup. Without synchronization, concurrent access can corrupt the data structure.

Metrics

Metric	Type	Description
api_server.dag_bag.cache_hit	Counter	Cache hits (including double-checked locking hits)
api_server.dag_bag.cache_miss	Counter	Confirmed misses (after double-check)
api_server.dag_bag.cache_clear	Counter	Cache clears
api_server.dag_bag.cache_size	Gauge	Current cache size (sampled at 10%)

Backward compatibility

• Default behavior unchanged for scheduler and triggerer (unbounded dict, no lock)
• API server gets caching by default (dag_cache_size=64, dag_cache_ttl=3600)
• Use dag_cache_size=0 to restore pre-change behavior (unbounded dict)
• No breaking changes to public APIs; get_serialized_dag_model() and get_dag() signatures preserved

Related

• Fix API server memory leak: bound DBDagBag version cache with LRU eviction #64326 (closed) -- similar fix with OrderedDict-based LRU, no TTL
• Add gunicorn support for API server with rolling worker restarts #60940 (merged) -- gunicorn support with rolling worker restarts (complementary, handles memory growth from any source)

[jason810496]

Nice improvement! LGTM overall.

With 100+ DAGs updating daily, memory grows ~500 MB/day, eventually causing OOM.

It’s surprising that the API server without cache eviction can cause system instability.

[potiuk]

It’s surprising that the API server without cache eviction can cause system instability.

One question. In gunicorn in Airflow 2 we had a way simpler solution. Simply the uvicorn servers have restarted every few (tens?) of minutes or every N requests - effectively cleaning the cache and also getting rid of some other side effects (and for example reloading UI plugins). Since api-server (except the cache) is essentially stateless, that did not have almost any negative side effects - except some load caused on the startup time and database refreshing happening then, but that's not much different than the caching implemented here provides.

Additionally that approach was far more "resilient" to any kinds of accumulation-type bugs, yes it was hiding them as well, but the overall stability and resilience to any kind of mistakes made with memory usage, or side-effects of imports or global state sharung was eventually high-up.

This approach is named "software rejuvenation" https://ieeexplore.ieee.org/document/466961 - there are some studies and recommendations to use it as it is effectively way more resilient and in complex systems it allows to handle much wide range of issues.

Maybe we should explore that as well (or instead) - I am not sure if fast-api/starlette has similar concept, but in case of all kinds of stateless webserves, the technique of restarting them gracefully while load-balancing requests has a long proven history.

Should we possibly do it instead of caching LRU/TTL ? That seems way more robust if this is easy and supported by Fast API

[kaxil]

One question. In gunicorn in Airflow 2 we had a way simpler solution. Simply the uvicorn servers have restarted every few (tens?) of minutes or every N requests - effectively cleaning the cache and also getting rid of some other side effects (and for example reloading UI plugins). Since api-server (except the cache) is essentially stateless, that did not have almost any negative side effects - except some load caused on the startup time and database refreshing happening then, but that's not much different than the caching implemented here provides.

Good idea, worth trying that out too. Marking this as draft to playaround with it

[kaxil]

@potiuk Alternate approach is here in #60919 which uses pure uvicorn signals to increment/decrement workers.

Limitations:

• workers=1 needs workaround (briefly scales to 2 during refresh)
• uvicorn's SIGTTOU kills newest worker (LIFO) unlike gunicorn which kills oldest (FIFO), so we send SIGTERM directly to old PIDs instead

The LIFO thing is worth noting since it's a non-obvious difference between uvicorn and gunicorn that anyone else looking at this would run into.

[kaxil]

Another alternative using gunicorn is in #60940

[kaxil]

Worth now comparing them side-by-side -- and will let other review all 3 of them. Will check back next week.

Based on trying it out, I don't like #60919 for gotcha's mentioned in that PR (and in the comment above).

#60940 is what I prefer as there are no such limitatons and we get benefits of gunicorn and uvicorn (gunicorn for worker lifeycle management, preload etc and uvicorn for async perf). We could pair it with Helm chart changes to cycle it after X duration and/or this PR.

[dheerajturaga]

Great! Thanks for this!

[dheerajturaga]

One reason why I ended up here was some false dag version increments due to dag meta data changes. Also some dags used "datetime.now()" as start date rather than a fixed startdate.

@dheerajturaga We should fix those underlying issue too. Could you create GH issue with what metadata is causing false increase? and for datetime.now --> if it is for start date -- that should never be used like that

I did do a deep dive into this. All of it was from datetime.now in start date.
Many of our dags just need a simple cron and start date in that scenario is largely meaningless so users just put something in there to get by

[pierrejeambrun]

A few questions/suggestion but looking good overall.

I'm a big confused by the use_cache logic. I would expect no caching if use_cache is not enabled, but it looks like it's controlling logging but some paths seem to still be leveraging cache even if use_cache=False

[pierrejeambrun]

Thanks! LGTM

[vatsrahul1001]

backport PR up for review #66862