Report Dag cache metrics under each component's own namespace by jason810496 · Pull Request #71815 · apache/airflow

jason810496 · 2026-08-19T05:54:44Z

Depends on:Honor the API server Dag cache TTL when no size limit is set #71814
Stacked with:Make the scheduler Dag cache size and TTL configurable #71816
Diff for early review:fix/api-dag-cache-ttl-ignored...feat/dagbag-cache-metric-namespace

Why

Every configured DBDagBag emitted cache counters under api_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.

The cache settings and metrics also made every DBDagBag caller opt into an interface intended only for long-lived component caches.

How

DBDagBag remains the simple base implementation for callers that do not configure eviction or cache metrics. CachedDBDagBag owns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.

The API server always constructs CachedDBDagBag from its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructs CachedDBDagBag under scheduler.dag_bag.*.

The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.

What

Reports API server cache metrics under api_server.dag_bag.*.
Reports scheduler cache metrics under scheduler.dag_bag.*.
Keeps short-lived and compatibility callers on the base DBDagBag.
Preserves the existing 0/0 no-eviction behavior while keeping configured component caches on CachedDBDagBag.

Supersedes #71276.

Was generative AI tooling used to co-author this PR?

Yes, with help from Claude Code (Opus 5) and Codex (GPT-5), following the project's AI-assisted contribution guidelines.

Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits.

uranusjr · 2026-08-20T06:51:22Z

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

jason810496

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

It makes sense to separate them by purpose, thanks for the suggestion. I just updated it.

github-actions · 2026-08-21T07:07:59Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker bcf1dbf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

jason810496 · 2026-08-19T05:54:44Z

Depends on:Honor the API server Dag cache TTL when no size limit is set #71814
Stacked with:Make the scheduler Dag cache size and TTL configurable #71816
Diff for early review:fix/api-dag-cache-ttl-ignored...feat/dagbag-cache-metric-namespace

Why

Every configured DBDagBag emitted cache counters under api_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.

The cache settings and metrics also made every DBDagBag caller opt into an interface intended only for long-lived component caches.

How

DBDagBag remains the simple base implementation for callers that do not configure eviction or cache metrics. CachedDBDagBag owns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.

The API server always constructs CachedDBDagBag from its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructs CachedDBDagBag under scheduler.dag_bag.*.

The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.

What

Reports API server cache metrics under api_server.dag_bag.*.
Reports scheduler cache metrics under scheduler.dag_bag.*.
Keeps short-lived and compatibility callers on the base DBDagBag.
Preserves the existing 0/0 no-eviction behavior while keeping configured component caches on CachedDBDagBag.

Supersedes #71276.

Was generative AI tooling used to co-author this PR?

Yes, with help from Claude Code (Opus 5) and Codex (GPT-5), following the project's AI-assisted contribution guidelines.

Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits.

uranusjr · 2026-08-20T06:51:22Z

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

jason810496

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

It makes sense to separate them by purpose, thanks for the suggestion. I just updated it.

github-actions · 2026-08-21T07:07:59Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker bcf1dbf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

jason810496 · 2026-08-19T05:54:44Z

Depends on:Honor the API server Dag cache TTL when no size limit is set #71814
Stacked with:Make the scheduler Dag cache size and TTL configurable #71816
Diff for early review:fix/api-dag-cache-ttl-ignored...feat/dagbag-cache-metric-namespace

Why

Every configured DBDagBag emitted cache counters under api_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.

The cache settings and metrics also made every DBDagBag caller opt into an interface intended only for long-lived component caches.

How

DBDagBag remains the simple base implementation for callers that do not configure eviction or cache metrics. CachedDBDagBag owns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.

The API server always constructs CachedDBDagBag from its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructs CachedDBDagBag under scheduler.dag_bag.*.

The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.

What

Reports API server cache metrics under api_server.dag_bag.*.
Reports scheduler cache metrics under scheduler.dag_bag.*.
Keeps short-lived and compatibility callers on the base DBDagBag.
Preserves the existing 0/0 no-eviction behavior while keeping configured component caches on CachedDBDagBag.

Supersedes #71276.

Was generative AI tooling used to co-author this PR?

Yes, with help from Claude Code (Opus 5) and Codex (GPT-5), following the project's AI-assisted contribution guidelines.

Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits.

uranusjr · 2026-08-20T06:51:22Z

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

jason810496

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

It makes sense to separate them by purpose, thanks for the suggestion. I just updated it.

github-actions · 2026-08-21T07:07:59Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker bcf1dbf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

jason810496 · 2026-08-19T05:54:44Z

Depends on:Honor the API server Dag cache TTL when no size limit is set #71814
Stacked with:Make the scheduler Dag cache size and TTL configurable #71816
Diff for early review:fix/api-dag-cache-ttl-ignored...feat/dagbag-cache-metric-namespace

Why

Every configured DBDagBag emitted cache counters under api_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.

The cache settings and metrics also made every DBDagBag caller opt into an interface intended only for long-lived component caches.

How

DBDagBag remains the simple base implementation for callers that do not configure eviction or cache metrics. CachedDBDagBag owns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.

The API server always constructs CachedDBDagBag from its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructs CachedDBDagBag under scheduler.dag_bag.*.

The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.

What

Reports API server cache metrics under api_server.dag_bag.*.
Reports scheduler cache metrics under scheduler.dag_bag.*.
Keeps short-lived and compatibility callers on the base DBDagBag.
Preserves the existing 0/0 no-eviction behavior while keeping configured component caches on CachedDBDagBag.

Supersedes #71276.

Was generative AI tooling used to co-author this PR?

Yes, with help from Claude Code (Opus 5) and Codex (GPT-5), following the project's AI-assisted contribution guidelines.

Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits.

uranusjr · 2026-08-20T06:51:22Z

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

jason810496

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

It makes sense to separate them by purpose, thanks for the suggestion. I just updated it.

github-actions · 2026-08-21T07:07:59Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker bcf1dbf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

jason810496 · 2026-08-19T05:54:44Z

Depends on:Honor the API server Dag cache TTL when no size limit is set #71814
Stacked with:Make the scheduler Dag cache size and TTL configurable #71816
Diff for early review:fix/api-dag-cache-ttl-ignored...feat/dagbag-cache-metric-namespace

Why

Every configured DBDagBag emitted cache counters under api_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.

The cache settings and metrics also made every DBDagBag caller opt into an interface intended only for long-lived component caches.

How

DBDagBag remains the simple base implementation for callers that do not configure eviction or cache metrics. CachedDBDagBag owns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.

The API server always constructs CachedDBDagBag from its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructs CachedDBDagBag under scheduler.dag_bag.*.

The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.

What

Reports API server cache metrics under api_server.dag_bag.*.
Reports scheduler cache metrics under scheduler.dag_bag.*.
Keeps short-lived and compatibility callers on the base DBDagBag.
Preserves the existing 0/0 no-eviction behavior while keeping configured component caches on CachedDBDagBag.

Supersedes #71276.

Was generative AI tooling used to co-author this PR?

Yes, with help from Claude Code (Opus 5) and Codex (GPT-5), following the project's AI-assisted contribution guidelines.

Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits.

uranusjr · 2026-08-20T06:51:22Z

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

jason810496

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

It makes sense to separate them by purpose, thanks for the suggestion. I just updated it.

github-actions · 2026-08-21T07:07:59Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker bcf1dbf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

jason810496 · 2026-08-19T05:54:44Z

Depends on:Honor the API server Dag cache TTL when no size limit is set #71814
Stacked with:Make the scheduler Dag cache size and TTL configurable #71816
Diff for early review:fix/api-dag-cache-ttl-ignored...feat/dagbag-cache-metric-namespace

Why

Every configured DBDagBag emitted cache counters under api_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.

The cache settings and metrics also made every DBDagBag caller opt into an interface intended only for long-lived component caches.

How

DBDagBag remains the simple base implementation for callers that do not configure eviction or cache metrics. CachedDBDagBag owns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.

The API server always constructs CachedDBDagBag from its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructs CachedDBDagBag under scheduler.dag_bag.*.

The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.

What

Reports API server cache metrics under api_server.dag_bag.*.
Reports scheduler cache metrics under scheduler.dag_bag.*.
Keeps short-lived and compatibility callers on the base DBDagBag.
Preserves the existing 0/0 no-eviction behavior while keeping configured component caches on CachedDBDagBag.

Supersedes #71276.

Was generative AI tooling used to co-author this PR?

Yes, with help from Claude Code (Opus 5) and Codex (GPT-5), following the project's AI-assisted contribution guidelines.

Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits.

uranusjr · 2026-08-20T06:51:22Z

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

jason810496

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

It makes sense to separate them by purpose, thanks for the suggestion. I just updated it.

github-actions · 2026-08-21T07:07:59Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker bcf1dbf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

jason810496 · 2026-08-19T05:54:44Z

Depends on:Honor the API server Dag cache TTL when no size limit is set #71814
Stacked with:Make the scheduler Dag cache size and TTL configurable #71816
Diff for early review:fix/api-dag-cache-ttl-ignored...feat/dagbag-cache-metric-namespace

Why

Every configured DBDagBag emitted cache counters under api_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.

The cache settings and metrics also made every DBDagBag caller opt into an interface intended only for long-lived component caches.

How

DBDagBag remains the simple base implementation for callers that do not configure eviction or cache metrics. CachedDBDagBag owns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.

The API server always constructs CachedDBDagBag from its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructs CachedDBDagBag under scheduler.dag_bag.*.

The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.

What

Reports API server cache metrics under api_server.dag_bag.*.
Reports scheduler cache metrics under scheduler.dag_bag.*.
Keeps short-lived and compatibility callers on the base DBDagBag.
Preserves the existing 0/0 no-eviction behavior while keeping configured component caches on CachedDBDagBag.

Supersedes #71276.

Was generative AI tooling used to co-author this PR?

Yes, with help from Claude Code (Opus 5) and Codex (GPT-5), following the project's AI-assisted contribution guidelines.

Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits.

uranusjr · 2026-08-20T06:51:22Z

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

jason810496

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

It makes sense to separate them by purpose, thanks for the suggestion. I just updated it.

github-actions · 2026-08-21T07:07:59Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker bcf1dbf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

jason810496 · 2026-08-19T05:54:44Z

Depends on:Honor the API server Dag cache TTL when no size limit is set #71814
Stacked with:Make the scheduler Dag cache size and TTL configurable #71816
Diff for early review:fix/api-dag-cache-ttl-ignored...feat/dagbag-cache-metric-namespace

Why

Every configured DBDagBag emitted cache counters under api_server.dag_bag.*. Once the scheduler gained a bounded cache in #71704, scheduler traffic was counted against the API server's series, so operators could not distinguish the two components.

The cache settings and metrics also made every DBDagBag caller opt into an interface intended only for long-lived component caches.

How

DBDagBag remains the simple base implementation for callers that do not configure eviction or cache metrics. CachedDBDagBag owns LRU, TTL, and no-eviction cache modes, along with thread safety and the required component metric namespace.

The API server always constructs CachedDBDagBag from its cache configuration. With size and TTL both set to 0, the subclass uses a plain unbounded dict and performs no eviction. The scheduler constructs CachedDBDagBag under scheduler.dag_bag.*.

The metrics registry check now matches every static part of a dynamic metric name, including names assembled from a caller-supplied prefix.

What

Reports API server cache metrics under api_server.dag_bag.*.
Reports scheduler cache metrics under scheduler.dag_bag.*.
Keeps short-lived and compatibility callers on the base DBDagBag.
Preserves the existing 0/0 no-eviction behavior while keeping configured component caches on CachedDBDagBag.

Supersedes #71276.

Was generative AI tooling used to co-author this PR?

Yes, with help from Claude Code (Opus 5) and Codex (GPT-5), following the project's AI-assisted contribution guidelines.

Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits.

uranusjr · 2026-08-20T06:51:22Z

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

jason810496

Instead of an argument in DBDagBag to make it double-duty, I feel it’s cleaner if we make the _use_cache=True variant a separate subclass i.e. class CachedDBDagBag. This way, the arguments will be cleaner (no more complicated optional arguments + checks in __init__, and the stats functions only need to be on the class they’re actually needed.

It makes sense to separate them by purpose, thanks for the suggestion. I just updated it.

github-actions · 2026-08-21T07:07:59Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker bcf1dbf v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…pace (apache#71815) * Report Dag cache metrics under each component's own namespace Every DBDagBag emitted its cache counters under api_server.dag_bag.*, so once the scheduler gained a cache its traffic was silently counted against the API server's series. An operator reading cache_hit or cache_size could not tell the two components apart, and the API server's numbers became wrong rather than merely incomplete. Each caller now supplies the namespace it reports under, and a cache built without one fails at construction instead of emitting a partially-formed metric name mid-request. The registry check matched dynamic metric names only by the static prefix ahead of their first variable, which cannot express a name assembled from a caller supplied prefix. It now matches on all static parts wherever the variable sits. * Scope Dag cache configuration to cached database Dag bags DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit. * Preserve no-eviction Dag cache configuration * Avoid redundant Dag cache checks (cherry picked from commit bcf1dbf) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

boring-cyborgBot added area:API Airflow's REST/HTTP API area:ConfigTemplates area:dev-tools area:Scheduler including HA (high availability) scheduler backport-to-v3-3-test Backport to v3-3-test kind:documentation labels Aug 19, 2026

jason810496 added this to the Airflow 3.3.2 milestone Aug 19, 2026

jason810496 force-pushed the feat/dagbag-cache-metric-namespace branch from 19d7b1d to f6847ccCompare August 19, 2026 07:25

jason810496 force-pushed the feat/dagbag-cache-metric-namespace branch from f6847cc to 8b2ad41Compare August 20, 2026 06:06

jason810496 marked this pull request as ready for review August 20, 2026 06:13

jason810496 requested review from XD-DENG, amoghrajesh, ashb, bugraoz93, choo121600, ephraimbuddy, gopidesupavan, henry3260, jscheffl, pierrejeambrun, potiuk, rawwar and shubhamraj-git as code owners August 20, 2026 06:13

jason810496 requested a review from uranusjr August 20, 2026 06:13

jason810496 marked this pull request as draft August 20, 2026 08:08

jason810496 added 2 commits August 20, 2026 08:46

Scope Dag cache configuration to cached database Dag bags
716ca5f
DBDagBag is used by callers that do not configure cache eviction or metrics. Keeping those concerns behind a dedicated subtype prevents optional constructor combinations and makes metric ownership explicit.

Preserve no-eviction Dag cache configuration
419421f

jason810496 requested a lite review from Copilot August 20, 2026 09:35

This comment was marked as off-topic.
Sign in to view

jason810496 marked this pull request as ready for review August 20, 2026 11:09

jason810496 commented Aug 20, 2026
View reviewed changes

uranusjr reviewed Aug 21, 2026
View reviewed changes

Comment threadairflow-core/src/airflow/models/dagbag.py Outdated

uranusjr reviewed Aug 21, 2026
View reviewed changes

Comment threadairflow-core/src/airflow/api_fastapi/common/dagbag.py

uranusjr approved these changes Aug 21, 2026
View reviewed changes

jason810496 marked this pull request as draft August 21, 2026 02:15

Avoid redundant Dag cache checks
b19dbfa

jason810496 requested a lite review from Copilot August 21, 2026 02:40

This comment was marked as off-topic.
Sign in to view

jason810496 marked this pull request as ready for review August 21, 2026 06:13

jason810496 merged commit bcf1dbf into apache:mainAug 21, 2026
162 checks passed