AIP-76: Hold Dag run until all upstream partitions arrive

[Lee-W]

Why

Closes: #59294

Why

Asset-partitioned Dags that aggregate many upstream slices into one downstream period (e.g., 60-minute-level events rolling up into one hourly Dag run) had no way to express that requirement — the scheduler would fire the downstream run as soon as any single upstream partition arrived.

This PR implements the rollup building block from AIP-76: a Window type that enumerates the full set of upstream partitions required for a downstream period, a RollupMapper that wires a source mapper to a window, and the scheduler logic to gate Dag runs until every required upstream key is present.

What

Partition mappers/windows

• Add Window ABC and six concrete implementations (HourWindow, DayWindow, WeekWindow, MonthWindow, QuarterWindow, YearWindow) to both airflow-core and the Task SDK
• Add RollupMapper that composes a source_mapper with a Window and exposes to_upstream(downstream_key) → frozenset[str]
• Add decode_downstream / encode_upstream hooks to PartitionMapper and implement them in _BaseTemporalMapper; StartOfWeekMapper gets a regex-based override because %V is ambiguous with strptime.
• Add week_start parameter to StartOfWeekMapper for non-Monday week starts

Scheduler

• Rewrite _create_dagruns_for_partitioned_asset_dags to bulk-fetch serialized Dags and partition-key logs, removing N+1 queries, and cap per-tick work at MAX_PARTITION_DAG_RUNS_PER_TICK
• Add _resolve_asset_partition_status / _check_rollup_asset_status to evaluate rollup satisfaction; non-rollup assets continue to satisfy immediately

Serialization

• Add encode_window / decode_window and extend mapper encoder/decoder to round-trip RollupMapper and all Window subclasses

UI / API

• Enrich next_run_assets endpoint with per-asset received_count, required_count, received_keys, required_keys, and is_rollup for partitioned Dags
• Update AssetNode and AssetSchedule components to surface rollup progress (e.g. "12 / 24 received")
• Add AssetProgressCell for inline progress in the Dags list

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

withDAG(
dag_id="daily_team_a_rollup",
schedule=PartitionedAssetTimetable(
assets=team_a_player_stats,
default_partition_mapper=RollupMapper(
source_mapper=StartOfDayMapper(),
window=DayWindow(),
),
),
catchup=False,
tags=["player-stats", "rollup"],
):
""" First rollup level: 24 hourly partitions of ``team_a_player_stats`` → one daily summary. ``StartOfDayMapper`` normalizes each upstream hourly timestamp (``%Y-%m-%dT%H:%M:%S``) to its day-start (``%Y-%m-%d``); ``DayWindow`` declares the downstream run needs all 24 hourly partitions before firing. Publishes ``daily_team_a`` so the monthly rollup below can consume it. """@task(outlets=[daily_team_a])defsummarise_team_a_day(dag_run=None):
"""Produce the full-day rollup once every hour has arrived."""ifTYPE_CHECKING:
assertdag_runprint(f"All 24 hourly partitions received. Day: {dag_run.partition_key}")
summarise_team_a_day()

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Copilot]

Pull request overview

Implements “rollup” support for partition mappers (AIP-76) by introducing a RollupMapper interface with to_upstream() and using it in the scheduler to wait for a complete set of upstream partition keys before creating partitioned asset-triggered DAG runs.

Changes:

• Add RollupMapper base class (core + task SDK) with an abstract to_upstream() contract.
• Implement to_upstream() for weekly and monthly temporal mappers (core + task SDK).
• Update the scheduler’s partitioned-asset DAG-run creation logic to enforce rollup completeness when a mapper supports it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
task-sdk/src/airflow/sdk/definitions/partition_mappers/base.py	Introduces SDK-side RollupMapper abstraction.
task-sdk/src/airflow/sdk/definitions/partition_mappers/temporal.py	Adds SDK to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/timetables/base.py	Adds get_partition_mapper() hook to the Timetable protocol.
airflow-core/src/airflow/partition_mappers/base.py	Introduces core-side RollupMapper abstraction.
airflow-core/src/airflow/partition_mappers/temporal.py	Adds core to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Uses rollup mapper behavior to decide when partitioned asset-triggered DAG runs are ready.

[Lee-W]

from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
CronPartitionTimetable,
PartitionedAssetTimetable,
WeeklyRollupMapper,
task,
)
daily_sales=Asset(uri="file://incoming/sales/daily.csv", name="daily_sales")
# Upstream Dag: produces one partition per day (key format: "2024-01-15T00:00:00")withDAG(
dag_id="ingest_daily_sales",
schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
):
@task(outlets=[daily_sales])defingest():
passingest()
# Downstream Dag: runs once all 7 daily partitions for a week have arrivedwithDAG(
dag_id="weekly_sales_report",
schedule=PartitionedAssetTimetable(
assets=daily_sales,
default_partition_mapper=WeeklyRollupMapper(),
),
catchup=False,
):
@taskdefgenerate_report(dag_run=None):
# dag_run.partition_key will be the week key, e.g. "2024-01-15 (W03)"print(dag_run.partition_key)
generate_report()

[Lee-W]

The backend part is basically wrapped up, but the frontend and API side need some work. The UI is quite weired for these cases now

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/partitioned_dag_runs.py:1

• Counting PartitionedAssetKeyLog.id can over-count when duplicate log rows exist for the same upstream partition key (e.g. retries/dup inserts), inflating total_received and potentially showing the run as satisfiable earlier than it should be. Consider counting distinct PartitionedAssetKeyLog.source_partition_key (and/or a distinct composite of (asset_id, source_partition_key)) to match the scheduler’s set-based satisfaction semantics.

# Licensed to the Apache Software Foundation (ASF) under one

[dstandish]

Hey @Lee-W

I was a little surprised to see so many lines for this change so I asked Claude to help me review whether there's any unnecessary complexity. Below are its findings. They seem plausible. What do you think?

---

The core design is clean (the Window types are tiny generators,
RollupMapper.to_upstream is a clear decode→expand→encode, and the scheduler gate comes down to one
expected.issubset(actual) check), and the test coverage on the scheduler paths is excellent. The
HA hardening in _create_dagruns_for_partitioned_asset_dags (bulk-fetch over the old N+1,
with_row_locks(skip_locked=True), the per-tick cap and deterministic order_by) all looks right.

One theme I'd like to resolve before merge: a meaningful chunk of complexity is self-induced,
concentrated in partition_mappers/temporal.py.

_compile_output_format_regex looks like more generality than the feature needs. Only two mappers
can't round-trip through strptime: StartOfWeekMapper (%V isn't strptime-parseable) and
StartOfQuarterMapper ({quarter} isn't a directive). The actual requirement is "recover Y/m/d" and
"recover Y/quarter". But the decode that motivated the whole compiler ends up discarding the token
it was built for:

StartOfWeekMapper.decode_downstream — %V is captured but never read

return datetime(int(match["Y"]), int(match["m"]), int(match["d"]))

Both decodes collapse to a dedicated per-mapper regex:

_WEEK_RE = re.compile(r"(?P\d{4})-(?P\d{2})-(?P\d{2})")
_QUARTER_RE = re.compile(r"(?P\d{4})-Q(?P[1-4])")

That drops the directive table, the {name} placeholder machinery, the placeholder_patterns arg,
and the five compile-time ValueError branches — plus the test class that only exists to exercise
that invented surface (test_rejects_adjacent_default_pattern_placeholders,
test_adjacent_placeholders_allowed_when_one_is_narrowed,
test_separator_between_default_placeholders_is_allowed). Roughly ~120 lines + tests, with
identical behavior for every shipped mapper. Could we go with the dedicated regexes for now and
add generality if/when a custom mapper actually needs it?

Follow-on: if the temporal decode simplifies, the base-class guard scaffolding in
partition_mappers/base.py — the init_subclass XOR check, expected_decoded_type, and the
runtime pairing check in RollupMapper.init — largely loses its purpose for the shipped
mappers, since it mostly exists to make the general decode/encode pair safe. Worth reassessing
whether it belongs now or arrives with the first real custom mapper.

Question (non-blocking): the audit-log path (_record_partition_audit_log /
_record_stale_apdr_audit_log) opens independent scoped=False sessions to survive an outer
rollback, plus a process-lifetime dedup set. For advisory Log rows where self.log.exception(...)
already records the same thing, is the rollback-survival guarantee worth the machinery for v1, or
would a plain log line do?

[Lee-W]

Hey @dstandish , thanks for reviewing. a few responses to the questions

Could we go with the dedicated regexes for now and add generality if/when a custom mapper actually needs it?

The regex compiler was added per @uranusjr's review: without it, a custom output_format forces the user to also override decode_downstream, which TP found surprising. Hardcoding _WEEK_RE / _QUARTER_RE saves the ~120 lines but is not that users friendly

is the rollback-survival guarantee worth the machinery for v1, or would a plain log line do?

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

[uranusjr]

I still find it somewhat awkward the UI API endpoints need to load the serialized dag to calculate data. But I guess it’s not too useful to overthink this unless there are known performance issues.

Should we add some links to Starlette for the format-parsing logic?

[dstandish]

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

I don't really have a strong feeling about that. I tried to look for it in the code but could not find it.

And I don't stand in the way of merging this thing....

But... I can't help feeling that this PR is bigger than it should be. It's feels so big that it's kind of overwhelming to properly review. It spreads the reviewer's attention over a very wide area.

[Lee-W]

Should we add some links to Starlette for the format-parsing logic?

yep, added a pointer to Starlette's compile_path in the _compile_output_format_regex docstring.

[potiuk]

@uranusjr's approval here is from Jun 2 (aa9f248), but there have been ~9 commits since — scheduler refactors, the partition-Dag-run clearing fix, mapper narrowing, etc. Worth a re-review so the green check reflects the current state before this merges. @uranusjr, could you re-confirm when you get a chance?

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[Lee-W]

Why

Closes: #59294

Why

Asset-partitioned Dags that aggregate many upstream slices into one downstream period (e.g., 60-minute-level events rolling up into one hourly Dag run) had no way to express that requirement — the scheduler would fire the downstream run as soon as any single upstream partition arrived.

This PR implements the rollup building block from AIP-76: a Window type that enumerates the full set of upstream partitions required for a downstream period, a RollupMapper that wires a source mapper to a window, and the scheduler logic to gate Dag runs until every required upstream key is present.

What

Partition mappers/windows

• Add Window ABC and six concrete implementations (HourWindow, DayWindow, WeekWindow, MonthWindow, QuarterWindow, YearWindow) to both airflow-core and the Task SDK
• Add RollupMapper that composes a source_mapper with a Window and exposes to_upstream(downstream_key) → frozenset[str]
• Add decode_downstream / encode_upstream hooks to PartitionMapper and implement them in _BaseTemporalMapper; StartOfWeekMapper gets a regex-based override because %V is ambiguous with strptime.
• Add week_start parameter to StartOfWeekMapper for non-Monday week starts

Scheduler

• Rewrite _create_dagruns_for_partitioned_asset_dags to bulk-fetch serialized Dags and partition-key logs, removing N+1 queries, and cap per-tick work at MAX_PARTITION_DAG_RUNS_PER_TICK
• Add _resolve_asset_partition_status / _check_rollup_asset_status to evaluate rollup satisfaction; non-rollup assets continue to satisfy immediately

Serialization

• Add encode_window / decode_window and extend mapper encoder/decoder to round-trip RollupMapper and all Window subclasses

UI / API

• Enrich next_run_assets endpoint with per-asset received_count, required_count, received_keys, required_keys, and is_rollup for partitioned Dags
• Update AssetNode and AssetSchedule components to surface rollup progress (e.g. "12 / 24 received")
• Add AssetProgressCell for inline progress in the Dags list

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

withDAG(
dag_id="daily_team_a_rollup",
schedule=PartitionedAssetTimetable(
assets=team_a_player_stats,
default_partition_mapper=RollupMapper(
source_mapper=StartOfDayMapper(),
window=DayWindow(),
),
),
catchup=False,
tags=["player-stats", "rollup"],
):
""" First rollup level: 24 hourly partitions of ``team_a_player_stats`` → one daily summary. ``StartOfDayMapper`` normalizes each upstream hourly timestamp (``%Y-%m-%dT%H:%M:%S``) to its day-start (``%Y-%m-%d``); ``DayWindow`` declares the downstream run needs all 24 hourly partitions before firing. Publishes ``daily_team_a`` so the monthly rollup below can consume it. """@task(outlets=[daily_team_a])defsummarise_team_a_day(dag_run=None):
"""Produce the full-day rollup once every hour has arrived."""ifTYPE_CHECKING:
assertdag_runprint(f"All 24 hourly partitions received. Day: {dag_run.partition_key}")
summarise_team_a_day()

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Copilot]

Pull request overview

Implements “rollup” support for partition mappers (AIP-76) by introducing a RollupMapper interface with to_upstream() and using it in the scheduler to wait for a complete set of upstream partition keys before creating partitioned asset-triggered DAG runs.

Changes:

• Add RollupMapper base class (core + task SDK) with an abstract to_upstream() contract.
• Implement to_upstream() for weekly and monthly temporal mappers (core + task SDK).
• Update the scheduler’s partitioned-asset DAG-run creation logic to enforce rollup completeness when a mapper supports it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
task-sdk/src/airflow/sdk/definitions/partition_mappers/base.py	Introduces SDK-side RollupMapper abstraction.
task-sdk/src/airflow/sdk/definitions/partition_mappers/temporal.py	Adds SDK to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/timetables/base.py	Adds get_partition_mapper() hook to the Timetable protocol.
airflow-core/src/airflow/partition_mappers/base.py	Introduces core-side RollupMapper abstraction.
airflow-core/src/airflow/partition_mappers/temporal.py	Adds core to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Uses rollup mapper behavior to decide when partitioned asset-triggered DAG runs are ready.

[Lee-W]

from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
CronPartitionTimetable,
PartitionedAssetTimetable,
WeeklyRollupMapper,
task,
)
daily_sales=Asset(uri="file://incoming/sales/daily.csv", name="daily_sales")
# Upstream Dag: produces one partition per day (key format: "2024-01-15T00:00:00")withDAG(
dag_id="ingest_daily_sales",
schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
):
@task(outlets=[daily_sales])defingest():
passingest()
# Downstream Dag: runs once all 7 daily partitions for a week have arrivedwithDAG(
dag_id="weekly_sales_report",
schedule=PartitionedAssetTimetable(
assets=daily_sales,
default_partition_mapper=WeeklyRollupMapper(),
),
catchup=False,
):
@taskdefgenerate_report(dag_run=None):
# dag_run.partition_key will be the week key, e.g. "2024-01-15 (W03)"print(dag_run.partition_key)
generate_report()

[Lee-W]

The backend part is basically wrapped up, but the frontend and API side need some work. The UI is quite weired for these cases now

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/partitioned_dag_runs.py:1

• Counting PartitionedAssetKeyLog.id can over-count when duplicate log rows exist for the same upstream partition key (e.g. retries/dup inserts), inflating total_received and potentially showing the run as satisfiable earlier than it should be. Consider counting distinct PartitionedAssetKeyLog.source_partition_key (and/or a distinct composite of (asset_id, source_partition_key)) to match the scheduler’s set-based satisfaction semantics.

# Licensed to the Apache Software Foundation (ASF) under one

[dstandish]

Hey @Lee-W

I was a little surprised to see so many lines for this change so I asked Claude to help me review whether there's any unnecessary complexity. Below are its findings. They seem plausible. What do you think?

---

The core design is clean (the Window types are tiny generators,
RollupMapper.to_upstream is a clear decode→expand→encode, and the scheduler gate comes down to one
expected.issubset(actual) check), and the test coverage on the scheduler paths is excellent. The
HA hardening in _create_dagruns_for_partitioned_asset_dags (bulk-fetch over the old N+1,
with_row_locks(skip_locked=True), the per-tick cap and deterministic order_by) all looks right.

One theme I'd like to resolve before merge: a meaningful chunk of complexity is self-induced,
concentrated in partition_mappers/temporal.py.

_compile_output_format_regex looks like more generality than the feature needs. Only two mappers
can't round-trip through strptime: StartOfWeekMapper (%V isn't strptime-parseable) and
StartOfQuarterMapper ({quarter} isn't a directive). The actual requirement is "recover Y/m/d" and
"recover Y/quarter". But the decode that motivated the whole compiler ends up discarding the token
it was built for:

StartOfWeekMapper.decode_downstream — %V is captured but never read

return datetime(int(match["Y"]), int(match["m"]), int(match["d"]))

Both decodes collapse to a dedicated per-mapper regex:

_WEEK_RE = re.compile(r"(?P\d{4})-(?P\d{2})-(?P\d{2})")
_QUARTER_RE = re.compile(r"(?P\d{4})-Q(?P[1-4])")

That drops the directive table, the {name} placeholder machinery, the placeholder_patterns arg,
and the five compile-time ValueError branches — plus the test class that only exists to exercise
that invented surface (test_rejects_adjacent_default_pattern_placeholders,
test_adjacent_placeholders_allowed_when_one_is_narrowed,
test_separator_between_default_placeholders_is_allowed). Roughly ~120 lines + tests, with
identical behavior for every shipped mapper. Could we go with the dedicated regexes for now and
add generality if/when a custom mapper actually needs it?

Follow-on: if the temporal decode simplifies, the base-class guard scaffolding in
partition_mappers/base.py — the init_subclass XOR check, expected_decoded_type, and the
runtime pairing check in RollupMapper.init — largely loses its purpose for the shipped
mappers, since it mostly exists to make the general decode/encode pair safe. Worth reassessing
whether it belongs now or arrives with the first real custom mapper.

Question (non-blocking): the audit-log path (_record_partition_audit_log /
_record_stale_apdr_audit_log) opens independent scoped=False sessions to survive an outer
rollback, plus a process-lifetime dedup set. For advisory Log rows where self.log.exception(...)
already records the same thing, is the rollback-survival guarantee worth the machinery for v1, or
would a plain log line do?

[Lee-W]

Hey @dstandish , thanks for reviewing. a few responses to the questions

Could we go with the dedicated regexes for now and add generality if/when a custom mapper actually needs it?

The regex compiler was added per @uranusjr's review: without it, a custom output_format forces the user to also override decode_downstream, which TP found surprising. Hardcoding _WEEK_RE / _QUARTER_RE saves the ~120 lines but is not that users friendly

is the rollback-survival guarantee worth the machinery for v1, or would a plain log line do?

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

[uranusjr]

I still find it somewhat awkward the UI API endpoints need to load the serialized dag to calculate data. But I guess it’s not too useful to overthink this unless there are known performance issues.

Should we add some links to Starlette for the format-parsing logic?

[dstandish]

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

I don't really have a strong feeling about that. I tried to look for it in the code but could not find it.

And I don't stand in the way of merging this thing....

But... I can't help feeling that this PR is bigger than it should be. It's feels so big that it's kind of overwhelming to properly review. It spreads the reviewer's attention over a very wide area.

[Lee-W]

Should we add some links to Starlette for the format-parsing logic?

yep, added a pointer to Starlette's compile_path in the _compile_output_format_regex docstring.

[potiuk]

@uranusjr's approval here is from Jun 2 (aa9f248), but there have been ~9 commits since — scheduler refactors, the partition-Dag-run clearing fix, mapper narrowing, etc. Worth a re-review so the green check reflects the current state before this merges. @uranusjr, could you re-confirm when you get a chance?

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[Lee-W]

Why

Closes: #59294

Why

Asset-partitioned Dags that aggregate many upstream slices into one downstream period (e.g., 60-minute-level events rolling up into one hourly Dag run) had no way to express that requirement — the scheduler would fire the downstream run as soon as any single upstream partition arrived.

This PR implements the rollup building block from AIP-76: a Window type that enumerates the full set of upstream partitions required for a downstream period, a RollupMapper that wires a source mapper to a window, and the scheduler logic to gate Dag runs until every required upstream key is present.

What

Partition mappers/windows

• Add Window ABC and six concrete implementations (HourWindow, DayWindow, WeekWindow, MonthWindow, QuarterWindow, YearWindow) to both airflow-core and the Task SDK
• Add RollupMapper that composes a source_mapper with a Window and exposes to_upstream(downstream_key) → frozenset[str]
• Add decode_downstream / encode_upstream hooks to PartitionMapper and implement them in _BaseTemporalMapper; StartOfWeekMapper gets a regex-based override because %V is ambiguous with strptime.
• Add week_start parameter to StartOfWeekMapper for non-Monday week starts

Scheduler

• Rewrite _create_dagruns_for_partitioned_asset_dags to bulk-fetch serialized Dags and partition-key logs, removing N+1 queries, and cap per-tick work at MAX_PARTITION_DAG_RUNS_PER_TICK
• Add _resolve_asset_partition_status / _check_rollup_asset_status to evaluate rollup satisfaction; non-rollup assets continue to satisfy immediately

Serialization

• Add encode_window / decode_window and extend mapper encoder/decoder to round-trip RollupMapper and all Window subclasses

UI / API

• Enrich next_run_assets endpoint with per-asset received_count, required_count, received_keys, required_keys, and is_rollup for partitioned Dags
• Update AssetNode and AssetSchedule components to surface rollup progress (e.g. "12 / 24 received")
• Add AssetProgressCell for inline progress in the Dags list

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

withDAG(
dag_id="daily_team_a_rollup",
schedule=PartitionedAssetTimetable(
assets=team_a_player_stats,
default_partition_mapper=RollupMapper(
source_mapper=StartOfDayMapper(),
window=DayWindow(),
),
),
catchup=False,
tags=["player-stats", "rollup"],
):
""" First rollup level: 24 hourly partitions of ``team_a_player_stats`` → one daily summary. ``StartOfDayMapper`` normalizes each upstream hourly timestamp (``%Y-%m-%dT%H:%M:%S``) to its day-start (``%Y-%m-%d``); ``DayWindow`` declares the downstream run needs all 24 hourly partitions before firing. Publishes ``daily_team_a`` so the monthly rollup below can consume it. """@task(outlets=[daily_team_a])defsummarise_team_a_day(dag_run=None):
"""Produce the full-day rollup once every hour has arrived."""ifTYPE_CHECKING:
assertdag_runprint(f"All 24 hourly partitions received. Day: {dag_run.partition_key}")
summarise_team_a_day()

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Copilot]

Pull request overview

Implements “rollup” support for partition mappers (AIP-76) by introducing a RollupMapper interface with to_upstream() and using it in the scheduler to wait for a complete set of upstream partition keys before creating partitioned asset-triggered DAG runs.

Changes:

• Add RollupMapper base class (core + task SDK) with an abstract to_upstream() contract.
• Implement to_upstream() for weekly and monthly temporal mappers (core + task SDK).
• Update the scheduler’s partitioned-asset DAG-run creation logic to enforce rollup completeness when a mapper supports it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
task-sdk/src/airflow/sdk/definitions/partition_mappers/base.py	Introduces SDK-side RollupMapper abstraction.
task-sdk/src/airflow/sdk/definitions/partition_mappers/temporal.py	Adds SDK to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/timetables/base.py	Adds get_partition_mapper() hook to the Timetable protocol.
airflow-core/src/airflow/partition_mappers/base.py	Introduces core-side RollupMapper abstraction.
airflow-core/src/airflow/partition_mappers/temporal.py	Adds core to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Uses rollup mapper behavior to decide when partitioned asset-triggered DAG runs are ready.

[Lee-W]

from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
CronPartitionTimetable,
PartitionedAssetTimetable,
WeeklyRollupMapper,
task,
)
daily_sales=Asset(uri="file://incoming/sales/daily.csv", name="daily_sales")
# Upstream Dag: produces one partition per day (key format: "2024-01-15T00:00:00")withDAG(
dag_id="ingest_daily_sales",
schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
):
@task(outlets=[daily_sales])defingest():
passingest()
# Downstream Dag: runs once all 7 daily partitions for a week have arrivedwithDAG(
dag_id="weekly_sales_report",
schedule=PartitionedAssetTimetable(
assets=daily_sales,
default_partition_mapper=WeeklyRollupMapper(),
),
catchup=False,
):
@taskdefgenerate_report(dag_run=None):
# dag_run.partition_key will be the week key, e.g. "2024-01-15 (W03)"print(dag_run.partition_key)
generate_report()

[Lee-W]

The backend part is basically wrapped up, but the frontend and API side need some work. The UI is quite weired for these cases now

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/partitioned_dag_runs.py:1

• Counting PartitionedAssetKeyLog.id can over-count when duplicate log rows exist for the same upstream partition key (e.g. retries/dup inserts), inflating total_received and potentially showing the run as satisfiable earlier than it should be. Consider counting distinct PartitionedAssetKeyLog.source_partition_key (and/or a distinct composite of (asset_id, source_partition_key)) to match the scheduler’s set-based satisfaction semantics.

# Licensed to the Apache Software Foundation (ASF) under one

[dstandish]

Hey @Lee-W

I was a little surprised to see so many lines for this change so I asked Claude to help me review whether there's any unnecessary complexity. Below are its findings. They seem plausible. What do you think?

---

The core design is clean (the Window types are tiny generators,
RollupMapper.to_upstream is a clear decode→expand→encode, and the scheduler gate comes down to one
expected.issubset(actual) check), and the test coverage on the scheduler paths is excellent. The
HA hardening in _create_dagruns_for_partitioned_asset_dags (bulk-fetch over the old N+1,
with_row_locks(skip_locked=True), the per-tick cap and deterministic order_by) all looks right.

One theme I'd like to resolve before merge: a meaningful chunk of complexity is self-induced,
concentrated in partition_mappers/temporal.py.

_compile_output_format_regex looks like more generality than the feature needs. Only two mappers
can't round-trip through strptime: StartOfWeekMapper (%V isn't strptime-parseable) and
StartOfQuarterMapper ({quarter} isn't a directive). The actual requirement is "recover Y/m/d" and
"recover Y/quarter". But the decode that motivated the whole compiler ends up discarding the token
it was built for:

StartOfWeekMapper.decode_downstream — %V is captured but never read

return datetime(int(match["Y"]), int(match["m"]), int(match["d"]))

Both decodes collapse to a dedicated per-mapper regex:

_WEEK_RE = re.compile(r"(?P\d{4})-(?P\d{2})-(?P\d{2})")
_QUARTER_RE = re.compile(r"(?P\d{4})-Q(?P[1-4])")

That drops the directive table, the {name} placeholder machinery, the placeholder_patterns arg,
and the five compile-time ValueError branches — plus the test class that only exists to exercise
that invented surface (test_rejects_adjacent_default_pattern_placeholders,
test_adjacent_placeholders_allowed_when_one_is_narrowed,
test_separator_between_default_placeholders_is_allowed). Roughly ~120 lines + tests, with
identical behavior for every shipped mapper. Could we go with the dedicated regexes for now and
add generality if/when a custom mapper actually needs it?

Follow-on: if the temporal decode simplifies, the base-class guard scaffolding in
partition_mappers/base.py — the init_subclass XOR check, expected_decoded_type, and the
runtime pairing check in RollupMapper.init — largely loses its purpose for the shipped
mappers, since it mostly exists to make the general decode/encode pair safe. Worth reassessing
whether it belongs now or arrives with the first real custom mapper.

Question (non-blocking): the audit-log path (_record_partition_audit_log /
_record_stale_apdr_audit_log) opens independent scoped=False sessions to survive an outer
rollback, plus a process-lifetime dedup set. For advisory Log rows where self.log.exception(...)
already records the same thing, is the rollback-survival guarantee worth the machinery for v1, or
would a plain log line do?

[Lee-W]

Hey @dstandish , thanks for reviewing. a few responses to the questions

Could we go with the dedicated regexes for now and add generality if/when a custom mapper actually needs it?

The regex compiler was added per @uranusjr's review: without it, a custom output_format forces the user to also override decode_downstream, which TP found surprising. Hardcoding _WEEK_RE / _QUARTER_RE saves the ~120 lines but is not that users friendly

is the rollback-survival guarantee worth the machinery for v1, or would a plain log line do?

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

[uranusjr]

I still find it somewhat awkward the UI API endpoints need to load the serialized dag to calculate data. But I guess it’s not too useful to overthink this unless there are known performance issues.

Should we add some links to Starlette for the format-parsing logic?

[dstandish]

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

I don't really have a strong feeling about that. I tried to look for it in the code but could not find it.

And I don't stand in the way of merging this thing....

But... I can't help feeling that this PR is bigger than it should be. It's feels so big that it's kind of overwhelming to properly review. It spreads the reviewer's attention over a very wide area.

[Lee-W]

Should we add some links to Starlette for the format-parsing logic?

yep, added a pointer to Starlette's compile_path in the _compile_output_format_regex docstring.

[potiuk]

@uranusjr's approval here is from Jun 2 (aa9f248), but there have been ~9 commits since — scheduler refactors, the partition-Dag-run clearing fix, mapper narrowing, etc. Worth a re-review so the green check reflects the current state before this merges. @uranusjr, could you re-confirm when you get a chance?

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[Lee-W]

Why

Closes: #59294

Why

Asset-partitioned Dags that aggregate many upstream slices into one downstream period (e.g., 60-minute-level events rolling up into one hourly Dag run) had no way to express that requirement — the scheduler would fire the downstream run as soon as any single upstream partition arrived.

This PR implements the rollup building block from AIP-76: a Window type that enumerates the full set of upstream partitions required for a downstream period, a RollupMapper that wires a source mapper to a window, and the scheduler logic to gate Dag runs until every required upstream key is present.

What

Partition mappers/windows

• Add Window ABC and six concrete implementations (HourWindow, DayWindow, WeekWindow, MonthWindow, QuarterWindow, YearWindow) to both airflow-core and the Task SDK
• Add RollupMapper that composes a source_mapper with a Window and exposes to_upstream(downstream_key) → frozenset[str]
• Add decode_downstream / encode_upstream hooks to PartitionMapper and implement them in _BaseTemporalMapper; StartOfWeekMapper gets a regex-based override because %V is ambiguous with strptime.
• Add week_start parameter to StartOfWeekMapper for non-Monday week starts

Scheduler

• Rewrite _create_dagruns_for_partitioned_asset_dags to bulk-fetch serialized Dags and partition-key logs, removing N+1 queries, and cap per-tick work at MAX_PARTITION_DAG_RUNS_PER_TICK
• Add _resolve_asset_partition_status / _check_rollup_asset_status to evaluate rollup satisfaction; non-rollup assets continue to satisfy immediately

Serialization

• Add encode_window / decode_window and extend mapper encoder/decoder to round-trip RollupMapper and all Window subclasses

UI / API

• Enrich next_run_assets endpoint with per-asset received_count, required_count, received_keys, required_keys, and is_rollup for partitioned Dags
• Update AssetNode and AssetSchedule components to surface rollup progress (e.g. "12 / 24 received")
• Add AssetProgressCell for inline progress in the Dags list

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

withDAG(
dag_id="daily_team_a_rollup",
schedule=PartitionedAssetTimetable(
assets=team_a_player_stats,
default_partition_mapper=RollupMapper(
source_mapper=StartOfDayMapper(),
window=DayWindow(),
),
),
catchup=False,
tags=["player-stats", "rollup"],
):
""" First rollup level: 24 hourly partitions of ``team_a_player_stats`` → one daily summary. ``StartOfDayMapper`` normalizes each upstream hourly timestamp (``%Y-%m-%dT%H:%M:%S``) to its day-start (``%Y-%m-%d``); ``DayWindow`` declares the downstream run needs all 24 hourly partitions before firing. Publishes ``daily_team_a`` so the monthly rollup below can consume it. """@task(outlets=[daily_team_a])defsummarise_team_a_day(dag_run=None):
"""Produce the full-day rollup once every hour has arrived."""ifTYPE_CHECKING:
assertdag_runprint(f"All 24 hourly partitions received. Day: {dag_run.partition_key}")
summarise_team_a_day()

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Copilot]

Pull request overview

Implements “rollup” support for partition mappers (AIP-76) by introducing a RollupMapper interface with to_upstream() and using it in the scheduler to wait for a complete set of upstream partition keys before creating partitioned asset-triggered DAG runs.

Changes:

• Add RollupMapper base class (core + task SDK) with an abstract to_upstream() contract.
• Implement to_upstream() for weekly and monthly temporal mappers (core + task SDK).
• Update the scheduler’s partitioned-asset DAG-run creation logic to enforce rollup completeness when a mapper supports it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
task-sdk/src/airflow/sdk/definitions/partition_mappers/base.py	Introduces SDK-side RollupMapper abstraction.
task-sdk/src/airflow/sdk/definitions/partition_mappers/temporal.py	Adds SDK to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/timetables/base.py	Adds get_partition_mapper() hook to the Timetable protocol.
airflow-core/src/airflow/partition_mappers/base.py	Introduces core-side RollupMapper abstraction.
airflow-core/src/airflow/partition_mappers/temporal.py	Adds core to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Uses rollup mapper behavior to decide when partitioned asset-triggered DAG runs are ready.

[Lee-W]

from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
CronPartitionTimetable,
PartitionedAssetTimetable,
WeeklyRollupMapper,
task,
)
daily_sales=Asset(uri="file://incoming/sales/daily.csv", name="daily_sales")
# Upstream Dag: produces one partition per day (key format: "2024-01-15T00:00:00")withDAG(
dag_id="ingest_daily_sales",
schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
):
@task(outlets=[daily_sales])defingest():
passingest()
# Downstream Dag: runs once all 7 daily partitions for a week have arrivedwithDAG(
dag_id="weekly_sales_report",
schedule=PartitionedAssetTimetable(
assets=daily_sales,
default_partition_mapper=WeeklyRollupMapper(),
),
catchup=False,
):
@taskdefgenerate_report(dag_run=None):
# dag_run.partition_key will be the week key, e.g. "2024-01-15 (W03)"print(dag_run.partition_key)
generate_report()

[Lee-W]

The backend part is basically wrapped up, but the frontend and API side need some work. The UI is quite weired for these cases now

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/partitioned_dag_runs.py:1

• Counting PartitionedAssetKeyLog.id can over-count when duplicate log rows exist for the same upstream partition key (e.g. retries/dup inserts), inflating total_received and potentially showing the run as satisfiable earlier than it should be. Consider counting distinct PartitionedAssetKeyLog.source_partition_key (and/or a distinct composite of (asset_id, source_partition_key)) to match the scheduler’s set-based satisfaction semantics.

# Licensed to the Apache Software Foundation (ASF) under one

[dstandish]

Hey @Lee-W

I was a little surprised to see so many lines for this change so I asked Claude to help me review whether there's any unnecessary complexity. Below are its findings. They seem plausible. What do you think?

---

The core design is clean (the Window types are tiny generators,
RollupMapper.to_upstream is a clear decode→expand→encode, and the scheduler gate comes down to one
expected.issubset(actual) check), and the test coverage on the scheduler paths is excellent. The
HA hardening in _create_dagruns_for_partitioned_asset_dags (bulk-fetch over the old N+1,
with_row_locks(skip_locked=True), the per-tick cap and deterministic order_by) all looks right.

One theme I'd like to resolve before merge: a meaningful chunk of complexity is self-induced,
concentrated in partition_mappers/temporal.py.

_compile_output_format_regex looks like more generality than the feature needs. Only two mappers
can't round-trip through strptime: StartOfWeekMapper (%V isn't strptime-parseable) and
StartOfQuarterMapper ({quarter} isn't a directive). The actual requirement is "recover Y/m/d" and
"recover Y/quarter". But the decode that motivated the whole compiler ends up discarding the token
it was built for:

StartOfWeekMapper.decode_downstream — %V is captured but never read

return datetime(int(match["Y"]), int(match["m"]), int(match["d"]))

Both decodes collapse to a dedicated per-mapper regex:

_WEEK_RE = re.compile(r"(?P\d{4})-(?P\d{2})-(?P\d{2})")
_QUARTER_RE = re.compile(r"(?P\d{4})-Q(?P[1-4])")

That drops the directive table, the {name} placeholder machinery, the placeholder_patterns arg,
and the five compile-time ValueError branches — plus the test class that only exists to exercise
that invented surface (test_rejects_adjacent_default_pattern_placeholders,
test_adjacent_placeholders_allowed_when_one_is_narrowed,
test_separator_between_default_placeholders_is_allowed). Roughly ~120 lines + tests, with
identical behavior for every shipped mapper. Could we go with the dedicated regexes for now and
add generality if/when a custom mapper actually needs it?

Follow-on: if the temporal decode simplifies, the base-class guard scaffolding in
partition_mappers/base.py — the init_subclass XOR check, expected_decoded_type, and the
runtime pairing check in RollupMapper.init — largely loses its purpose for the shipped
mappers, since it mostly exists to make the general decode/encode pair safe. Worth reassessing
whether it belongs now or arrives with the first real custom mapper.

Question (non-blocking): the audit-log path (_record_partition_audit_log /
_record_stale_apdr_audit_log) opens independent scoped=False sessions to survive an outer
rollback, plus a process-lifetime dedup set. For advisory Log rows where self.log.exception(...)
already records the same thing, is the rollback-survival guarantee worth the machinery for v1, or
would a plain log line do?

[Lee-W]

Hey @dstandish , thanks for reviewing. a few responses to the questions

Could we go with the dedicated regexes for now and add generality if/when a custom mapper actually needs it?

The regex compiler was added per @uranusjr's review: without it, a custom output_format forces the user to also override decode_downstream, which TP found surprising. Hardcoding _WEEK_RE / _QUARTER_RE saves the ~120 lines but is not that users friendly

is the rollback-survival guarantee worth the machinery for v1, or would a plain log line do?

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

[uranusjr]

I still find it somewhat awkward the UI API endpoints need to load the serialized dag to calculate data. But I guess it’s not too useful to overthink this unless there are known performance issues.

Should we add some links to Starlette for the format-parsing logic?

[dstandish]

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

I don't really have a strong feeling about that. I tried to look for it in the code but could not find it.

And I don't stand in the way of merging this thing....

But... I can't help feeling that this PR is bigger than it should be. It's feels so big that it's kind of overwhelming to properly review. It spreads the reviewer's attention over a very wide area.

[Lee-W]

Should we add some links to Starlette for the format-parsing logic?

yep, added a pointer to Starlette's compile_path in the _compile_output_format_regex docstring.

[potiuk]

@uranusjr's approval here is from Jun 2 (aa9f248), but there have been ~9 commits since — scheduler refactors, the partition-Dag-run clearing fix, mapper narrowing, etc. Worth a re-review so the green check reflects the current state before this merges. @uranusjr, could you re-confirm when you get a chance?

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[Lee-W]

Why

Closes: #59294

Why

Asset-partitioned Dags that aggregate many upstream slices into one downstream period (e.g., 60-minute-level events rolling up into one hourly Dag run) had no way to express that requirement — the scheduler would fire the downstream run as soon as any single upstream partition arrived.

This PR implements the rollup building block from AIP-76: a Window type that enumerates the full set of upstream partitions required for a downstream period, a RollupMapper that wires a source mapper to a window, and the scheduler logic to gate Dag runs until every required upstream key is present.

What

Partition mappers/windows

• Add Window ABC and six concrete implementations (HourWindow, DayWindow, WeekWindow, MonthWindow, QuarterWindow, YearWindow) to both airflow-core and the Task SDK
• Add RollupMapper that composes a source_mapper with a Window and exposes to_upstream(downstream_key) → frozenset[str]
• Add decode_downstream / encode_upstream hooks to PartitionMapper and implement them in _BaseTemporalMapper; StartOfWeekMapper gets a regex-based override because %V is ambiguous with strptime.
• Add week_start parameter to StartOfWeekMapper for non-Monday week starts

Scheduler

• Rewrite _create_dagruns_for_partitioned_asset_dags to bulk-fetch serialized Dags and partition-key logs, removing N+1 queries, and cap per-tick work at MAX_PARTITION_DAG_RUNS_PER_TICK
• Add _resolve_asset_partition_status / _check_rollup_asset_status to evaluate rollup satisfaction; non-rollup assets continue to satisfy immediately

Serialization

• Add encode_window / decode_window and extend mapper encoder/decoder to round-trip RollupMapper and all Window subclasses

UI / API

• Enrich next_run_assets endpoint with per-asset received_count, required_count, received_keys, required_keys, and is_rollup for partitioned Dags
• Update AssetNode and AssetSchedule components to surface rollup progress (e.g. "12 / 24 received")
• Add AssetProgressCell for inline progress in the Dags list

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

withDAG(
dag_id="daily_team_a_rollup",
schedule=PartitionedAssetTimetable(
assets=team_a_player_stats,
default_partition_mapper=RollupMapper(
source_mapper=StartOfDayMapper(),
window=DayWindow(),
),
),
catchup=False,
tags=["player-stats", "rollup"],
):
""" First rollup level: 24 hourly partitions of ``team_a_player_stats`` → one daily summary. ``StartOfDayMapper`` normalizes each upstream hourly timestamp (``%Y-%m-%dT%H:%M:%S``) to its day-start (``%Y-%m-%d``); ``DayWindow`` declares the downstream run needs all 24 hourly partitions before firing. Publishes ``daily_team_a`` so the monthly rollup below can consume it. """@task(outlets=[daily_team_a])defsummarise_team_a_day(dag_run=None):
"""Produce the full-day rollup once every hour has arrived."""ifTYPE_CHECKING:
assertdag_runprint(f"All 24 hourly partitions received. Day: {dag_run.partition_key}")
summarise_team_a_day()

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Copilot]

Pull request overview

Implements “rollup” support for partition mappers (AIP-76) by introducing a RollupMapper interface with to_upstream() and using it in the scheduler to wait for a complete set of upstream partition keys before creating partitioned asset-triggered DAG runs.

Changes:

• Add RollupMapper base class (core + task SDK) with an abstract to_upstream() contract.
• Implement to_upstream() for weekly and monthly temporal mappers (core + task SDK).
• Update the scheduler’s partitioned-asset DAG-run creation logic to enforce rollup completeness when a mapper supports it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
task-sdk/src/airflow/sdk/definitions/partition_mappers/base.py	Introduces SDK-side RollupMapper abstraction.
task-sdk/src/airflow/sdk/definitions/partition_mappers/temporal.py	Adds SDK to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/timetables/base.py	Adds get_partition_mapper() hook to the Timetable protocol.
airflow-core/src/airflow/partition_mappers/base.py	Introduces core-side RollupMapper abstraction.
airflow-core/src/airflow/partition_mappers/temporal.py	Adds core to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Uses rollup mapper behavior to decide when partitioned asset-triggered DAG runs are ready.

[Lee-W]

from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
CronPartitionTimetable,
PartitionedAssetTimetable,
WeeklyRollupMapper,
task,
)
daily_sales=Asset(uri="file://incoming/sales/daily.csv", name="daily_sales")
# Upstream Dag: produces one partition per day (key format: "2024-01-15T00:00:00")withDAG(
dag_id="ingest_daily_sales",
schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
):
@task(outlets=[daily_sales])defingest():
passingest()
# Downstream Dag: runs once all 7 daily partitions for a week have arrivedwithDAG(
dag_id="weekly_sales_report",
schedule=PartitionedAssetTimetable(
assets=daily_sales,
default_partition_mapper=WeeklyRollupMapper(),
),
catchup=False,
):
@taskdefgenerate_report(dag_run=None):
# dag_run.partition_key will be the week key, e.g. "2024-01-15 (W03)"print(dag_run.partition_key)
generate_report()

[Lee-W]

The backend part is basically wrapped up, but the frontend and API side need some work. The UI is quite weired for these cases now

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/partitioned_dag_runs.py:1

• Counting PartitionedAssetKeyLog.id can over-count when duplicate log rows exist for the same upstream partition key (e.g. retries/dup inserts), inflating total_received and potentially showing the run as satisfiable earlier than it should be. Consider counting distinct PartitionedAssetKeyLog.source_partition_key (and/or a distinct composite of (asset_id, source_partition_key)) to match the scheduler’s set-based satisfaction semantics.

# Licensed to the Apache Software Foundation (ASF) under one

[dstandish]

Hey @Lee-W

I was a little surprised to see so many lines for this change so I asked Claude to help me review whether there's any unnecessary complexity. Below are its findings. They seem plausible. What do you think?

---

The core design is clean (the Window types are tiny generators,
RollupMapper.to_upstream is a clear decode→expand→encode, and the scheduler gate comes down to one
expected.issubset(actual) check), and the test coverage on the scheduler paths is excellent. The
HA hardening in _create_dagruns_for_partitioned_asset_dags (bulk-fetch over the old N+1,
with_row_locks(skip_locked=True), the per-tick cap and deterministic order_by) all looks right.

One theme I'd like to resolve before merge: a meaningful chunk of complexity is self-induced,
concentrated in partition_mappers/temporal.py.

_compile_output_format_regex looks like more generality than the feature needs. Only two mappers
can't round-trip through strptime: StartOfWeekMapper (%V isn't strptime-parseable) and
StartOfQuarterMapper ({quarter} isn't a directive). The actual requirement is "recover Y/m/d" and
"recover Y/quarter". But the decode that motivated the whole compiler ends up discarding the token
it was built for:

StartOfWeekMapper.decode_downstream — %V is captured but never read

return datetime(int(match["Y"]), int(match["m"]), int(match["d"]))

Both decodes collapse to a dedicated per-mapper regex:

_WEEK_RE = re.compile(r"(?P\d{4})-(?P\d{2})-(?P\d{2})")
_QUARTER_RE = re.compile(r"(?P\d{4})-Q(?P[1-4])")

That drops the directive table, the {name} placeholder machinery, the placeholder_patterns arg,
and the five compile-time ValueError branches — plus the test class that only exists to exercise
that invented surface (test_rejects_adjacent_default_pattern_placeholders,
test_adjacent_placeholders_allowed_when_one_is_narrowed,
test_separator_between_default_placeholders_is_allowed). Roughly ~120 lines + tests, with
identical behavior for every shipped mapper. Could we go with the dedicated regexes for now and
add generality if/when a custom mapper actually needs it?

Follow-on: if the temporal decode simplifies, the base-class guard scaffolding in
partition_mappers/base.py — the init_subclass XOR check, expected_decoded_type, and the
runtime pairing check in RollupMapper.init — largely loses its purpose for the shipped
mappers, since it mostly exists to make the general decode/encode pair safe. Worth reassessing
whether it belongs now or arrives with the first real custom mapper.

Question (non-blocking): the audit-log path (_record_partition_audit_log /
_record_stale_apdr_audit_log) opens independent scoped=False sessions to survive an outer
rollback, plus a process-lifetime dedup set. For advisory Log rows where self.log.exception(...)
already records the same thing, is the rollback-survival guarantee worth the machinery for v1, or
would a plain log line do?

[Lee-W]

Hey @dstandish , thanks for reviewing. a few responses to the questions

Could we go with the dedicated regexes for now and add generality if/when a custom mapper actually needs it?

The regex compiler was added per @uranusjr's review: without it, a custom output_format forces the user to also override decode_downstream, which TP found surprising. Hardcoding _WEEK_RE / _QUARTER_RE saves the ~120 lines but is not that users friendly

is the rollback-survival guarantee worth the machinery for v1, or would a plain log line do?

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

[uranusjr]

I still find it somewhat awkward the UI API endpoints need to load the serialized dag to calculate data. But I guess it’s not too useful to overthink this unless there are known performance issues.

Should we add some links to Starlette for the format-parsing logic?

[dstandish]

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

I don't really have a strong feeling about that. I tried to look for it in the code but could not find it.

And I don't stand in the way of merging this thing....

But... I can't help feeling that this PR is bigger than it should be. It's feels so big that it's kind of overwhelming to properly review. It spreads the reviewer's attention over a very wide area.

[Lee-W]

Should we add some links to Starlette for the format-parsing logic?

yep, added a pointer to Starlette's compile_path in the _compile_output_format_regex docstring.

[potiuk]

@uranusjr's approval here is from Jun 2 (aa9f248), but there have been ~9 commits since — scheduler refactors, the partition-Dag-run clearing fix, mapper narrowing, etc. Worth a re-review so the green check reflects the current state before this merges. @uranusjr, could you re-confirm when you get a chance?

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[Lee-W]

Why

Closes: #59294

Why

Asset-partitioned Dags that aggregate many upstream slices into one downstream period (e.g., 60-minute-level events rolling up into one hourly Dag run) had no way to express that requirement — the scheduler would fire the downstream run as soon as any single upstream partition arrived.

This PR implements the rollup building block from AIP-76: a Window type that enumerates the full set of upstream partitions required for a downstream period, a RollupMapper that wires a source mapper to a window, and the scheduler logic to gate Dag runs until every required upstream key is present.

What

Partition mappers/windows

• Add Window ABC and six concrete implementations (HourWindow, DayWindow, WeekWindow, MonthWindow, QuarterWindow, YearWindow) to both airflow-core and the Task SDK
• Add RollupMapper that composes a source_mapper with a Window and exposes to_upstream(downstream_key) → frozenset[str]
• Add decode_downstream / encode_upstream hooks to PartitionMapper and implement them in _BaseTemporalMapper; StartOfWeekMapper gets a regex-based override because %V is ambiguous with strptime.
• Add week_start parameter to StartOfWeekMapper for non-Monday week starts

Scheduler

• Rewrite _create_dagruns_for_partitioned_asset_dags to bulk-fetch serialized Dags and partition-key logs, removing N+1 queries, and cap per-tick work at MAX_PARTITION_DAG_RUNS_PER_TICK
• Add _resolve_asset_partition_status / _check_rollup_asset_status to evaluate rollup satisfaction; non-rollup assets continue to satisfy immediately

Serialization

• Add encode_window / decode_window and extend mapper encoder/decoder to round-trip RollupMapper and all Window subclasses

UI / API

• Enrich next_run_assets endpoint with per-asset received_count, required_count, received_keys, required_keys, and is_rollup for partitioned Dags
• Update AssetNode and AssetSchedule components to surface rollup progress (e.g. "12 / 24 received")
• Add AssetProgressCell for inline progress in the Dags list

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

withDAG(
dag_id="daily_team_a_rollup",
schedule=PartitionedAssetTimetable(
assets=team_a_player_stats,
default_partition_mapper=RollupMapper(
source_mapper=StartOfDayMapper(),
window=DayWindow(),
),
),
catchup=False,
tags=["player-stats", "rollup"],
):
""" First rollup level: 24 hourly partitions of ``team_a_player_stats`` → one daily summary. ``StartOfDayMapper`` normalizes each upstream hourly timestamp (``%Y-%m-%dT%H:%M:%S``) to its day-start (``%Y-%m-%d``); ``DayWindow`` declares the downstream run needs all 24 hourly partitions before firing. Publishes ``daily_team_a`` so the monthly rollup below can consume it. """@task(outlets=[daily_team_a])defsummarise_team_a_day(dag_run=None):
"""Produce the full-day rollup once every hour has arrived."""ifTYPE_CHECKING:
assertdag_runprint(f"All 24 hourly partitions received. Day: {dag_run.partition_key}")
summarise_team_a_day()

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Copilot]

Pull request overview

Implements “rollup” support for partition mappers (AIP-76) by introducing a RollupMapper interface with to_upstream() and using it in the scheduler to wait for a complete set of upstream partition keys before creating partitioned asset-triggered DAG runs.

Changes:

• Add RollupMapper base class (core + task SDK) with an abstract to_upstream() contract.
• Implement to_upstream() for weekly and monthly temporal mappers (core + task SDK).
• Update the scheduler’s partitioned-asset DAG-run creation logic to enforce rollup completeness when a mapper supports it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
task-sdk/src/airflow/sdk/definitions/partition_mappers/base.py	Introduces SDK-side RollupMapper abstraction.
task-sdk/src/airflow/sdk/definitions/partition_mappers/temporal.py	Adds SDK to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/timetables/base.py	Adds get_partition_mapper() hook to the Timetable protocol.
airflow-core/src/airflow/partition_mappers/base.py	Introduces core-side RollupMapper abstraction.
airflow-core/src/airflow/partition_mappers/temporal.py	Adds core to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Uses rollup mapper behavior to decide when partitioned asset-triggered DAG runs are ready.

[Lee-W]

from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
CronPartitionTimetable,
PartitionedAssetTimetable,
WeeklyRollupMapper,
task,
)
daily_sales=Asset(uri="file://incoming/sales/daily.csv", name="daily_sales")
# Upstream Dag: produces one partition per day (key format: "2024-01-15T00:00:00")withDAG(
dag_id="ingest_daily_sales",
schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
):
@task(outlets=[daily_sales])defingest():
passingest()
# Downstream Dag: runs once all 7 daily partitions for a week have arrivedwithDAG(
dag_id="weekly_sales_report",
schedule=PartitionedAssetTimetable(
assets=daily_sales,
default_partition_mapper=WeeklyRollupMapper(),
),
catchup=False,
):
@taskdefgenerate_report(dag_run=None):
# dag_run.partition_key will be the week key, e.g. "2024-01-15 (W03)"print(dag_run.partition_key)
generate_report()

[Lee-W]

The backend part is basically wrapped up, but the frontend and API side need some work. The UI is quite weired for these cases now

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/partitioned_dag_runs.py:1

• Counting PartitionedAssetKeyLog.id can over-count when duplicate log rows exist for the same upstream partition key (e.g. retries/dup inserts), inflating total_received and potentially showing the run as satisfiable earlier than it should be. Consider counting distinct PartitionedAssetKeyLog.source_partition_key (and/or a distinct composite of (asset_id, source_partition_key)) to match the scheduler’s set-based satisfaction semantics.

# Licensed to the Apache Software Foundation (ASF) under one

[dstandish]

Hey @Lee-W

I was a little surprised to see so many lines for this change so I asked Claude to help me review whether there's any unnecessary complexity. Below are its findings. They seem plausible. What do you think?

---

The core design is clean (the Window types are tiny generators,
RollupMapper.to_upstream is a clear decode→expand→encode, and the scheduler gate comes down to one
expected.issubset(actual) check), and the test coverage on the scheduler paths is excellent. The
HA hardening in _create_dagruns_for_partitioned_asset_dags (bulk-fetch over the old N+1,
with_row_locks(skip_locked=True), the per-tick cap and deterministic order_by) all looks right.

One theme I'd like to resolve before merge: a meaningful chunk of complexity is self-induced,
concentrated in partition_mappers/temporal.py.

_compile_output_format_regex looks like more generality than the feature needs. Only two mappers
can't round-trip through strptime: StartOfWeekMapper (%V isn't strptime-parseable) and
StartOfQuarterMapper ({quarter} isn't a directive). The actual requirement is "recover Y/m/d" and
"recover Y/quarter". But the decode that motivated the whole compiler ends up discarding the token
it was built for:

StartOfWeekMapper.decode_downstream — %V is captured but never read

return datetime(int(match["Y"]), int(match["m"]), int(match["d"]))

Both decodes collapse to a dedicated per-mapper regex:

_WEEK_RE = re.compile(r"(?P\d{4})-(?P\d{2})-(?P\d{2})")
_QUARTER_RE = re.compile(r"(?P\d{4})-Q(?P[1-4])")

That drops the directive table, the {name} placeholder machinery, the placeholder_patterns arg,
and the five compile-time ValueError branches — plus the test class that only exists to exercise
that invented surface (test_rejects_adjacent_default_pattern_placeholders,
test_adjacent_placeholders_allowed_when_one_is_narrowed,
test_separator_between_default_placeholders_is_allowed). Roughly ~120 lines + tests, with
identical behavior for every shipped mapper. Could we go with the dedicated regexes for now and
add generality if/when a custom mapper actually needs it?

Follow-on: if the temporal decode simplifies, the base-class guard scaffolding in
partition_mappers/base.py — the init_subclass XOR check, expected_decoded_type, and the
runtime pairing check in RollupMapper.init — largely loses its purpose for the shipped
mappers, since it mostly exists to make the general decode/encode pair safe. Worth reassessing
whether it belongs now or arrives with the first real custom mapper.

Question (non-blocking): the audit-log path (_record_partition_audit_log /
_record_stale_apdr_audit_log) opens independent scoped=False sessions to survive an outer
rollback, plus a process-lifetime dedup set. For advisory Log rows where self.log.exception(...)
already records the same thing, is the rollback-survival guarantee worth the machinery for v1, or
would a plain log line do?

[Lee-W]

Hey @dstandish , thanks for reviewing. a few responses to the questions

Could we go with the dedicated regexes for now and add generality if/when a custom mapper actually needs it?

The regex compiler was added per @uranusjr's review: without it, a custom output_format forces the user to also override decode_downstream, which TP found surprising. Hardcoding _WEEK_RE / _QUARTER_RE saves the ~120 lines but is not that users friendly

is the rollback-survival guarantee worth the machinery for v1, or would a plain log line do?

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

[uranusjr]

I still find it somewhat awkward the UI API endpoints need to load the serialized dag to calculate data. But I guess it’s not too useful to overthink this unless there are known performance issues.

Should we add some links to Starlette for the format-parsing logic?

[dstandish]

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

I don't really have a strong feeling about that. I tried to look for it in the code but could not find it.

And I don't stand in the way of merging this thing....

But... I can't help feeling that this PR is bigger than it should be. It's feels so big that it's kind of overwhelming to properly review. It spreads the reviewer's attention over a very wide area.

[Lee-W]

Should we add some links to Starlette for the format-parsing logic?

yep, added a pointer to Starlette's compile_path in the _compile_output_format_regex docstring.

[potiuk]

@uranusjr's approval here is from Jun 2 (aa9f248), but there have been ~9 commits since — scheduler refactors, the partition-Dag-run clearing fix, mapper narrowing, etc. Worth a re-review so the green check reflects the current state before this merges. @uranusjr, could you re-confirm when you get a chance?

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[Lee-W]

Why

Closes: #59294

Why

Asset-partitioned Dags that aggregate many upstream slices into one downstream period (e.g., 60-minute-level events rolling up into one hourly Dag run) had no way to express that requirement — the scheduler would fire the downstream run as soon as any single upstream partition arrived.

This PR implements the rollup building block from AIP-76: a Window type that enumerates the full set of upstream partitions required for a downstream period, a RollupMapper that wires a source mapper to a window, and the scheduler logic to gate Dag runs until every required upstream key is present.

What

Partition mappers/windows

• Add Window ABC and six concrete implementations (HourWindow, DayWindow, WeekWindow, MonthWindow, QuarterWindow, YearWindow) to both airflow-core and the Task SDK
• Add RollupMapper that composes a source_mapper with a Window and exposes to_upstream(downstream_key) → frozenset[str]
• Add decode_downstream / encode_upstream hooks to PartitionMapper and implement them in _BaseTemporalMapper; StartOfWeekMapper gets a regex-based override because %V is ambiguous with strptime.
• Add week_start parameter to StartOfWeekMapper for non-Monday week starts

Scheduler

• Rewrite _create_dagruns_for_partitioned_asset_dags to bulk-fetch serialized Dags and partition-key logs, removing N+1 queries, and cap per-tick work at MAX_PARTITION_DAG_RUNS_PER_TICK
• Add _resolve_asset_partition_status / _check_rollup_asset_status to evaluate rollup satisfaction; non-rollup assets continue to satisfy immediately

Serialization

• Add encode_window / decode_window and extend mapper encoder/decoder to round-trip RollupMapper and all Window subclasses

UI / API

• Enrich next_run_assets endpoint with per-asset received_count, required_count, received_keys, required_keys, and is_rollup for partitioned Dags
• Update AssetNode and AssetSchedule components to surface rollup progress (e.g. "12 / 24 received")
• Add AssetProgressCell for inline progress in the Dags list

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

withDAG(
dag_id="daily_team_a_rollup",
schedule=PartitionedAssetTimetable(
assets=team_a_player_stats,
default_partition_mapper=RollupMapper(
source_mapper=StartOfDayMapper(),
window=DayWindow(),
),
),
catchup=False,
tags=["player-stats", "rollup"],
):
""" First rollup level: 24 hourly partitions of ``team_a_player_stats`` → one daily summary. ``StartOfDayMapper`` normalizes each upstream hourly timestamp (``%Y-%m-%dT%H:%M:%S``) to its day-start (``%Y-%m-%d``); ``DayWindow`` declares the downstream run needs all 24 hourly partitions before firing. Publishes ``daily_team_a`` so the monthly rollup below can consume it. """@task(outlets=[daily_team_a])defsummarise_team_a_day(dag_run=None):
"""Produce the full-day rollup once every hour has arrived."""ifTYPE_CHECKING:
assertdag_runprint(f"All 24 hourly partitions received. Day: {dag_run.partition_key}")
summarise_team_a_day()

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Copilot]

Pull request overview

Implements “rollup” support for partition mappers (AIP-76) by introducing a RollupMapper interface with to_upstream() and using it in the scheduler to wait for a complete set of upstream partition keys before creating partitioned asset-triggered DAG runs.

Changes:

• Add RollupMapper base class (core + task SDK) with an abstract to_upstream() contract.
• Implement to_upstream() for weekly and monthly temporal mappers (core + task SDK).
• Update the scheduler’s partitioned-asset DAG-run creation logic to enforce rollup completeness when a mapper supports it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
task-sdk/src/airflow/sdk/definitions/partition_mappers/base.py	Introduces SDK-side RollupMapper abstraction.
task-sdk/src/airflow/sdk/definitions/partition_mappers/temporal.py	Adds SDK to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/timetables/base.py	Adds get_partition_mapper() hook to the Timetable protocol.
airflow-core/src/airflow/partition_mappers/base.py	Introduces core-side RollupMapper abstraction.
airflow-core/src/airflow/partition_mappers/temporal.py	Adds core to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Uses rollup mapper behavior to decide when partitioned asset-triggered DAG runs are ready.

[Lee-W]

from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
CronPartitionTimetable,
PartitionedAssetTimetable,
WeeklyRollupMapper,
task,
)
daily_sales=Asset(uri="file://incoming/sales/daily.csv", name="daily_sales")
# Upstream Dag: produces one partition per day (key format: "2024-01-15T00:00:00")withDAG(
dag_id="ingest_daily_sales",
schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
):
@task(outlets=[daily_sales])defingest():
passingest()
# Downstream Dag: runs once all 7 daily partitions for a week have arrivedwithDAG(
dag_id="weekly_sales_report",
schedule=PartitionedAssetTimetable(
assets=daily_sales,
default_partition_mapper=WeeklyRollupMapper(),
),
catchup=False,
):
@taskdefgenerate_report(dag_run=None):
# dag_run.partition_key will be the week key, e.g. "2024-01-15 (W03)"print(dag_run.partition_key)
generate_report()

[Lee-W]

The backend part is basically wrapped up, but the frontend and API side need some work. The UI is quite weired for these cases now

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/partitioned_dag_runs.py:1

• Counting PartitionedAssetKeyLog.id can over-count when duplicate log rows exist for the same upstream partition key (e.g. retries/dup inserts), inflating total_received and potentially showing the run as satisfiable earlier than it should be. Consider counting distinct PartitionedAssetKeyLog.source_partition_key (and/or a distinct composite of (asset_id, source_partition_key)) to match the scheduler’s set-based satisfaction semantics.

# Licensed to the Apache Software Foundation (ASF) under one

[dstandish]

Hey @Lee-W

I was a little surprised to see so many lines for this change so I asked Claude to help me review whether there's any unnecessary complexity. Below are its findings. They seem plausible. What do you think?

---

The core design is clean (the Window types are tiny generators,
RollupMapper.to_upstream is a clear decode→expand→encode, and the scheduler gate comes down to one
expected.issubset(actual) check), and the test coverage on the scheduler paths is excellent. The
HA hardening in _create_dagruns_for_partitioned_asset_dags (bulk-fetch over the old N+1,
with_row_locks(skip_locked=True), the per-tick cap and deterministic order_by) all looks right.

One theme I'd like to resolve before merge: a meaningful chunk of complexity is self-induced,
concentrated in partition_mappers/temporal.py.

_compile_output_format_regex looks like more generality than the feature needs. Only two mappers
can't round-trip through strptime: StartOfWeekMapper (%V isn't strptime-parseable) and
StartOfQuarterMapper ({quarter} isn't a directive). The actual requirement is "recover Y/m/d" and
"recover Y/quarter". But the decode that motivated the whole compiler ends up discarding the token
it was built for:

StartOfWeekMapper.decode_downstream — %V is captured but never read

return datetime(int(match["Y"]), int(match["m"]), int(match["d"]))

Both decodes collapse to a dedicated per-mapper regex:

_WEEK_RE = re.compile(r"(?P\d{4})-(?P\d{2})-(?P\d{2})")
_QUARTER_RE = re.compile(r"(?P\d{4})-Q(?P[1-4])")

That drops the directive table, the {name} placeholder machinery, the placeholder_patterns arg,
and the five compile-time ValueError branches — plus the test class that only exists to exercise
that invented surface (test_rejects_adjacent_default_pattern_placeholders,
test_adjacent_placeholders_allowed_when_one_is_narrowed,
test_separator_between_default_placeholders_is_allowed). Roughly ~120 lines + tests, with
identical behavior for every shipped mapper. Could we go with the dedicated regexes for now and
add generality if/when a custom mapper actually needs it?

Follow-on: if the temporal decode simplifies, the base-class guard scaffolding in
partition_mappers/base.py — the init_subclass XOR check, expected_decoded_type, and the
runtime pairing check in RollupMapper.init — largely loses its purpose for the shipped
mappers, since it mostly exists to make the general decode/encode pair safe. Worth reassessing
whether it belongs now or arrives with the first real custom mapper.

Question (non-blocking): the audit-log path (_record_partition_audit_log /
_record_stale_apdr_audit_log) opens independent scoped=False sessions to survive an outer
rollback, plus a process-lifetime dedup set. For advisory Log rows where self.log.exception(...)
already records the same thing, is the rollback-survival guarantee worth the machinery for v1, or
would a plain log line do?

[Lee-W]

Hey @dstandish , thanks for reviewing. a few responses to the questions

Could we go with the dedicated regexes for now and add generality if/when a custom mapper actually needs it?

The regex compiler was added per @uranusjr's review: without it, a custom output_format forces the user to also override decode_downstream, which TP found surprising. Hardcoding _WEEK_RE / _QUARTER_RE saves the ~120 lines but is not that users friendly

is the rollback-survival guarantee worth the machinery for v1, or would a plain log line do?

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

[uranusjr]

I still find it somewhat awkward the UI API endpoints need to load the serialized dag to calculate data. But I guess it’s not too useful to overthink this unless there are known performance issues.

Should we add some links to Starlette for the format-parsing logic?

[dstandish]

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

I don't really have a strong feeling about that. I tried to look for it in the code but could not find it.

And I don't stand in the way of merging this thing....

But... I can't help feeling that this PR is bigger than it should be. It's feels so big that it's kind of overwhelming to properly review. It spreads the reviewer's attention over a very wide area.

[Lee-W]

Should we add some links to Starlette for the format-parsing logic?

yep, added a pointer to Starlette's compile_path in the _compile_output_format_regex docstring.

[potiuk]

@uranusjr's approval here is from Jun 2 (aa9f248), but there have been ~9 commits since — scheduler refactors, the partition-Dag-run clearing fix, mapper narrowing, etc. Worth a re-review so the green check reflects the current state before this merges. @uranusjr, could you re-confirm when you get a chance?

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

[Lee-W]

Why

Closes: #59294

Why

Asset-partitioned Dags that aggregate many upstream slices into one downstream period (e.g., 60-minute-level events rolling up into one hourly Dag run) had no way to express that requirement — the scheduler would fire the downstream run as soon as any single upstream partition arrived.

This PR implements the rollup building block from AIP-76: a Window type that enumerates the full set of upstream partitions required for a downstream period, a RollupMapper that wires a source mapper to a window, and the scheduler logic to gate Dag runs until every required upstream key is present.

What

Partition mappers/windows

• Add Window ABC and six concrete implementations (HourWindow, DayWindow, WeekWindow, MonthWindow, QuarterWindow, YearWindow) to both airflow-core and the Task SDK
• Add RollupMapper that composes a source_mapper with a Window and exposes to_upstream(downstream_key) → frozenset[str]
• Add decode_downstream / encode_upstream hooks to PartitionMapper and implement them in _BaseTemporalMapper; StartOfWeekMapper gets a regex-based override because %V is ambiguous with strptime.
• Add week_start parameter to StartOfWeekMapper for non-Monday week starts

Scheduler

• Rewrite _create_dagruns_for_partitioned_asset_dags to bulk-fetch serialized Dags and partition-key logs, removing N+1 queries, and cap per-tick work at MAX_PARTITION_DAG_RUNS_PER_TICK
• Add _resolve_asset_partition_status / _check_rollup_asset_status to evaluate rollup satisfaction; non-rollup assets continue to satisfy immediately

Serialization

• Add encode_window / decode_window and extend mapper encoder/decoder to round-trip RollupMapper and all Window subclasses

UI / API

• Enrich next_run_assets endpoint with per-asset received_count, required_count, received_keys, required_keys, and is_rollup for partitioned Dags
• Update AssetNode and AssetSchedule components to surface rollup progress (e.g. "12 / 24 received")
• Add AssetProgressCell for inline progress in the Dags list

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Sonnet 4.6

Generated-by: Claude Sonnet 4.6 following the guidelines

withDAG(
dag_id="daily_team_a_rollup",
schedule=PartitionedAssetTimetable(
assets=team_a_player_stats,
default_partition_mapper=RollupMapper(
source_mapper=StartOfDayMapper(),
window=DayWindow(),
),
),
catchup=False,
tags=["player-stats", "rollup"],
):
""" First rollup level: 24 hourly partitions of ``team_a_player_stats`` → one daily summary. ``StartOfDayMapper`` normalizes each upstream hourly timestamp (``%Y-%m-%dT%H:%M:%S``) to its day-start (``%Y-%m-%d``); ``DayWindow`` declares the downstream run needs all 24 hourly partitions before firing. Publishes ``daily_team_a`` so the monthly rollup below can consume it. """@task(outlets=[daily_team_a])defsummarise_team_a_day(dag_run=None):
"""Produce the full-day rollup once every hour has arrived."""ifTYPE_CHECKING:
assertdag_runprint(f"All 24 hourly partitions received. Day: {dag_run.partition_key}")
summarise_team_a_day()

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[Copilot]

Pull request overview

Implements “rollup” support for partition mappers (AIP-76) by introducing a RollupMapper interface with to_upstream() and using it in the scheduler to wait for a complete set of upstream partition keys before creating partitioned asset-triggered DAG runs.

Changes:

• Add RollupMapper base class (core + task SDK) with an abstract to_upstream() contract.
• Implement to_upstream() for weekly and monthly temporal mappers (core + task SDK).
• Update the scheduler’s partitioned-asset DAG-run creation logic to enforce rollup completeness when a mapper supports it.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
task-sdk/src/airflow/sdk/definitions/partition_mappers/base.py	Introduces SDK-side RollupMapper abstraction.
task-sdk/src/airflow/sdk/definitions/partition_mappers/temporal.py	Adds SDK to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/timetables/base.py	Adds get_partition_mapper() hook to the Timetable protocol.
airflow-core/src/airflow/partition_mappers/base.py	Introduces core-side RollupMapper abstraction.
airflow-core/src/airflow/partition_mappers/temporal.py	Adds core to_upstream() for week/month temporal rollups.
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Uses rollup mapper behavior to decide when partitioned asset-triggered DAG runs are ready.

[Lee-W]

from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
CronPartitionTimetable,
PartitionedAssetTimetable,
WeeklyRollupMapper,
task,
)
daily_sales=Asset(uri="file://incoming/sales/daily.csv", name="daily_sales")
# Upstream Dag: produces one partition per day (key format: "2024-01-15T00:00:00")withDAG(
dag_id="ingest_daily_sales",
schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
):
@task(outlets=[daily_sales])defingest():
passingest()
# Downstream Dag: runs once all 7 daily partitions for a week have arrivedwithDAG(
dag_id="weekly_sales_report",
schedule=PartitionedAssetTimetable(
assets=daily_sales,
default_partition_mapper=WeeklyRollupMapper(),
),
catchup=False,
):
@taskdefgenerate_report(dag_run=None):
# dag_run.partition_key will be the week key, e.g. "2024-01-15 (W03)"print(dag_run.partition_key)
generate_report()

[Lee-W]

The backend part is basically wrapped up, but the frontend and API side need some work. The UI is quite weired for these cases now

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/partitioned_dag_runs.py:1

• Counting PartitionedAssetKeyLog.id can over-count when duplicate log rows exist for the same upstream partition key (e.g. retries/dup inserts), inflating total_received and potentially showing the run as satisfiable earlier than it should be. Consider counting distinct PartitionedAssetKeyLog.source_partition_key (and/or a distinct composite of (asset_id, source_partition_key)) to match the scheduler’s set-based satisfaction semantics.

# Licensed to the Apache Software Foundation (ASF) under one

[dstandish]

Hey @Lee-W

I was a little surprised to see so many lines for this change so I asked Claude to help me review whether there's any unnecessary complexity. Below are its findings. They seem plausible. What do you think?

---

The core design is clean (the Window types are tiny generators,
RollupMapper.to_upstream is a clear decode→expand→encode, and the scheduler gate comes down to one
expected.issubset(actual) check), and the test coverage on the scheduler paths is excellent. The
HA hardening in _create_dagruns_for_partitioned_asset_dags (bulk-fetch over the old N+1,
with_row_locks(skip_locked=True), the per-tick cap and deterministic order_by) all looks right.

One theme I'd like to resolve before merge: a meaningful chunk of complexity is self-induced,
concentrated in partition_mappers/temporal.py.

_compile_output_format_regex looks like more generality than the feature needs. Only two mappers
can't round-trip through strptime: StartOfWeekMapper (%V isn't strptime-parseable) and
StartOfQuarterMapper ({quarter} isn't a directive). The actual requirement is "recover Y/m/d" and
"recover Y/quarter". But the decode that motivated the whole compiler ends up discarding the token
it was built for:

StartOfWeekMapper.decode_downstream — %V is captured but never read

return datetime(int(match["Y"]), int(match["m"]), int(match["d"]))

Both decodes collapse to a dedicated per-mapper regex:

_WEEK_RE = re.compile(r"(?P\d{4})-(?P\d{2})-(?P\d{2})")
_QUARTER_RE = re.compile(r"(?P\d{4})-Q(?P[1-4])")

That drops the directive table, the {name} placeholder machinery, the placeholder_patterns arg,
and the five compile-time ValueError branches — plus the test class that only exists to exercise
that invented surface (test_rejects_adjacent_default_pattern_placeholders,
test_adjacent_placeholders_allowed_when_one_is_narrowed,
test_separator_between_default_placeholders_is_allowed). Roughly ~120 lines + tests, with
identical behavior for every shipped mapper. Could we go with the dedicated regexes for now and
add generality if/when a custom mapper actually needs it?

Follow-on: if the temporal decode simplifies, the base-class guard scaffolding in
partition_mappers/base.py — the init_subclass XOR check, expected_decoded_type, and the
runtime pairing check in RollupMapper.init — largely loses its purpose for the shipped
mappers, since it mostly exists to make the general decode/encode pair safe. Worth reassessing
whether it belongs now or arrives with the first real custom mapper.

Question (non-blocking): the audit-log path (_record_partition_audit_log /
_record_stale_apdr_audit_log) opens independent scoped=False sessions to survive an outer
rollback, plus a process-lifetime dedup set. For advisory Log rows where self.log.exception(...)
already records the same thing, is the rollback-survival guarantee worth the machinery for v1, or
would a plain log line do?

[Lee-W]

Hey @dstandish , thanks for reviewing. a few responses to the questions

Could we go with the dedicated regexes for now and add generality if/when a custom mapper actually needs it?

The regex compiler was added per @uranusjr's review: without it, a custom output_format forces the user to also override decode_downstream, which TP found surprising. Hardcoding _WEEK_RE / _QUARTER_RE saves the ~120 lines but is not that users friendly

is the rollback-survival guarantee worth the machinery for v1, or would a plain log line do?

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

[uranusjr]

I still find it somewhat awkward the UI API endpoints need to load the serialized dag to calculate data. But I guess it’s not too useful to overthink this unless there are known performance issues.

Should we add some links to Starlette for the format-parsing logic?

[dstandish]

Agree it might not worth the machinery for v1. The only reason I added it was for better discoverability. I can drop both to plain self.log and add it back if someone asks it. WDYT?

I don't really have a strong feeling about that. I tried to look for it in the code but could not find it.

And I don't stand in the way of merging this thing....

But... I can't help feeling that this PR is bigger than it should be. It's feels so big that it's kind of overwhelming to properly review. It spreads the reviewer's attention over a very wide area.

[Lee-W]

Should we add some links to Starlette for the format-parsing logic?

yep, added a pointer to Starlette's compile_path in the _compile_output_format_regex docstring.

[potiuk]

@uranusjr's approval here is from Jun 2 (aa9f248), but there have been ~9 commits since — scheduler refactors, the partition-Dag-run clearing fix, mapper narrowing, etc. Worth a re-review so the green check reflects the current state before this merges. @uranusjr, could you re-confirm when you get a chance?

---

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting