Skip to content

Expose upstream partition key on triggering asset events - #69115

Merged
Lee-W merged 1 commit into
apache:mainfrom
astronomer:expose-partition-key-on-triggering-asset-events
Jun 29, 2026
Merged

Expose upstream partition key on triggering asset events#69115
Lee-W merged 1 commit into
apache:mainfrom
astronomer:expose-partition-key-on-triggering-asset-events

Conversation

@Lee-W

@Lee-WLee-W commented Jun 29, 2026

Copy link
Copy Markdown
Member

Why

  • A downstream consumer task couldn't tell which upstream partitions fed its run — triggering_asset_events (and dag_run.consumed_asset_events) exposed each event reference without a partition_key.
  • The sibling schema used for inlet_events (AssetEventResponse) already carries partition_key under the same unreleased feature, so the consumed-event reference was just an incomplete rollout of it.

What

  • Add partition_key to AssetEventDagRunReference (core datamodel) and regenerate the task-sdk _generated.py from it.
  • The field is mapper-agnostic: the scheduler reads consumed events from PartitionedAssetKeyLog regardless of mapper kind, so rollup (N→1, multiple events), fan-out (1→N, single event), and identity (1→1, single event) all populate it. Authors read it run-scoped, with no inlet wiring or DB lookup: [e.partition_key for e in triggering_asset_events[asset]].
from __future__ importannotationsfromairflow.sdkimport (
DAG,
Asset,
FixedKeyMapper,
PartitionedAssetTimetable,
PartitionedAtRuntime,
RollupMapper,
SegmentWindow,
task,
)
region_stats=Asset(name="region_stats", uri="memory://region_stats")
withDAG(
dag_id="produce_region_stats",
schedule=PartitionedAtRuntime(),
catchup=False,
tags=["play", "partitions"],
):
@task(outlets=[region_stats])defemit_regions(*, outlet_events):
"""Emit one partition per region from a single run."""outlet_events[region_stats].add_partitions(["us", "eu", "apac"])
emit_regions()
withDAG(
dag_id="rollup_region_stats",
schedule=PartitionedAssetTimetable(
assets=region_stats,
default_partition_mapper=RollupMapper(
upstream_mapper=FixedKeyMapper("all_regions"),
window=SegmentWindow(["us", "eu", "apac"]),
),
),
catchup=False,
tags=["play", "partitions"],
):
@taskdefsummarize(*, dag_run, triggering_asset_events):
"""Print the downstream key and the upstream partitions that produced it."""contributed= [event.partition_keyforeventintriggering_asset_events[region_stats]]
print(f"downstream partition = {dag_run.partition_key}")
print(f"contributing upstream partitions = {sorted(contributed)}")
summarize()

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborgboring-cyborgBot added area:API Airflow's REST/HTTP API area:task-sdk labels Jun 29, 2026
@Lee-WLee-W self-assigned this Jun 29, 2026
@Lee-WLee-W added the backport-to-v3-3-test Backport to v3-3-test label Jun 29, 2026
@goransh-walia

Copy link
Copy Markdown

Nice clean implementation. One minor thought: adding a docstring would help future contributors. But overall this looks solid.

Add partition_key to the consumed-asset-event reference so a downstream
task can read which upstream partitions fed its run via
triggering_asset_events (and dag_run.consumed_asset_events). The field is
stripped for execution API clients older than 2026-04-06.
@Lee-W
Lee-Wforce-pushed the expose-partition-key-on-triggering-asset-events branch from 3b3b232 to fa8edf0CompareJune 29, 2026 08:47
@Lee-W
Lee-W merged commit 1dcdcd7 into apache:mainJun 29, 2026
102 checks passed
@Lee-W
Lee-W deleted the expose-partition-key-on-triggering-asset-events branch June 29, 2026 09:51
@Lee-WLee-W added this to the Airflow 3.3.0 milestone Jun 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-3-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jun 29, 2026
…pache#69115)
(cherry picked from commit 1dcdcd7)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jun 29, 2026
…pache#69115)
(cherry picked from commit 1dcdcd7)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Jun 29, 2026
…69115) (#69120)
(cherry picked from commit 1dcdcd7)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jun 30, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:APIAirflow's REST/HTTP APIarea:task-sdkbackport-to-v3-3-testBackport to v3-3-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@Lee-W@goransh-walia@pierrejeambrun@amoghrajesh