Reference asset events from asset_dag_run_queue by uranusjr · Pull Request #70972 · apache/airflow

uranusjr · 2026-08-03T08:41:01Z

More details in #70970.

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix#62501 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity.

This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

dingo4dev · 2026-08-03T14:42:10Z

@uranusjr Thank you for this follow-up fix. Moving ADRQ to reference AssetEvent directly resolves the race I hit in #62501 while restoring transaction atomicity. I really appreciate the careful implementation and quick turnaround.

uranusjr · 2026-08-04T10:05:33Z

Special note: This SHOULD NOT be backported.

Lee-W

a few nits

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix 9e5b770 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity. This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

apache#70972 already consumes queued asset events by id, so concurrent mapped outlets no longer strand events behind a timestamp watermark. These tests lock that contract for mapped producers so a later consume-path change cannot silently drop events from triggering_asset_events. closes: apache#54659

uranusjr · 2026-08-03T08:41:01Z

More details in #70970.

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix#62501 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity.

This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

dingo4dev · 2026-08-03T14:42:10Z

@uranusjr Thank you for this follow-up fix. Moving ADRQ to reference AssetEvent directly resolves the race I hit in #62501 while restoring transaction atomicity. I really appreciate the careful implementation and quick turnaround.

uranusjr · 2026-08-04T10:05:33Z

Special note: This SHOULD NOT be backported.

Lee-W

a few nits

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix 9e5b770 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity. This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

apache#70972 already consumes queued asset events by id, so concurrent mapped outlets no longer strand events behind a timestamp watermark. These tests lock that contract for mapped producers so a later consume-path change cannot silently drop events from triggering_asset_events. closes: apache#54659

uranusjr · 2026-08-03T08:41:01Z

More details in #70970.

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix#62501 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity.

This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

dingo4dev · 2026-08-03T14:42:10Z

@uranusjr Thank you for this follow-up fix. Moving ADRQ to reference AssetEvent directly resolves the race I hit in #62501 while restoring transaction atomicity. I really appreciate the careful implementation and quick turnaround.

uranusjr · 2026-08-04T10:05:33Z

Special note: This SHOULD NOT be backported.

Lee-W

a few nits

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix 9e5b770 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity. This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

apache#70972 already consumes queued asset events by id, so concurrent mapped outlets no longer strand events behind a timestamp watermark. These tests lock that contract for mapped producers so a later consume-path change cannot silently drop events from triggering_asset_events. closes: apache#54659

uranusjr · 2026-08-03T08:41:01Z

More details in #70970.

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix#62501 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity.

This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

dingo4dev · 2026-08-03T14:42:10Z

@uranusjr Thank you for this follow-up fix. Moving ADRQ to reference AssetEvent directly resolves the race I hit in #62501 while restoring transaction atomicity. I really appreciate the careful implementation and quick turnaround.

uranusjr · 2026-08-04T10:05:33Z

Special note: This SHOULD NOT be backported.

Lee-W

a few nits

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix 9e5b770 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity. This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

apache#70972 already consumes queued asset events by id, so concurrent mapped outlets no longer strand events behind a timestamp watermark. These tests lock that contract for mapped producers so a later consume-path change cannot silently drop events from triggering_asset_events. closes: apache#54659

uranusjr · 2026-08-03T08:41:01Z

More details in #70970.

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix#62501 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity.

This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

dingo4dev · 2026-08-03T14:42:10Z

@uranusjr Thank you for this follow-up fix. Moving ADRQ to reference AssetEvent directly resolves the race I hit in #62501 while restoring transaction atomicity. I really appreciate the careful implementation and quick turnaround.

uranusjr · 2026-08-04T10:05:33Z

Special note: This SHOULD NOT be backported.

Lee-W

a few nits

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix 9e5b770 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity. This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

apache#70972 already consumes queued asset events by id, so concurrent mapped outlets no longer strand events behind a timestamp watermark. These tests lock that contract for mapped producers so a later consume-path change cannot silently drop events from triggering_asset_events. closes: apache#54659

uranusjr · 2026-08-03T08:41:01Z

More details in #70970.

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix#62501 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity.

This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

dingo4dev · 2026-08-03T14:42:10Z

@uranusjr Thank you for this follow-up fix. Moving ADRQ to reference AssetEvent directly resolves the race I hit in #62501 while restoring transaction atomicity. I really appreciate the careful implementation and quick turnaround.

uranusjr · 2026-08-04T10:05:33Z

Special note: This SHOULD NOT be backported.

Lee-W

a few nits

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix 9e5b770 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity. This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

apache#70972 already consumes queued asset events by id, so concurrent mapped outlets no longer strand events behind a timestamp watermark. These tests lock that contract for mapped producers so a later consume-path change cannot silently drop events from triggering_asset_events. closes: apache#54659

uranusjr · 2026-08-03T08:41:01Z

More details in #70970.

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix#62501 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity.

This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

dingo4dev · 2026-08-03T14:42:10Z

@uranusjr Thank you for this follow-up fix. Moving ADRQ to reference AssetEvent directly resolves the race I hit in #62501 while restoring transaction atomicity. I really appreciate the careful implementation and quick turnaround.

uranusjr · 2026-08-04T10:05:33Z

Special note: This SHOULD NOT be backported.

Lee-W

a few nits

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix 9e5b770 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity. This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

apache#70972 already consumes queued asset events by id, so concurrent mapped outlets no longer strand events behind a timestamp watermark. These tests lock that contract for mapped producers so a later consume-path change cannot silently drop events from triggering_asset_events. closes: apache#54659

uranusjr · 2026-08-03T08:41:01Z

More details in #70970.

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix#62501 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity.

This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

dingo4dev · 2026-08-03T14:42:10Z

@uranusjr Thank you for this follow-up fix. Moving ADRQ to reference AssetEvent directly resolves the race I hit in #62501 while restoring transaction atomicity. I really appreciate the careful implementation and quick turnaround.

uranusjr · 2026-08-04T10:05:33Z

Special note: This SHOULD NOT be backported.

Lee-W

a few nits

Asset-triggered scheduling had the scheduler reconstruct which AssetEvents fed a run from a created_at time window. The prior fix 9e5b770 committed each AssetEvent in separate short-lived sessions so it was visible early, but that broke atomicity. This commit changes the ADRQ to directly reference the asset event that generates it, instead of using an implicit asset_id + timestamp combination. Atomicity is restored since we no longer needs to commit separately to get the timestamps right, while still being able to reference the correct asset events from the ADRQ.

apache#70972 already consumes queued asset events by id, so concurrent mapped outlets no longer strand events behind a timestamp watermark. These tests lock that contract for mapped producers so a later consume-path change cannot silently drop events from triggering_asset_events. closes: apache#54659

uranusjr requested review from Lee-W, XD-DENG, ashb and ephraimbuddy as code owners August 3, 2026 08:41

boring-cyborgBot added area:API Airflow's REST/HTTP API area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation labels Aug 3, 2026

uranusjr added this to the Airflow 3.4.0 milestone Aug 3, 2026

uranusjr force-pushed the explore/asset-registration-atomicity-option-c branch 2 times, most recently from fc8d415 to 7a06763Compare August 3, 2026 12:40

kaxil reviewed Aug 3, 2026
View reviewed changes

uranusjr force-pushed the explore/asset-registration-atomicity-option-c branch 2 times, most recently from 673bdbd to c3ca4d1Compare August 4, 2026 08:18

uranusjr mentioned this pull request Aug 4, 2026
Release TI lock before asset listener callbacks #70951
Merged

uranusjr force-pushed the explore/asset-registration-atomicity-option-c branch from c3ca4d1 to c0d809aCompare August 4, 2026 09:15

kaxil approved these changes Aug 4, 2026
View reviewed changes

Lee-W approved these changes Aug 4, 2026
View reviewed changes

Comment threadairflow-core/docs/howto/usage-cli.rst Outdated
Comment threadairflow-core/newsfragments/70972.significant.rst Outdated
Comment threadairflow-core/newsfragments/70972.significant.rst Outdated
Comment threadairflow-core/tests/unit/assets/test_manager.py Outdated

uranusjr force-pushed the explore/asset-registration-atomicity-option-c branch from c0d809a to bece6e3Compare August 4, 2026 11:22

uranusjr merged commit 44acf38 into apache:mainAug 5, 2026
166 of 168 checks passed

uranusjr deleted the explore/asset-registration-atomicity-option-c branch August 5, 2026 01:13

dingo4dev mentioned this pull request Aug 5, 2026
Remove unreachable AssetManager slow-path helper #70996
Closed
1 task

Andrushika mentioned this pull request Aug 7, 2026
Fix deadlock when concurrent tasks queue the same asset's downstream dags #70128
Closed
1 task

dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
Reference asset events from asset_dag_run_queue (apache#70972)
b190623

This was referenced Aug 23, 2026
Consume all mapped asset events on the next triggered run #71991
Closed
Consume every queued asset event from concurrent mapped outlets #71994
Closed
Keep queued asset events when a Dag is at max_active_runs #71992
Closed

uranusjr mentioned this pull request Sep 1, 2026
Asset-triggered DAGs miss events from concurrently completing dynamic mapped tasks #54659
Closed
2 tasks