Skip to content

fix(scheduler): load dag_run eagering for execution API validation - #56916

Merged
Lee-W merged 1 commit into
apache:mainfrom
astronomer:fix-asset-DetachedInstanceError
Oct 21, 2025
Merged

fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit into
apache:mainfrom
astronomer:fix-asset-DetachedInstanceError

Conversation

@Lee-W

@Lee-WLee-W commented Oct 21, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Why

Closes: #56763

how to reproduce (through breeze on e24cfb0)

  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ importannotationsimportosimporttimeimportpendulumfromairflow.sdkimportAsset, Metadata, dag, taskasset=Asset("test")
@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defproducer():
@task(outlets=[asset])defproducer_task():
yieldMetadata(asset=asset, extra={"test": 1})
producer_task()
@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defconsumer():
@task()defconsumer_task(ds_str):
print(ds_str)
time.sleep(5)
os._exit(1) # immediate exit, no cleanup, no heartbeatconsumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")
producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick

What

Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborgboring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Oct 21, 2025
@Lee-W
Lee-W requested review from kaxil and removed request for XD-DENG and ashbOctober 21, 2025 01:13
@Lee-W
Lee-Wforce-pushed the fix-asset-DetachedInstanceError branch from 6f35b7f to d896184CompareOctober 21, 2025 01:17
kaxil
kaxil approved these changes Oct 21, 2025
View reviewed changes

@kaxilkaxil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)

@Lee-W

Lee-W commented Oct 21, 2025•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Were you able to reproduce the error & does this fix it. (It should but worth verifying)

yep, I'm now simplifying the sample code I used and will add to the desciprion

@Lee-W

Lee-W commented Oct 21, 2025

Copy link
Copy Markdown
MemberAuthor

Updated the Dag and steps I used in detail

@kaxilkaxil added this to the Airflow 3.1.1 milestone Oct 21, 2025
@Lee-W
Lee-W merged commit 8ffb7b1 into apache:mainOct 21, 2025
60 checks passed
@Lee-W
Lee-W deleted the fix-asset-DetachedInstanceError branch October 21, 2025 02:07
@Lee-WLee-W mentioned this pull request Oct 21, 2025
Merged
github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Oct 21, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
456d965
…alidation (apache#56916)
(cherry picked from commit 8ffb7b1)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit to potiuk/airflow that referenced this pull request Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
4afd95c
…ation (apache#56916)"
This reverts commit 8ffb7b1.
@potiukpotiuk mentioned this pull request Oct 21, 2025
Merged
@potiuk

potiuk commented Oct 21, 2025

Copy link
Copy Markdown
Member

Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.

See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014

potiuk added a commit that referenced this pull request Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
209fb2e
…ation (#56916)" (#56943)
This reverts commit 8ffb7b1.
Jonpaco23 pushed a commit to Jonpaco23/airflow that referenced this pull request Oct 21, 2025
@potiuk@Jonpaco23
Revert "fix(scheduler): load dag_run eagering for execution API valid…
9e491da
…ation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
potiuk added a commit to potiuk/airflow that referenced this pull request Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
4e83430
…on API validation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiukpotiuk mentioned this pull request Oct 21, 2025
Merged
potiuk added a commit that referenced this pull request Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
5d8860f
…on API validation (#56916)" (#56943) (#56956)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit to astronomer/airflow that referenced this pull request Oct 22, 2025
@Lee-W
Reapply "fix(scheduler): load dag_run eagering for execution API vali…
dcb726b
…dation (apache#56916)" (apache#56943)
This reverts commit 209fb2e.
@Lee-WLee-W mentioned this pull request Oct 22, 2025
Merged
potiuk pushed a commit that referenced this pull request Oct 22, 2025
@Lee-W
fix(scheduler): load dag_run eagering for execution API validation (#…
833717a
…57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actionsBot pushed a commit that referenced this pull request Oct 22, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
208073c
…alidation (#57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actionsgithub-actionsBot mentioned this pull request Oct 22, 2025
Merged
Lee-W added a commit that referenced this pull request Oct 22, 2025
@github-actions@Lee-W
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
27e6e19
…alidation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit to astronomer/airflow that referenced this pull request Oct 22, 2025
@github-actions@Lee-W@kaxil
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
2c26772
…alidation (apache#57010) (apache#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit that referenced this pull request Oct 22, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
2218482
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit that referenced this pull request Oct 23, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves@kaxil
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
ead0603
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit that referenced this pull request Oct 25, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Add possibility of removing extra translations missing in…
0a65c9c
… en (#57060) (#57064)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)
Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
@potiukpotiuk mentioned this pull request Nov 8, 2025
Merged
@Lee-WLee-W mentioned this pull request Dec 23, 2025
Closed
1 task
@SameerMesiah97SameerMesiah97 mentioned this pull request Dec 27, 2025
Merged
@sicarulsicarul mentioned this pull request Apr 17, 2026
Merged
1 task
sicarul added a commit to sicarul/incubator-airflow that referenced this pull request Apr 17, 2026
@sicarul@claude
Preload source_aliases in process_executor_events
e8d0ea6
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-Authored-By: Claude <noreply@anthropic.com>
kaxil pushed a commit that referenced this pull request Apr 30, 2026
@sicarul@claude
Preload source_aliases in process_executor_events (#65422)
ef0ff5c
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-authored-by: Claude <noreply@anthropic.com>
@github-actionsgithub-actionsBot mentioned this pull request Apr 30, 2026
Merged
github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Apr 30, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
5c0f3f6
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 1, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
4ce64b4
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
potiuk pushed a commit that referenced this pull request May 1, 2026
@github-actions@sicarul@claude
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
817ffa9
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
@github-actions@sicarul
@claude@vatsrahul1001
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
d4efad0
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001vatsrahul1001 mentioned this pull request May 21, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes

Assignees

No one assigned

Labels

area:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks

3 participants

@Lee-W@potiuk@kaxil
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
fix(scheduler): load dag_run eagering for execution API validation by Lee-W · Pull Request #56916 · apache/airflow · GitHub
Skip to content
fix(scheduler): load dag_run eagering for execution API validation - #56916
Merged
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Oct 21, 2025
Merged
fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Conversation
@Lee-W
@Lee-WLee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Why
Closes: #56763
how to reproduce (through breeze on e24cfb0)
  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ importannotationsimportosimporttimeimportpendulumfromairflow.sdkimportAsset, Metadata, dag, taskasset=Asset("test")
@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defproducer():
@task(outlets=[asset])defproducer_task():
yieldMetadata(asset=asset, extra={"test": 1})
producer_task()
@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defconsumer():
@task()defconsumer_task(ds_str):
print(ds_str)
time.sleep(5)
os._exit(1) # immediate exit, no cleanup, no heartbeatconsumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")
producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick
What
Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Oct 21, 2025
@Lee-W
Lee-W
requested review from
kaxil
and removed request for
XD-DENG and ashbOctober 21, 2025 01:13
@Lee-W
Lee-Wforce-pushed
the
fix-asset-DetachedInstanceError
branch
from
6f35b7f to
d896184CompareOctober 21, 2025 01:17
kaxil
kaxil
approved these changes
Oct 21, 2025
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)
@Lee-W
Lee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Were you able to reproduce the error & does this fix it. (It should but worth verifying)
yep, I'm now simplifying the sample code I used and will add to the desciprion
@Lee-W
Lee-W
commented
Oct 21, 2025

Copy link


Copy Markdown

MemberAuthor
Updated the Dag and steps I used in detail
@kaxilkaxil
added this to the Airflow 3.1.1 milestone
Oct 21, 2025
@Lee-W
Lee-W
merged commit 8ffb7b1
into
apache:mainOct 21, 2025

60 checks passed

@Lee-W
Lee-W
deleted the

fix-asset-DetachedInstanceError

branch
October 21, 2025 02:07
@Lee-WLee-W
mentioned this pull request
Oct 21, 2025
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 21, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
456d965
…alidation (apache#56916)
(cherry picked from commit 8ffb7b1)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
4afd95c
…ation (apache#56916)"
This reverts commit 8ffb7b1.
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

@potiuk
potiuk
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.
See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014
potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
209fb2e
…ation (#56916)" (#56943)
This reverts commit 8ffb7b1.
Jonpaco23 pushed a commit
to Jonpaco23/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk@Jonpaco23
Revert "fix(scheduler): load dag_run eagering for execution API valid…
9e491da
…ation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
4e83430
…on API validation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
5d8860f
…on API validation (#56916)" (#56943) (#56956)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@Lee-W
Reapply "fix(scheduler): load dag_run eagering for execution API vali…
dcb726b
…dation (apache#56916)" (apache#56943)
This reverts commit 209fb2e.
@Lee-WLee-W
mentioned this pull request
Oct 22, 2025
 Merged

potiuk pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-W
fix(scheduler): load dag_run eagering for execution API validation (#…
833717a
…57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actionsBot
pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
208073c
…alidation (#57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Oct 22, 2025
 Merged

Lee-W added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
27e6e19
…alidation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W@kaxil
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
2c26772
…alidation (apache#57010) (apache#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
2218482
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit
that referenced
this pull request
Oct 23, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves@kaxil
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
ead0603
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit
that referenced
this pull request
Oct 25, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Add possibility of removing extra translations missing in…
0a65c9c
… en (#57060) (#57064)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)
Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
@potiukpotiuk
mentioned this pull request
Nov 8, 2025
 Merged

@Lee-WLee-W
mentioned this pull request
Dec 23, 2025

Closed

1 task
@SameerMesiah97SameerMesiah97
mentioned this pull request
Dec 27, 2025
 Merged

@sicarulsicarul
mentioned this pull request
Apr 17, 2026
 Merged

1 task
sicarul added a commit
to sicarul/incubator-airflow
that referenced
this pull request
Apr 17, 2026
@sicarul@claude
Preload source_aliases in process_executor_events
e8d0ea6
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-Authored-By: Claude <noreply@anthropic.com>
kaxil pushed a commit
that referenced
this pull request
Apr 30, 2026
@sicarul@claude
Preload source_aliases in process_executor_events (#65422)
ef0ff5c
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-authored-by: Claude <noreply@anthropic.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Apr 30, 2026
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Apr 30, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
5c0f3f6
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
May 1, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
4ce64b4
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
potiuk pushed a commit
that referenced
this pull request
May 1, 2026
@github-actions@sicarul@claude
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
817ffa9
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
vatsrahul1001 pushed a commit
that referenced
this pull request
May 20, 2026
@github-actions@sicarul
@claude@vatsrahul1001
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
d4efad0
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001vatsrahul1001
mentioned this pull request
May 21, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes

Assignees


No one assigned


Labels

area:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks


3 participants

@Lee-W@potiuk@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(scheduler): load dag_run eagering for execution API validation by Lee-W · Pull Request #56916 · apache/airflow · GitHub
Skip to content
fix(scheduler): load dag_run eagering for execution API validation - #56916
Merged
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Oct 21, 2025
Merged
fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Conversation
@Lee-W
@Lee-WLee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Why
Closes: #56763
how to reproduce (through breeze on e24cfb0)
  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ importannotationsimportosimporttimeimportpendulumfromairflow.sdkimportAsset, Metadata, dag, taskasset=Asset("test")
@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defproducer():
@task(outlets=[asset])defproducer_task():
yieldMetadata(asset=asset, extra={"test": 1})
producer_task()
@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defconsumer():
@task()defconsumer_task(ds_str):
print(ds_str)
time.sleep(5)
os._exit(1) # immediate exit, no cleanup, no heartbeatconsumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")
producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick
What
Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Oct 21, 2025
@Lee-W
Lee-W
requested review from
kaxil
and removed request for
XD-DENG and ashbOctober 21, 2025 01:13
@Lee-W
Lee-Wforce-pushed
the
fix-asset-DetachedInstanceError
branch
from
6f35b7f to
d896184CompareOctober 21, 2025 01:17
kaxil
kaxil
approved these changes
Oct 21, 2025
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)
@Lee-W
Lee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Were you able to reproduce the error & does this fix it. (It should but worth verifying)
yep, I'm now simplifying the sample code I used and will add to the desciprion
@Lee-W
Lee-W
commented
Oct 21, 2025

Copy link


Copy Markdown

MemberAuthor
Updated the Dag and steps I used in detail
@kaxilkaxil
added this to the Airflow 3.1.1 milestone
Oct 21, 2025
@Lee-W
Lee-W
merged commit 8ffb7b1
into
apache:mainOct 21, 2025

60 checks passed

@Lee-W
Lee-W
deleted the

fix-asset-DetachedInstanceError

branch
October 21, 2025 02:07
@Lee-WLee-W
mentioned this pull request
Oct 21, 2025
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 21, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
456d965
…alidation (apache#56916)
(cherry picked from commit 8ffb7b1)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
4afd95c
…ation (apache#56916)"
This reverts commit 8ffb7b1.
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

@potiuk
potiuk
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.
See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014
potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
209fb2e
…ation (#56916)" (#56943)
This reverts commit 8ffb7b1.
Jonpaco23 pushed a commit
to Jonpaco23/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk@Jonpaco23
Revert "fix(scheduler): load dag_run eagering for execution API valid…
9e491da
…ation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
4e83430
…on API validation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
5d8860f
…on API validation (#56916)" (#56943) (#56956)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@Lee-W
Reapply "fix(scheduler): load dag_run eagering for execution API vali…
dcb726b
…dation (apache#56916)" (apache#56943)
This reverts commit 209fb2e.
@Lee-WLee-W
mentioned this pull request
Oct 22, 2025
 Merged

potiuk pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-W
fix(scheduler): load dag_run eagering for execution API validation (#…
833717a
…57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actionsBot
pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
208073c
…alidation (#57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Oct 22, 2025
 Merged

Lee-W added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
27e6e19
…alidation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W@kaxil
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
2c26772
…alidation (apache#57010) (apache#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
2218482
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit
that referenced
this pull request
Oct 23, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves@kaxil
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
ead0603
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit
that referenced
this pull request
Oct 25, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Add possibility of removing extra translations missing in…
0a65c9c
… en (#57060) (#57064)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)
Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
@potiukpotiuk
mentioned this pull request
Nov 8, 2025
 Merged

@Lee-WLee-W
mentioned this pull request
Dec 23, 2025

Closed

1 task
@SameerMesiah97SameerMesiah97
mentioned this pull request
Dec 27, 2025
 Merged

@sicarulsicarul
mentioned this pull request
Apr 17, 2026
 Merged

1 task
sicarul added a commit
to sicarul/incubator-airflow
that referenced
this pull request
Apr 17, 2026
@sicarul@claude
Preload source_aliases in process_executor_events
e8d0ea6
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-Authored-By: Claude <noreply@anthropic.com>
kaxil pushed a commit
that referenced
this pull request
Apr 30, 2026
@sicarul@claude
Preload source_aliases in process_executor_events (#65422)
ef0ff5c
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-authored-by: Claude <noreply@anthropic.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Apr 30, 2026
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Apr 30, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
5c0f3f6
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
May 1, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
4ce64b4
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
potiuk pushed a commit
that referenced
this pull request
May 1, 2026
@github-actions@sicarul@claude
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
817ffa9
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
vatsrahul1001 pushed a commit
that referenced
this pull request
May 20, 2026
@github-actions@sicarul
@claude@vatsrahul1001
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
d4efad0
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001vatsrahul1001
mentioned this pull request
May 21, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes

Assignees


No one assigned


Labels

area:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks


3 participants

@Lee-W@potiuk@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(scheduler): load dag_run eagering for execution API validation by Lee-W · Pull Request #56916 · apache/airflow · GitHub
Skip to content
fix(scheduler): load dag_run eagering for execution API validation - #56916
Merged
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Oct 21, 2025
Merged
fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Conversation
@Lee-W
@Lee-WLee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Why
Closes: #56763
how to reproduce (through breeze on e24cfb0)
  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ importannotationsimportosimporttimeimportpendulumfromairflow.sdkimportAsset, Metadata, dag, taskasset=Asset("test")
@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defproducer():
@task(outlets=[asset])defproducer_task():
yieldMetadata(asset=asset, extra={"test": 1})
producer_task()
@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defconsumer():
@task()defconsumer_task(ds_str):
print(ds_str)
time.sleep(5)
os._exit(1) # immediate exit, no cleanup, no heartbeatconsumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")
producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick
What
Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Oct 21, 2025
@Lee-W
Lee-W
requested review from
kaxil
and removed request for
XD-DENG and ashbOctober 21, 2025 01:13
@Lee-W
Lee-Wforce-pushed
the
fix-asset-DetachedInstanceError
branch
from
6f35b7f to
d896184CompareOctober 21, 2025 01:17
kaxil
kaxil
approved these changes
Oct 21, 2025
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)
@Lee-W
Lee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Were you able to reproduce the error & does this fix it. (It should but worth verifying)
yep, I'm now simplifying the sample code I used and will add to the desciprion
@Lee-W
Lee-W
commented
Oct 21, 2025

Copy link


Copy Markdown

MemberAuthor
Updated the Dag and steps I used in detail
@kaxilkaxil
added this to the Airflow 3.1.1 milestone
Oct 21, 2025
@Lee-W
Lee-W
merged commit 8ffb7b1
into
apache:mainOct 21, 2025

60 checks passed

@Lee-W
Lee-W
deleted the

fix-asset-DetachedInstanceError

branch
October 21, 2025 02:07
@Lee-WLee-W
mentioned this pull request
Oct 21, 2025
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 21, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
456d965
…alidation (apache#56916)
(cherry picked from commit 8ffb7b1)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
4afd95c
…ation (apache#56916)"
This reverts commit 8ffb7b1.
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

@potiuk
potiuk
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.
See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014
potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
209fb2e
…ation (#56916)" (#56943)
This reverts commit 8ffb7b1.
Jonpaco23 pushed a commit
to Jonpaco23/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk@Jonpaco23
Revert "fix(scheduler): load dag_run eagering for execution API valid…
9e491da
…ation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
4e83430
…on API validation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
5d8860f
…on API validation (#56916)" (#56943) (#56956)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@Lee-W
Reapply "fix(scheduler): load dag_run eagering for execution API vali…
dcb726b
…dation (apache#56916)" (apache#56943)
This reverts commit 209fb2e.
@Lee-WLee-W
mentioned this pull request
Oct 22, 2025
 Merged

potiuk pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-W
fix(scheduler): load dag_run eagering for execution API validation (#…
833717a
…57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actionsBot
pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
208073c
…alidation (#57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Oct 22, 2025
 Merged

Lee-W added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
27e6e19
…alidation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W@kaxil
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
2c26772
…alidation (apache#57010) (apache#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
2218482
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit
that referenced
this pull request
Oct 23, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves@kaxil
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
ead0603
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit
that referenced
this pull request
Oct 25, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Add possibility of removing extra translations missing in…
0a65c9c
… en (#57060) (#57064)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)
Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
@potiukpotiuk
mentioned this pull request
Nov 8, 2025
 Merged

@Lee-WLee-W
mentioned this pull request
Dec 23, 2025

Closed

1 task
@SameerMesiah97SameerMesiah97
mentioned this pull request
Dec 27, 2025
 Merged

@sicarulsicarul
mentioned this pull request
Apr 17, 2026
 Merged

1 task
sicarul added a commit
to sicarul/incubator-airflow
that referenced
this pull request
Apr 17, 2026
@sicarul@claude
Preload source_aliases in process_executor_events
e8d0ea6
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-Authored-By: Claude <noreply@anthropic.com>
kaxil pushed a commit
that referenced
this pull request
Apr 30, 2026
@sicarul@claude
Preload source_aliases in process_executor_events (#65422)
ef0ff5c
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-authored-by: Claude <noreply@anthropic.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Apr 30, 2026
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Apr 30, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
5c0f3f6
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
May 1, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
4ce64b4
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
potiuk pushed a commit
that referenced
this pull request
May 1, 2026
@github-actions@sicarul@claude
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
817ffa9
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
vatsrahul1001 pushed a commit
that referenced
this pull request
May 20, 2026
@github-actions@sicarul
@claude@vatsrahul1001
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
d4efad0
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001vatsrahul1001
mentioned this pull request
May 21, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes

Assignees


No one assigned


Labels

area:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks


3 participants

@Lee-W@potiuk@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
fix(scheduler): load dag_run eagering for execution API validation by Lee-W · Pull Request #56916 · apache/airflow · GitHub
Skip to content
fix(scheduler): load dag_run eagering for execution API validation - #56916
Merged
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Oct 21, 2025
Merged
fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Conversation
@Lee-W
@Lee-WLee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Why
Closes: #56763
how to reproduce (through breeze on e24cfb0)
  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ importannotationsimportosimporttimeimportpendulumfromairflow.sdkimportAsset, Metadata, dag, taskasset=Asset("test")
@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defproducer():
@task(outlets=[asset])defproducer_task():
yieldMetadata(asset=asset, extra={"test": 1})
producer_task()
@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defconsumer():
@task()defconsumer_task(ds_str):
print(ds_str)
time.sleep(5)
os._exit(1) # immediate exit, no cleanup, no heartbeatconsumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")
producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick
What
Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Oct 21, 2025
@Lee-W
Lee-W
requested review from
kaxil
and removed request for
XD-DENG and ashbOctober 21, 2025 01:13
@Lee-W
Lee-Wforce-pushed
the
fix-asset-DetachedInstanceError
branch
from
6f35b7f to
d896184CompareOctober 21, 2025 01:17
kaxil
kaxil
approved these changes
Oct 21, 2025
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)
@Lee-W
Lee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Were you able to reproduce the error & does this fix it. (It should but worth verifying)
yep, I'm now simplifying the sample code I used and will add to the desciprion
@Lee-W
Lee-W
commented
Oct 21, 2025

Copy link


Copy Markdown

MemberAuthor
Updated the Dag and steps I used in detail
@kaxilkaxil
added this to the Airflow 3.1.1 milestone
Oct 21, 2025
@Lee-W
Lee-W
merged commit 8ffb7b1
into
apache:mainOct 21, 2025

60 checks passed

@Lee-W
Lee-W
deleted the

fix-asset-DetachedInstanceError

branch
October 21, 2025 02:07
@Lee-WLee-W
mentioned this pull request
Oct 21, 2025
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 21, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
456d965
…alidation (apache#56916)
(cherry picked from commit 8ffb7b1)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
4afd95c
…ation (apache#56916)"
This reverts commit 8ffb7b1.
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

@potiuk
potiuk
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.
See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014
potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
209fb2e
…ation (#56916)" (#56943)
This reverts commit 8ffb7b1.
Jonpaco23 pushed a commit
to Jonpaco23/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk@Jonpaco23
Revert "fix(scheduler): load dag_run eagering for execution API valid…
9e491da
…ation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
4e83430
…on API validation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
5d8860f
…on API validation (#56916)" (#56943) (#56956)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@Lee-W
Reapply "fix(scheduler): load dag_run eagering for execution API vali…
dcb726b
…dation (apache#56916)" (apache#56943)
This reverts commit 209fb2e.
@Lee-WLee-W
mentioned this pull request
Oct 22, 2025
 Merged

potiuk pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-W
fix(scheduler): load dag_run eagering for execution API validation (#…
833717a
…57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actionsBot
pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
208073c
…alidation (#57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Oct 22, 2025
 Merged

Lee-W added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
27e6e19
…alidation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W@kaxil
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
2c26772
…alidation (apache#57010) (apache#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
2218482
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit
that referenced
this pull request
Oct 23, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves@kaxil
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
ead0603
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit
that referenced
this pull request
Oct 25, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Add possibility of removing extra translations missing in…
0a65c9c
… en (#57060) (#57064)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)
Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
@potiukpotiuk
mentioned this pull request
Nov 8, 2025
 Merged

@Lee-WLee-W
mentioned this pull request
Dec 23, 2025

Closed

1 task
@SameerMesiah97SameerMesiah97
mentioned this pull request
Dec 27, 2025
 Merged

@sicarulsicarul
mentioned this pull request
Apr 17, 2026
 Merged

1 task
sicarul added a commit
to sicarul/incubator-airflow
that referenced
this pull request
Apr 17, 2026
@sicarul@claude
Preload source_aliases in process_executor_events
e8d0ea6
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-Authored-By: Claude <noreply@anthropic.com>
kaxil pushed a commit
that referenced
this pull request
Apr 30, 2026
@sicarul@claude
Preload source_aliases in process_executor_events (#65422)
ef0ff5c
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-authored-by: Claude <noreply@anthropic.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Apr 30, 2026
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Apr 30, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
5c0f3f6
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
May 1, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
4ce64b4
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
potiuk pushed a commit
that referenced
this pull request
May 1, 2026
@github-actions@sicarul@claude
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
817ffa9
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
vatsrahul1001 pushed a commit
that referenced
this pull request
May 20, 2026
@github-actions@sicarul
@claude@vatsrahul1001
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
d4efad0
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001vatsrahul1001
mentioned this pull request
May 21, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes

Assignees


No one assigned


Labels

area:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks


3 participants

@Lee-W@potiuk@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(scheduler): load dag_run eagering for execution API validation by Lee-W · Pull Request #56916 · apache/airflow · GitHub
Skip to content
fix(scheduler): load dag_run eagering for execution API validation - #56916
Merged
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Oct 21, 2025
Merged
fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Conversation
@Lee-W
@Lee-WLee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Why
Closes: #56763
how to reproduce (through breeze on e24cfb0)
  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ importannotationsimportosimporttimeimportpendulumfromairflow.sdkimportAsset, Metadata, dag, taskasset=Asset("test")
@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defproducer():
@task(outlets=[asset])defproducer_task():
yieldMetadata(asset=asset, extra={"test": 1})
producer_task()
@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defconsumer():
@task()defconsumer_task(ds_str):
print(ds_str)
time.sleep(5)
os._exit(1) # immediate exit, no cleanup, no heartbeatconsumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")
producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick
What
Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Oct 21, 2025
@Lee-W
Lee-W
requested review from
kaxil
and removed request for
XD-DENG and ashbOctober 21, 2025 01:13
@Lee-W
Lee-Wforce-pushed
the
fix-asset-DetachedInstanceError
branch
from
6f35b7f to
d896184CompareOctober 21, 2025 01:17
kaxil
kaxil
approved these changes
Oct 21, 2025
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)
@Lee-W
Lee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Were you able to reproduce the error & does this fix it. (It should but worth verifying)
yep, I'm now simplifying the sample code I used and will add to the desciprion
@Lee-W
Lee-W
commented
Oct 21, 2025

Copy link


Copy Markdown

MemberAuthor
Updated the Dag and steps I used in detail
@kaxilkaxil
added this to the Airflow 3.1.1 milestone
Oct 21, 2025
@Lee-W
Lee-W
merged commit 8ffb7b1
into
apache:mainOct 21, 2025

60 checks passed

@Lee-W
Lee-W
deleted the

fix-asset-DetachedInstanceError

branch
October 21, 2025 02:07
@Lee-WLee-W
mentioned this pull request
Oct 21, 2025
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 21, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
456d965
…alidation (apache#56916)
(cherry picked from commit 8ffb7b1)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
4afd95c
…ation (apache#56916)"
This reverts commit 8ffb7b1.
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

@potiuk
potiuk
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.
See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014
potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
209fb2e
…ation (#56916)" (#56943)
This reverts commit 8ffb7b1.
Jonpaco23 pushed a commit
to Jonpaco23/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk@Jonpaco23
Revert "fix(scheduler): load dag_run eagering for execution API valid…
9e491da
…ation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
4e83430
…on API validation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
5d8860f
…on API validation (#56916)" (#56943) (#56956)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@Lee-W
Reapply "fix(scheduler): load dag_run eagering for execution API vali…
dcb726b
…dation (apache#56916)" (apache#56943)
This reverts commit 209fb2e.
@Lee-WLee-W
mentioned this pull request
Oct 22, 2025
 Merged

potiuk pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-W
fix(scheduler): load dag_run eagering for execution API validation (#…
833717a
…57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actionsBot
pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
208073c
…alidation (#57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Oct 22, 2025
 Merged

Lee-W added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
27e6e19
…alidation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W@kaxil
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
2c26772
…alidation (apache#57010) (apache#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
2218482
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit
that referenced
this pull request
Oct 23, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves@kaxil
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
ead0603
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit
that referenced
this pull request
Oct 25, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Add possibility of removing extra translations missing in…
0a65c9c
… en (#57060) (#57064)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)
Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
@potiukpotiuk
mentioned this pull request
Nov 8, 2025
 Merged

@Lee-WLee-W
mentioned this pull request
Dec 23, 2025

Closed

1 task
@SameerMesiah97SameerMesiah97
mentioned this pull request
Dec 27, 2025
 Merged

@sicarulsicarul
mentioned this pull request
Apr 17, 2026
 Merged

1 task
sicarul added a commit
to sicarul/incubator-airflow
that referenced
this pull request
Apr 17, 2026
@sicarul@claude
Preload source_aliases in process_executor_events
e8d0ea6
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-Authored-By: Claude <noreply@anthropic.com>
kaxil pushed a commit
that referenced
this pull request
Apr 30, 2026
@sicarul@claude
Preload source_aliases in process_executor_events (#65422)
ef0ff5c
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-authored-by: Claude <noreply@anthropic.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Apr 30, 2026
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Apr 30, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
5c0f3f6
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
May 1, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
4ce64b4
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
potiuk pushed a commit
that referenced
this pull request
May 1, 2026
@github-actions@sicarul@claude
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
817ffa9
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
vatsrahul1001 pushed a commit
that referenced
this pull request
May 20, 2026
@github-actions@sicarul
@claude@vatsrahul1001
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
d4efad0
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001vatsrahul1001
mentioned this pull request
May 21, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes

Assignees


No one assigned


Labels

area:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks


3 participants

@Lee-W@potiuk@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(scheduler): load dag_run eagering for execution API validation by Lee-W · Pull Request #56916 · apache/airflow · GitHub
Skip to content
fix(scheduler): load dag_run eagering for execution API validation - #56916
Merged
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Oct 21, 2025
Merged
fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Conversation
@Lee-W
@Lee-WLee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Why
Closes: #56763
how to reproduce (through breeze on e24cfb0)
  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ importannotationsimportosimporttimeimportpendulumfromairflow.sdkimportAsset, Metadata, dag, taskasset=Asset("test")
@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defproducer():
@task(outlets=[asset])defproducer_task():
yieldMetadata(asset=asset, extra={"test": 1})
producer_task()
@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defconsumer():
@task()defconsumer_task(ds_str):
print(ds_str)
time.sleep(5)
os._exit(1) # immediate exit, no cleanup, no heartbeatconsumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")
producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick
What
Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Oct 21, 2025
@Lee-W
Lee-W
requested review from
kaxil
and removed request for
XD-DENG and ashbOctober 21, 2025 01:13
@Lee-W
Lee-Wforce-pushed
the
fix-asset-DetachedInstanceError
branch
from
6f35b7f to
d896184CompareOctober 21, 2025 01:17
kaxil
kaxil
approved these changes
Oct 21, 2025
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)
@Lee-W
Lee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Were you able to reproduce the error & does this fix it. (It should but worth verifying)
yep, I'm now simplifying the sample code I used and will add to the desciprion
@Lee-W
Lee-W
commented
Oct 21, 2025

Copy link


Copy Markdown

MemberAuthor
Updated the Dag and steps I used in detail
@kaxilkaxil
added this to the Airflow 3.1.1 milestone
Oct 21, 2025
@Lee-W
Lee-W
merged commit 8ffb7b1
into
apache:mainOct 21, 2025

60 checks passed

@Lee-W
Lee-W
deleted the

fix-asset-DetachedInstanceError

branch
October 21, 2025 02:07
@Lee-WLee-W
mentioned this pull request
Oct 21, 2025
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 21, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
456d965
…alidation (apache#56916)
(cherry picked from commit 8ffb7b1)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
4afd95c
…ation (apache#56916)"
This reverts commit 8ffb7b1.
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

@potiuk
potiuk
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.
See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014
potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
209fb2e
…ation (#56916)" (#56943)
This reverts commit 8ffb7b1.
Jonpaco23 pushed a commit
to Jonpaco23/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk@Jonpaco23
Revert "fix(scheduler): load dag_run eagering for execution API valid…
9e491da
…ation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
4e83430
…on API validation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
5d8860f
…on API validation (#56916)" (#56943) (#56956)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@Lee-W
Reapply "fix(scheduler): load dag_run eagering for execution API vali…
dcb726b
…dation (apache#56916)" (apache#56943)
This reverts commit 209fb2e.
@Lee-WLee-W
mentioned this pull request
Oct 22, 2025
 Merged

potiuk pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-W
fix(scheduler): load dag_run eagering for execution API validation (#…
833717a
…57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actionsBot
pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
208073c
…alidation (#57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Oct 22, 2025
 Merged

Lee-W added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
27e6e19
…alidation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W@kaxil
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
2c26772
…alidation (apache#57010) (apache#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
2218482
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit
that referenced
this pull request
Oct 23, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves@kaxil
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
ead0603
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit
that referenced
this pull request
Oct 25, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Add possibility of removing extra translations missing in…
0a65c9c
… en (#57060) (#57064)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)
Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
@potiukpotiuk
mentioned this pull request
Nov 8, 2025
 Merged

@Lee-WLee-W
mentioned this pull request
Dec 23, 2025

Closed

1 task
@SameerMesiah97SameerMesiah97
mentioned this pull request
Dec 27, 2025
 Merged

@sicarulsicarul
mentioned this pull request
Apr 17, 2026
 Merged

1 task
sicarul added a commit
to sicarul/incubator-airflow
that referenced
this pull request
Apr 17, 2026
@sicarul@claude
Preload source_aliases in process_executor_events
e8d0ea6
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-Authored-By: Claude <noreply@anthropic.com>
kaxil pushed a commit
that referenced
this pull request
Apr 30, 2026
@sicarul@claude
Preload source_aliases in process_executor_events (#65422)
ef0ff5c
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-authored-by: Claude <noreply@anthropic.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Apr 30, 2026
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Apr 30, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
5c0f3f6
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
May 1, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
4ce64b4
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
potiuk pushed a commit
that referenced
this pull request
May 1, 2026
@github-actions@sicarul@claude
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
817ffa9
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
vatsrahul1001 pushed a commit
that referenced
this pull request
May 20, 2026
@github-actions@sicarul
@claude@vatsrahul1001
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
d4efad0
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001vatsrahul1001
mentioned this pull request
May 21, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes

Assignees


No one assigned


Labels

area:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks


3 participants

@Lee-W@potiuk@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
fix(scheduler): load dag_run eagering for execution API validation by Lee-W · Pull Request #56916 · apache/airflow · GitHub
Skip to content
fix(scheduler): load dag_run eagering for execution API validation - #56916
Merged
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Oct 21, 2025
Merged
fix(scheduler): load dag_run eagering for execution API validation#56916
Lee-W merged 1 commit into
apache:mainfrom 
astronomer:fix-asset-DetachedInstanceError
Conversation
@Lee-W
@Lee-WLee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Why
Closes: #56763
how to reproduce (through breeze on e24cfb0)
  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ importannotationsimportosimporttimeimportpendulumfromairflow.sdkimportAsset, Metadata, dag, taskasset=Asset("test")
@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defproducer():
@task(outlets=[asset])defproducer_task():
yieldMetadata(asset=asset, extra={"test": 1})
producer_task()
@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)defconsumer():
@task()defconsumer_task(ds_str):
print(ds_str)
time.sleep(5)
os._exit(1) # immediate exit, no cleanup, no heartbeatconsumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")
producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick
What
Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Oct 21, 2025
@Lee-W
Lee-W
requested review from
kaxil
and removed request for
XD-DENG and ashbOctober 21, 2025 01:13
@Lee-W
Lee-Wforce-pushed
the
fix-asset-DetachedInstanceError
branch
from
6f35b7f to
d896184CompareOctober 21, 2025 01:17
kaxil
kaxil
approved these changes
Oct 21, 2025
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)
@Lee-W
Lee-W
commented
Oct 21, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Were you able to reproduce the error & does this fix it. (It should but worth verifying)
yep, I'm now simplifying the sample code I used and will add to the desciprion
@Lee-W
Lee-W
commented
Oct 21, 2025

Copy link


Copy Markdown

MemberAuthor
Updated the Dag and steps I used in detail
@kaxilkaxil
added this to the Airflow 3.1.1 milestone
Oct 21, 2025
@Lee-W
Lee-W
merged commit 8ffb7b1
into
apache:mainOct 21, 2025

60 checks passed

@Lee-W
Lee-W
deleted the

fix-asset-DetachedInstanceError

branch
October 21, 2025 02:07
@Lee-WLee-W
mentioned this pull request
Oct 21, 2025
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 21, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
456d965
…alidation (apache#56916)
(cherry picked from commit 8ffb7b1)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
4afd95c
…ation (apache#56916)"
This reverts commit 8ffb7b1.
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

@potiuk
potiuk
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.
See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014
potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
Revert "fix(scheduler): load dag_run eagering for execution API valid…
209fb2e
…ation (#56916)" (#56943)
This reverts commit 8ffb7b1.
Jonpaco23 pushed a commit
to Jonpaco23/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk@Jonpaco23
Revert "fix(scheduler): load dag_run eagering for execution API valid…
9e491da
…ation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
4e83430
…on API validation (apache#56916)" (apache#56943)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
@potiukpotiuk
mentioned this pull request
Oct 21, 2025
 Merged

potiuk added a commit
that referenced
this pull request
Oct 21, 2025
@potiuk
[v3-1-test] Revert "fix(scheduler): load dag_run eagering for executi…
5d8860f
…on API validation (#56916)" (#56943) (#56956)
This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@Lee-W
Reapply "fix(scheduler): load dag_run eagering for execution API vali…
dcb726b
…dation (apache#56916)" (apache#56943)
This reverts commit 209fb2e.
@Lee-WLee-W
mentioned this pull request
Oct 22, 2025
 Merged

potiuk pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-W
fix(scheduler): load dag_run eagering for execution API validation (#…
833717a
…57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actionsBot
pushed a commit
that referenced
this pull request
Oct 22, 2025
@Lee-WYour friendly bot
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
208073c
…alidation (#57010)
* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
This reverts commit 209fb2e.
* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Oct 22, 2025
 Merged

Lee-W added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
27e6e19
…alidation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Oct 22, 2025
@github-actions@Lee-W@kaxil
[v3-1-test] fix(scheduler): load dag_run eagering for execution API v…
2c26772
…alidation (apache#57010) (apache#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit
that referenced
this pull request
Oct 22, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
2218482
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit
that referenced
this pull request
Oct 23, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves@kaxil
[v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#…
ead0603
…57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit
that referenced
this pull request
Oct 25, 2025
@github-actions@sjyangkevin
@tirkarthi@pierrejeambrun@Lee-W@potiuk@anshuksi282-ksolves
[v3-1-test] Add possibility of removing extra translations missing in…
0a65c9c
… en (#57060) (#57064)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)
* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)
(cherry picked from commit 8d71c07)
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* [v3-1-test] Fix dags list not rendering (#57037) (#57046)
(cherry picked from commit 8668fe5)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)
Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
* [v3-1-test] Add new PL translations. (#56359) (#57055)
(cherry picked from commit 6583b6d)
* Add last missing translation for 3.1.1 (#57057)
* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)
* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)
* fix: adjust PanelButtons spacing and alignment
* Fix PanelButtons layout and button styling
* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)
Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
@potiukpotiuk
mentioned this pull request
Nov 8, 2025
 Merged

@Lee-WLee-W
mentioned this pull request
Dec 23, 2025

Closed

1 task
@SameerMesiah97SameerMesiah97
mentioned this pull request
Dec 27, 2025
 Merged

@sicarulsicarul
mentioned this pull request
Apr 17, 2026
 Merged

1 task
sicarul added a commit
to sicarul/incubator-airflow
that referenced
this pull request
Apr 17, 2026
@sicarul@claude
Preload source_aliases in process_executor_events
e8d0ea6
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-Authored-By: Claude <noreply@anthropic.com>
kaxil pushed a commit
that referenced
this pull request
Apr 30, 2026
@sicarul@claude
Preload source_aliases in process_executor_events (#65422)
ef0ff5c
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
Co-authored-by: Claude <noreply@anthropic.com>
@github-actionsgithub-actionsBot
mentioned this pull request
Apr 30, 2026
 Merged

github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Apr 30, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
5c0f3f6
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
May 1, 2026
@sicarul@claudeYour friendly bot
[v3-2-test] Preload source_aliases in process_executor_events (apache…
4ce64b4
…#65422)
PR apache#56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
potiuk pushed a commit
that referenced
this pull request
May 1, 2026
@github-actions@sicarul@claude
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
817ffa9
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
vatsrahul1001 pushed a commit
that referenced
this pull request
May 20, 2026
@github-actions@sicarul
@claude@vatsrahul1001
[v3-2-test] Preload source_aliases in process_executor_events (#65422) (
d4efad0
#66191)
PR #56916 added a _eager_load_dag_run_for_validation helper that
returns two loader options so DRDataModel.model_validate can walk
DagRun.consumed_asset_events[*].asset and source_aliases without
triggering lazy loads on a detached ORM instance. The helper's
docstring example applies both loaders, and _adopt_or_reset_orphaned_tasks
applies both. SchedulerJobRunner.process_executor_events only applies
asset_loader and discards alias_loader, leaving
AssetEvent.source_aliases lazy-loaded.
When Celery/Kubernetes executor reports a failed-while-queued event
for an asset-triggered DagRun whose consumed AssetEvent has a
non-empty source_aliases collection, SQLAlchemy detaches the
AssetEvent by the time pydantic walks the DagRun, and source_aliases
raises DetachedInstanceError wrapped as a pydantic ValidationError.
The exception escapes _run_scheduler_loop, the scheduler exits,
which orphans in-flight task instances. The next scheduler's
_adopt_or_reset_orphaned_tasks picks them up and replays the same
validation path, crashing again.
Apply alias_loader alongside asset_loader at this call site.
Extend test_process_executor_events_with_asset_events so the
AssetEvent now carries a non-empty source_aliases collection and
the assertion verifies the alias survives into the callback payload.
(cherry picked from commit ef0ff5c)
Co-authored-by: Pablo Seibelt <pabloseibelt@sicarul.com>
Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001vatsrahul1001
mentioned this pull request
May 21, 2026

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes

Assignees


No one assigned


Labels

area:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks


3 participants

@Lee-W@potiuk@kaxil