Ensure teardown tasks are executed when DAG run is set to failed by jscheffl · Pull Request #45530 · apache/airflow

jscheffl · 2025-01-09T22:27:32Z

Related to Slack topic: https://apache-airflow.slack.com/archives/CCR6P6JRL/p1736440079894049

We noticed that if a DAG run is set to failed, all tasks are either set to failed or skipped. But if Teardown Tasks are used in a DAG, they are not executed. This could lead to infrastructure or external dependencies not properly cleaned-up.

This PR changes the behavior and does NOT fail/skip teardown tasks if a DAG is set to failed.
A side effect as consequence might be that the DAG is after the call NOT failed, else if it would set it to failed, then any teardown task (even if not skipped/failed) will not scheduled anymore.

shahar1

Nice catch!
One edge case that I thought about though* - what if the corresponding setup task hasn't finished running yet? (if such exists ofc)
For example:

If you set the DAG run to failed before the cluster was created, the delete_cluster should be skipped.

* Maybe the current architecture already handles, but it's worth checking before merging this PR.

jscheffl · 2025-01-11T17:08:48Z

Nice catch! One edge case that I thought about though* - what if the corresponding setup task hasn't finished running yet? (if such exists ofc) For example: If you set the DAG run to failed before the cluster was created, the delete_cluster should be skipped.
Maybe the current architecture already handles, but it's worth checking before merging this PR.

Oh, yeah. Tested this explicitly. And actually the Scheduler had this feature before (added this to docs as well, was also new to me): If the setup task is skipped (which is the state when you mark as failed and it was not running) then the teardown is also skipped,

jscheffl · 2025-01-11T17:09:14Z

Added some docs as I thought about that slightly the behavior changes - to ensure it is properly documented.

…o failed (#45530) * Ensure teardown tasks are executed when DAG run is set to failed * Also handle the case of setting DAG to success * Add some documentation to behavior changes * Add some documentation to behavior changes (cherry picked from commit 1e8977a) Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>

github-actions · 2025-01-11T17:52:11Z

Backport successfully created: v2-10-test

Status	Branch	Result
✅	v2-10-test

…o failed (apache#45530) * Ensure teardown tasks are executed when DAG run is set to failed * Also handle the case of setting DAG to success * Add some documentation to behavior changes * Add some documentation to behavior changes (cherry picked from commit 1e8977a) Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>

…o failed (#45530) (#45581) * [v2-10-test] Ensure teardown tasks are executed when DAG run is set to failed (#45530) * Ensure teardown tasks are executed when DAG run is set to failed * Also handle the case of setting DAG to success * Add some documentation to behavior changes * Add some documentation to behavior changes (cherry picked from commit 1e8977a) Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com> * Remove type hints only working in Airflow 3 --------- Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com> Co-authored-by: Jens Scheffler <jens.scheffler@de.bosch.com>

…che#45530) * Ensure teardown tasks are executed when DAG run is set to failed * Also handle the case of setting DAG to success * Add some documentation to behavior changes * Add some documentation to behavior changes

Lee-W · 2025-01-13T23:37:02Z

just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?

potiuk · 2025-01-14T16:15:50Z

just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?

I'd say it's a bugfix. Not all "change behaviour" is "breaking change". Every bugfix is a "behavioural change" in essence, the important thing is what is the feature intention and whether the change is "fixing" things or "changing intention on how things should work". SemVer is all about intentions and not about whether behaviour changes or not - see https://semver.org/

In this case it seems reasonable to assume that this behaviour was intention of the "teardown" behaviour - so technically it's a bugfix (and it's been already backported to 2.10 it seems.

It might require a newsfragment, but IMHO - that's more than enough.

jscheffl · 2025-01-14T20:20:17Z

just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?

For sure it does not need migration rules. No code change needed.

Yeah and thought also a bit about if it is breaking, asked in #random / Slack for feedback and nobody except @dstandish was responding. Then after passing... I think it is really in the intend for why we have Setup/Teardown. Behaviour change is "just" because of the side effect that the DAG is not immediately failed if you set it to failed as Teardown need to be scheduled.

…che#45530) * Ensure teardown tasks are executed when DAG run is set to failed * Also handle the case of setting DAG to success * Add some documentation to behavior changes * Add some documentation to behavior changes

rakesh-shah · 2025-07-11T04:45:08Z

So @jscheffl we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one.
thanks

jscheffl · 2025-07-11T05:38:29Z

So @jscheffl we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one. thanks

This was fixed in 2.10.5 - See milestone flag on the right side. Also Release Notes help in this

rakesh-shah · 2025-07-11T11:32:15Z

So @jscheffl we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one. thanks
This was fixed in 2.10.5 - See milestone flag on the right side. Also Release Notes help in this

Yeah thanks for that @jscheffl I saw the release notes and it says it will work for teardown when the state is changed manually to failed

But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?

jscheffl · 2025-07-11T20:43:20Z

But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?

This sounds like a different use case. But you can test it and if you think it is a bug then please open a new issue for this as it seems to be unrelated. But please test with 2.11.0 or most recent 3.0

rakesh-shah · 2025-07-14T04:50:31Z

But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?
This sounds like a different use case. But you can test it and if you think it is a bug then please open a new issue for this as it seems to be unrelated. But please test with 2.11.0 or most recent 3.0

Sure @jscheffl sounds good will test it and revert

jscheffl added the AIP-52 Automatic setup and teardown tasks label Jan 9, 2025

jscheffl added this to the Airflow 2.10.5 milestone Jan 9, 2025

boring-cyborgBot added the area:API Airflow's REST/HTTP API label Jan 9, 2025

jscheffl added the backport-to-v2-10-test label Jan 9, 2025

jscheffl requested a review from dstandish January 9, 2025 22:28

jscheffl force-pushed the bugfix/ensure-teardown-is-executed-when-dag-set-failed branch from b9d8425 to 196b89cCompare January 9, 2025 23:08

jscheffl marked this pull request as ready for review January 9, 2025 23:11

jscheffl requested review from ephraimbuddy and pierrejeambrun as code owners January 9, 2025 23:11

shahar1 approved these changes Jan 10, 2025
View reviewed changes

jscheffl added 4 commits January 11, 2025 17:09

Ensure teardown tasks are executed when DAG run is set to failed
0a4f0f9

Also handle the case of setting DAG to success
5b30de2

Add some documentation to behavior changes
68edfad

Add some documentation to behavior changes
031d7af

jscheffl force-pushed the bugfix/ensure-teardown-is-executed-when-dag-set-failed branch from 196b89c to 031d7afCompare January 11, 2025 17:07

jscheffl requested a review from potiuk as a code owner January 11, 2025 17:07

jscheffl added the type:bug-fix Changelog: Bug Fixes label Jan 11, 2025

jscheffl merged commit 1e8977a into apache:mainJan 11, 2025

potiuk mentioned this pull request Jan 16, 2025
Update existing significant newsfragments with the later introduced template format #45678
Merged

JoostDuisters mentioned this pull request Feb 20, 2025
Marking DagRun as success with teardown task stopped working #46937
Closed
2 tasks

NewtonChutney mentioned this pull request Apr 5, 2025
on_failure_callback is not called when task is manually marked as failed #38935
Closed
2 tasks

jscheffl deleted the bugfix/ensure-teardown-is-executed-when-dag-set-failed branch October 5, 2025 07:43

Navigation Menu

Uh oh!

Ensure teardown tasks are executed when DAG run is set to failed - #45530

Ensure teardown tasks are executed when DAG run is set to failed#45530
jscheffl merged 4 commits into
apache:mainfrom
jscheffl:bugfix/ensure-teardown-is-executed-when-dag-set-failed

jscheffl commented Jan 9, 2025

Uh oh!

shahar1 left a comment

Uh oh!

jscheffl commented Jan 11, 2025

Uh oh!

jscheffl commented Jan 11, 2025

Uh oh!

github-actionsBot commented Jan 11, 2025

Uh oh!

Lee-W commented Jan 13, 2025

Uh oh!

potiuk commented Jan 14, 2025

Uh oh!

jscheffl commented Jan 14, 2025

Uh oh!

rakesh-shah commented Jul 11, 2025•
edited
Loading

Uh oh!

jscheffl commented Jul 11, 2025

Uh oh!

rakesh-shah commented Jul 11, 2025

Uh oh!

jscheffl commented Jul 11, 2025

Uh oh!

rakesh-shah commented Jul 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

jscheffl commented Jan 9, 2025

Uh oh!

shahar1 left a comment

Choose a reason for hiding this comment

Uh oh!

jscheffl commented Jan 11, 2025

Uh oh!

jscheffl commented Jan 11, 2025

Uh oh!

github-actionsBot commented Jan 11, 2025

Backport successfully created: v2-10-test

Uh oh!

Lee-W commented Jan 13, 2025

Uh oh!

potiuk commented Jan 14, 2025

Uh oh!

jscheffl commented Jan 14, 2025

Uh oh!

rakesh-shah commented Jul 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jscheffl commented Jul 11, 2025

Uh oh!

rakesh-shah commented Jul 11, 2025

Uh oh!

jscheffl commented Jul 11, 2025

Uh oh!

rakesh-shah commented Jul 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rakesh-shah commented Jul 11, 2025•
edited
Loading