Fix EMR container job not cancelled on deferral timeout

[aurangzaib048]

When EmrContainerOperator runs in deferrable mode and the trigger times out or the task is killed, the EMR job keeps running on the cluster. This leads to orphaned jobs consuming resources and duplicate executions on retry.

This PR adds cancel-on-kill support to EmrContainerTrigger following the same proven pattern as EMrServerlessStartJobTrigger (PR #51883):

• Override run() in EmrContainerTrigger to catch asyncio.CancelledError and cancel the EMR job via hook.stop_query() when safe to do so
• Add safe_to_cancel() check to distinguish user-initiated kills from triggerer restarts (avoids cancelling jobs during triggerer restart)
• Add cancel_on_kill parameter (default True) for opt-out
• Update EmrContainerOperator.execute_complete() to cancel the job when the trigger reports a failure/timeout event
• All cancellation paths are wrapped in try/except to ensure proper error propagation (CancelledError is always re-raised, original AirflowException is preserved)

closes: #60517

---

Was generative AI tooling used to co-author this PR?

• Yes

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes orphaned EMR on EKS container jobs when deferrable tasks time out or are killed by adding cancel-on-cancel semantics to the trigger and ensuring the operator cancels the job on terminal error events.

Changes:

• Add cancel_on_kill support to EmrContainerTrigger, including a safe_to_cancel() check and explicit CancelledError handling to stop the remote job.
• Update EmrContainerOperator.execute_complete() to stop the EMR job when the trigger reports failure/timeout.
• Add/extend unit tests to cover trigger serialization and cancellation behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
providers/amazon/src/airflow/providers/amazon/aws/triggers/emr.py	Adds cancel-on-kill behavior and safe cancellation checks in EmrContainerTrigger.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py	Cancels EMR container jobs in execute_complete() on non-success events; passes cancel_on_kill=True to trigger.
providers/amazon/tests/unit/amazon/aws/triggers/test_emr.py	Adds serialization and cancellation-path unit tests for EmrContainerTrigger.
providers/amazon/tests/unit/amazon/aws/operators/test_emr_containers.py	Adds unit tests verifying job cancellation on execute_complete() failure.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[vincbeck]

I understand the goal here but I am also wondering if it is worth it. You're adding a lot of code just for that use case, which creates significant complexity in the operator. By any chance, is there any way to make it simpler?

[potiuk]

@aurangzaib048 Thanks for working through all the review feedback — every thread on the PR is resolved. This PR has a small issue that needs to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally, fix whatever it flags, and push.

What to do next:

• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Once CI goes green, the PR is ready for re-review.

Leaving the PR open (not converting to draft) since you have clearly worked through the earlier reviewer feedback — all 20 review threads are resolved. Just this one CI fix left. There is no rush — take your time.

[vincbeck]

Something we might want to leverage? #65590

[o-nikolas]

Something we might want to leverage? #65590

Yes, let's definitely make use of this if we can here!

[potiuk]

@aurangzaib048 — Your unresolved review thread(s) from @o-nikolas, @vincbeck appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@o-nikolas, @vincbeck — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @aurangzaib048 will follow up.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Still waiting this comment to be addressed: https://github.com/apache/airflow/pull/64770/changes#r3124747034

[aurangzaib048]

Adopted the BaseTrigger.on_kill() pattern from #65590, this removed safe_to_cancel / get_task_state / get_task_instance from EmrContainerTrigger entirely.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[aurangzaib048]

When EmrContainerOperator runs in deferrable mode and the trigger times out or the task is killed, the EMR job keeps running on the cluster. This leads to orphaned jobs consuming resources and duplicate executions on retry.

This PR adds cancel-on-kill support to EmrContainerTrigger following the same proven pattern as EMrServerlessStartJobTrigger (PR #51883):

• Override run() in EmrContainerTrigger to catch asyncio.CancelledError and cancel the EMR job via hook.stop_query() when safe to do so
• Add safe_to_cancel() check to distinguish user-initiated kills from triggerer restarts (avoids cancelling jobs during triggerer restart)
• Add cancel_on_kill parameter (default True) for opt-out
• Update EmrContainerOperator.execute_complete() to cancel the job when the trigger reports a failure/timeout event
• All cancellation paths are wrapped in try/except to ensure proper error propagation (CancelledError is always re-raised, original AirflowException is preserved)

closes: #60517

---

Was generative AI tooling used to co-author this PR?

• Yes

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes orphaned EMR on EKS container jobs when deferrable tasks time out or are killed by adding cancel-on-cancel semantics to the trigger and ensuring the operator cancels the job on terminal error events.

Changes:

• Add cancel_on_kill support to EmrContainerTrigger, including a safe_to_cancel() check and explicit CancelledError handling to stop the remote job.
• Update EmrContainerOperator.execute_complete() to stop the EMR job when the trigger reports failure/timeout.
• Add/extend unit tests to cover trigger serialization and cancellation behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
providers/amazon/src/airflow/providers/amazon/aws/triggers/emr.py	Adds cancel-on-kill behavior and safe cancellation checks in EmrContainerTrigger.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py	Cancels EMR container jobs in execute_complete() on non-success events; passes cancel_on_kill=True to trigger.
providers/amazon/tests/unit/amazon/aws/triggers/test_emr.py	Adds serialization and cancellation-path unit tests for EmrContainerTrigger.
providers/amazon/tests/unit/amazon/aws/operators/test_emr_containers.py	Adds unit tests verifying job cancellation on execute_complete() failure.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[vincbeck]

I understand the goal here but I am also wondering if it is worth it. You're adding a lot of code just for that use case, which creates significant complexity in the operator. By any chance, is there any way to make it simpler?

[potiuk]

@aurangzaib048 Thanks for working through all the review feedback — every thread on the PR is resolved. This PR has a small issue that needs to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally, fix whatever it flags, and push.

What to do next:

• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Once CI goes green, the PR is ready for re-review.

Leaving the PR open (not converting to draft) since you have clearly worked through the earlier reviewer feedback — all 20 review threads are resolved. Just this one CI fix left. There is no rush — take your time.

[vincbeck]

Something we might want to leverage? #65590

[o-nikolas]

Something we might want to leverage? #65590

Yes, let's definitely make use of this if we can here!

[potiuk]

@aurangzaib048 — Your unresolved review thread(s) from @o-nikolas, @vincbeck appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@o-nikolas, @vincbeck — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @aurangzaib048 will follow up.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Still waiting this comment to be addressed: https://github.com/apache/airflow/pull/64770/changes#r3124747034

[aurangzaib048]

Adopted the BaseTrigger.on_kill() pattern from #65590, this removed safe_to_cancel / get_task_state / get_task_instance from EmrContainerTrigger entirely.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[aurangzaib048]

When EmrContainerOperator runs in deferrable mode and the trigger times out or the task is killed, the EMR job keeps running on the cluster. This leads to orphaned jobs consuming resources and duplicate executions on retry.

This PR adds cancel-on-kill support to EmrContainerTrigger following the same proven pattern as EMrServerlessStartJobTrigger (PR #51883):

• Override run() in EmrContainerTrigger to catch asyncio.CancelledError and cancel the EMR job via hook.stop_query() when safe to do so
• Add safe_to_cancel() check to distinguish user-initiated kills from triggerer restarts (avoids cancelling jobs during triggerer restart)
• Add cancel_on_kill parameter (default True) for opt-out
• Update EmrContainerOperator.execute_complete() to cancel the job when the trigger reports a failure/timeout event
• All cancellation paths are wrapped in try/except to ensure proper error propagation (CancelledError is always re-raised, original AirflowException is preserved)

closes: #60517

---

Was generative AI tooling used to co-author this PR?

• Yes

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes orphaned EMR on EKS container jobs when deferrable tasks time out or are killed by adding cancel-on-cancel semantics to the trigger and ensuring the operator cancels the job on terminal error events.

Changes:

• Add cancel_on_kill support to EmrContainerTrigger, including a safe_to_cancel() check and explicit CancelledError handling to stop the remote job.
• Update EmrContainerOperator.execute_complete() to stop the EMR job when the trigger reports failure/timeout.
• Add/extend unit tests to cover trigger serialization and cancellation behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
providers/amazon/src/airflow/providers/amazon/aws/triggers/emr.py	Adds cancel-on-kill behavior and safe cancellation checks in EmrContainerTrigger.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py	Cancels EMR container jobs in execute_complete() on non-success events; passes cancel_on_kill=True to trigger.
providers/amazon/tests/unit/amazon/aws/triggers/test_emr.py	Adds serialization and cancellation-path unit tests for EmrContainerTrigger.
providers/amazon/tests/unit/amazon/aws/operators/test_emr_containers.py	Adds unit tests verifying job cancellation on execute_complete() failure.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[vincbeck]

I understand the goal here but I am also wondering if it is worth it. You're adding a lot of code just for that use case, which creates significant complexity in the operator. By any chance, is there any way to make it simpler?

[potiuk]

@aurangzaib048 Thanks for working through all the review feedback — every thread on the PR is resolved. This PR has a small issue that needs to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally, fix whatever it flags, and push.

What to do next:

• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Once CI goes green, the PR is ready for re-review.

Leaving the PR open (not converting to draft) since you have clearly worked through the earlier reviewer feedback — all 20 review threads are resolved. Just this one CI fix left. There is no rush — take your time.

[vincbeck]

Something we might want to leverage? #65590

[o-nikolas]

Something we might want to leverage? #65590

Yes, let's definitely make use of this if we can here!

[potiuk]

@aurangzaib048 — Your unresolved review thread(s) from @o-nikolas, @vincbeck appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@o-nikolas, @vincbeck — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @aurangzaib048 will follow up.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Still waiting this comment to be addressed: https://github.com/apache/airflow/pull/64770/changes#r3124747034

[aurangzaib048]

Adopted the BaseTrigger.on_kill() pattern from #65590, this removed safe_to_cancel / get_task_state / get_task_instance from EmrContainerTrigger entirely.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[aurangzaib048]

When EmrContainerOperator runs in deferrable mode and the trigger times out or the task is killed, the EMR job keeps running on the cluster. This leads to orphaned jobs consuming resources and duplicate executions on retry.

This PR adds cancel-on-kill support to EmrContainerTrigger following the same proven pattern as EMrServerlessStartJobTrigger (PR #51883):

• Override run() in EmrContainerTrigger to catch asyncio.CancelledError and cancel the EMR job via hook.stop_query() when safe to do so
• Add safe_to_cancel() check to distinguish user-initiated kills from triggerer restarts (avoids cancelling jobs during triggerer restart)
• Add cancel_on_kill parameter (default True) for opt-out
• Update EmrContainerOperator.execute_complete() to cancel the job when the trigger reports a failure/timeout event
• All cancellation paths are wrapped in try/except to ensure proper error propagation (CancelledError is always re-raised, original AirflowException is preserved)

closes: #60517

---

Was generative AI tooling used to co-author this PR?

• Yes

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes orphaned EMR on EKS container jobs when deferrable tasks time out or are killed by adding cancel-on-cancel semantics to the trigger and ensuring the operator cancels the job on terminal error events.

Changes:

• Add cancel_on_kill support to EmrContainerTrigger, including a safe_to_cancel() check and explicit CancelledError handling to stop the remote job.
• Update EmrContainerOperator.execute_complete() to stop the EMR job when the trigger reports failure/timeout.
• Add/extend unit tests to cover trigger serialization and cancellation behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
providers/amazon/src/airflow/providers/amazon/aws/triggers/emr.py	Adds cancel-on-kill behavior and safe cancellation checks in EmrContainerTrigger.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py	Cancels EMR container jobs in execute_complete() on non-success events; passes cancel_on_kill=True to trigger.
providers/amazon/tests/unit/amazon/aws/triggers/test_emr.py	Adds serialization and cancellation-path unit tests for EmrContainerTrigger.
providers/amazon/tests/unit/amazon/aws/operators/test_emr_containers.py	Adds unit tests verifying job cancellation on execute_complete() failure.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[vincbeck]

I understand the goal here but I am also wondering if it is worth it. You're adding a lot of code just for that use case, which creates significant complexity in the operator. By any chance, is there any way to make it simpler?

[potiuk]

@aurangzaib048 Thanks for working through all the review feedback — every thread on the PR is resolved. This PR has a small issue that needs to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally, fix whatever it flags, and push.

What to do next:

• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Once CI goes green, the PR is ready for re-review.

Leaving the PR open (not converting to draft) since you have clearly worked through the earlier reviewer feedback — all 20 review threads are resolved. Just this one CI fix left. There is no rush — take your time.

[vincbeck]

Something we might want to leverage? #65590

[o-nikolas]

Something we might want to leverage? #65590

Yes, let's definitely make use of this if we can here!

[potiuk]

@aurangzaib048 — Your unresolved review thread(s) from @o-nikolas, @vincbeck appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@o-nikolas, @vincbeck — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @aurangzaib048 will follow up.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Still waiting this comment to be addressed: https://github.com/apache/airflow/pull/64770/changes#r3124747034

[aurangzaib048]

Adopted the BaseTrigger.on_kill() pattern from #65590, this removed safe_to_cancel / get_task_state / get_task_instance from EmrContainerTrigger entirely.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[aurangzaib048]

When EmrContainerOperator runs in deferrable mode and the trigger times out or the task is killed, the EMR job keeps running on the cluster. This leads to orphaned jobs consuming resources and duplicate executions on retry.

This PR adds cancel-on-kill support to EmrContainerTrigger following the same proven pattern as EMrServerlessStartJobTrigger (PR #51883):

• Override run() in EmrContainerTrigger to catch asyncio.CancelledError and cancel the EMR job via hook.stop_query() when safe to do so
• Add safe_to_cancel() check to distinguish user-initiated kills from triggerer restarts (avoids cancelling jobs during triggerer restart)
• Add cancel_on_kill parameter (default True) for opt-out
• Update EmrContainerOperator.execute_complete() to cancel the job when the trigger reports a failure/timeout event
• All cancellation paths are wrapped in try/except to ensure proper error propagation (CancelledError is always re-raised, original AirflowException is preserved)

closes: #60517

---

Was generative AI tooling used to co-author this PR?

• Yes

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes orphaned EMR on EKS container jobs when deferrable tasks time out or are killed by adding cancel-on-cancel semantics to the trigger and ensuring the operator cancels the job on terminal error events.

Changes:

• Add cancel_on_kill support to EmrContainerTrigger, including a safe_to_cancel() check and explicit CancelledError handling to stop the remote job.
• Update EmrContainerOperator.execute_complete() to stop the EMR job when the trigger reports failure/timeout.
• Add/extend unit tests to cover trigger serialization and cancellation behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
providers/amazon/src/airflow/providers/amazon/aws/triggers/emr.py	Adds cancel-on-kill behavior and safe cancellation checks in EmrContainerTrigger.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py	Cancels EMR container jobs in execute_complete() on non-success events; passes cancel_on_kill=True to trigger.
providers/amazon/tests/unit/amazon/aws/triggers/test_emr.py	Adds serialization and cancellation-path unit tests for EmrContainerTrigger.
providers/amazon/tests/unit/amazon/aws/operators/test_emr_containers.py	Adds unit tests verifying job cancellation on execute_complete() failure.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[vincbeck]

I understand the goal here but I am also wondering if it is worth it. You're adding a lot of code just for that use case, which creates significant complexity in the operator. By any chance, is there any way to make it simpler?

[potiuk]

@aurangzaib048 Thanks for working through all the review feedback — every thread on the PR is resolved. This PR has a small issue that needs to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally, fix whatever it flags, and push.

What to do next:

• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Once CI goes green, the PR is ready for re-review.

Leaving the PR open (not converting to draft) since you have clearly worked through the earlier reviewer feedback — all 20 review threads are resolved. Just this one CI fix left. There is no rush — take your time.

[vincbeck]

Something we might want to leverage? #65590

[o-nikolas]

Something we might want to leverage? #65590

Yes, let's definitely make use of this if we can here!

[potiuk]

@aurangzaib048 — Your unresolved review thread(s) from @o-nikolas, @vincbeck appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@o-nikolas, @vincbeck — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @aurangzaib048 will follow up.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Still waiting this comment to be addressed: https://github.com/apache/airflow/pull/64770/changes#r3124747034

[aurangzaib048]

Adopted the BaseTrigger.on_kill() pattern from #65590, this removed safe_to_cancel / get_task_state / get_task_instance from EmrContainerTrigger entirely.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[aurangzaib048]

When EmrContainerOperator runs in deferrable mode and the trigger times out or the task is killed, the EMR job keeps running on the cluster. This leads to orphaned jobs consuming resources and duplicate executions on retry.

This PR adds cancel-on-kill support to EmrContainerTrigger following the same proven pattern as EMrServerlessStartJobTrigger (PR #51883):

• Override run() in EmrContainerTrigger to catch asyncio.CancelledError and cancel the EMR job via hook.stop_query() when safe to do so
• Add safe_to_cancel() check to distinguish user-initiated kills from triggerer restarts (avoids cancelling jobs during triggerer restart)
• Add cancel_on_kill parameter (default True) for opt-out
• Update EmrContainerOperator.execute_complete() to cancel the job when the trigger reports a failure/timeout event
• All cancellation paths are wrapped in try/except to ensure proper error propagation (CancelledError is always re-raised, original AirflowException is preserved)

closes: #60517

---

Was generative AI tooling used to co-author this PR?

• Yes

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes orphaned EMR on EKS container jobs when deferrable tasks time out or are killed by adding cancel-on-cancel semantics to the trigger and ensuring the operator cancels the job on terminal error events.

Changes:

• Add cancel_on_kill support to EmrContainerTrigger, including a safe_to_cancel() check and explicit CancelledError handling to stop the remote job.
• Update EmrContainerOperator.execute_complete() to stop the EMR job when the trigger reports failure/timeout.
• Add/extend unit tests to cover trigger serialization and cancellation behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
providers/amazon/src/airflow/providers/amazon/aws/triggers/emr.py	Adds cancel-on-kill behavior and safe cancellation checks in EmrContainerTrigger.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py	Cancels EMR container jobs in execute_complete() on non-success events; passes cancel_on_kill=True to trigger.
providers/amazon/tests/unit/amazon/aws/triggers/test_emr.py	Adds serialization and cancellation-path unit tests for EmrContainerTrigger.
providers/amazon/tests/unit/amazon/aws/operators/test_emr_containers.py	Adds unit tests verifying job cancellation on execute_complete() failure.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[vincbeck]

I understand the goal here but I am also wondering if it is worth it. You're adding a lot of code just for that use case, which creates significant complexity in the operator. By any chance, is there any way to make it simpler?

[potiuk]

@aurangzaib048 Thanks for working through all the review feedback — every thread on the PR is resolved. This PR has a small issue that needs to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally, fix whatever it flags, and push.

What to do next:

• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Once CI goes green, the PR is ready for re-review.

Leaving the PR open (not converting to draft) since you have clearly worked through the earlier reviewer feedback — all 20 review threads are resolved. Just this one CI fix left. There is no rush — take your time.

[vincbeck]

Something we might want to leverage? #65590

[o-nikolas]

Something we might want to leverage? #65590

Yes, let's definitely make use of this if we can here!

[potiuk]

@aurangzaib048 — Your unresolved review thread(s) from @o-nikolas, @vincbeck appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@o-nikolas, @vincbeck — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @aurangzaib048 will follow up.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Still waiting this comment to be addressed: https://github.com/apache/airflow/pull/64770/changes#r3124747034

[aurangzaib048]

Adopted the BaseTrigger.on_kill() pattern from #65590, this removed safe_to_cancel / get_task_state / get_task_instance from EmrContainerTrigger entirely.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[aurangzaib048]

When EmrContainerOperator runs in deferrable mode and the trigger times out or the task is killed, the EMR job keeps running on the cluster. This leads to orphaned jobs consuming resources and duplicate executions on retry.

This PR adds cancel-on-kill support to EmrContainerTrigger following the same proven pattern as EMrServerlessStartJobTrigger (PR #51883):

• Override run() in EmrContainerTrigger to catch asyncio.CancelledError and cancel the EMR job via hook.stop_query() when safe to do so
• Add safe_to_cancel() check to distinguish user-initiated kills from triggerer restarts (avoids cancelling jobs during triggerer restart)
• Add cancel_on_kill parameter (default True) for opt-out
• Update EmrContainerOperator.execute_complete() to cancel the job when the trigger reports a failure/timeout event
• All cancellation paths are wrapped in try/except to ensure proper error propagation (CancelledError is always re-raised, original AirflowException is preserved)

closes: #60517

---

Was generative AI tooling used to co-author this PR?

• Yes

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes orphaned EMR on EKS container jobs when deferrable tasks time out or are killed by adding cancel-on-cancel semantics to the trigger and ensuring the operator cancels the job on terminal error events.

Changes:

• Add cancel_on_kill support to EmrContainerTrigger, including a safe_to_cancel() check and explicit CancelledError handling to stop the remote job.
• Update EmrContainerOperator.execute_complete() to stop the EMR job when the trigger reports failure/timeout.
• Add/extend unit tests to cover trigger serialization and cancellation behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
providers/amazon/src/airflow/providers/amazon/aws/triggers/emr.py	Adds cancel-on-kill behavior and safe cancellation checks in EmrContainerTrigger.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py	Cancels EMR container jobs in execute_complete() on non-success events; passes cancel_on_kill=True to trigger.
providers/amazon/tests/unit/amazon/aws/triggers/test_emr.py	Adds serialization and cancellation-path unit tests for EmrContainerTrigger.
providers/amazon/tests/unit/amazon/aws/operators/test_emr_containers.py	Adds unit tests verifying job cancellation on execute_complete() failure.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[vincbeck]

I understand the goal here but I am also wondering if it is worth it. You're adding a lot of code just for that use case, which creates significant complexity in the operator. By any chance, is there any way to make it simpler?

[potiuk]

@aurangzaib048 Thanks for working through all the review feedback — every thread on the PR is resolved. This PR has a small issue that needs to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally, fix whatever it flags, and push.

What to do next:

• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Once CI goes green, the PR is ready for re-review.

Leaving the PR open (not converting to draft) since you have clearly worked through the earlier reviewer feedback — all 20 review threads are resolved. Just this one CI fix left. There is no rush — take your time.

[vincbeck]

Something we might want to leverage? #65590

[o-nikolas]

Something we might want to leverage? #65590

Yes, let's definitely make use of this if we can here!

[potiuk]

@aurangzaib048 — Your unresolved review thread(s) from @o-nikolas, @vincbeck appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@o-nikolas, @vincbeck — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @aurangzaib048 will follow up.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Still waiting this comment to be addressed: https://github.com/apache/airflow/pull/64770/changes#r3124747034

[aurangzaib048]

Adopted the BaseTrigger.on_kill() pattern from #65590, this removed safe_to_cancel / get_task_state / get_task_instance from EmrContainerTrigger entirely.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[aurangzaib048]

When EmrContainerOperator runs in deferrable mode and the trigger times out or the task is killed, the EMR job keeps running on the cluster. This leads to orphaned jobs consuming resources and duplicate executions on retry.

This PR adds cancel-on-kill support to EmrContainerTrigger following the same proven pattern as EMrServerlessStartJobTrigger (PR #51883):

• Override run() in EmrContainerTrigger to catch asyncio.CancelledError and cancel the EMR job via hook.stop_query() when safe to do so
• Add safe_to_cancel() check to distinguish user-initiated kills from triggerer restarts (avoids cancelling jobs during triggerer restart)
• Add cancel_on_kill parameter (default True) for opt-out
• Update EmrContainerOperator.execute_complete() to cancel the job when the trigger reports a failure/timeout event
• All cancellation paths are wrapped in try/except to ensure proper error propagation (CancelledError is always re-raised, original AirflowException is preserved)

closes: #60517

---

Was generative AI tooling used to co-author this PR?

• Yes

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes orphaned EMR on EKS container jobs when deferrable tasks time out or are killed by adding cancel-on-cancel semantics to the trigger and ensuring the operator cancels the job on terminal error events.

Changes:

• Add cancel_on_kill support to EmrContainerTrigger, including a safe_to_cancel() check and explicit CancelledError handling to stop the remote job.
• Update EmrContainerOperator.execute_complete() to stop the EMR job when the trigger reports failure/timeout.
• Add/extend unit tests to cover trigger serialization and cancellation behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
providers/amazon/src/airflow/providers/amazon/aws/triggers/emr.py	Adds cancel-on-kill behavior and safe cancellation checks in EmrContainerTrigger.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py	Cancels EMR container jobs in execute_complete() on non-success events; passes cancel_on_kill=True to trigger.
providers/amazon/tests/unit/amazon/aws/triggers/test_emr.py	Adds serialization and cancellation-path unit tests for EmrContainerTrigger.
providers/amazon/tests/unit/amazon/aws/operators/test_emr_containers.py	Adds unit tests verifying job cancellation on execute_complete() failure.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[vincbeck]

I understand the goal here but I am also wondering if it is worth it. You're adding a lot of code just for that use case, which creates significant complexity in the operator. By any chance, is there any way to make it simpler?

[potiuk]

@aurangzaib048 Thanks for working through all the review feedback — every thread on the PR is resolved. This PR has a small issue that needs to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally, fix whatever it flags, and push.

What to do next:

• Make sure static checks pass locally: prek run --from-ref main --stage pre-commit.
• Once CI goes green, the PR is ready for re-review.

Leaving the PR open (not converting to draft) since you have clearly worked through the earlier reviewer feedback — all 20 review threads are resolved. Just this one CI fix left. There is no rush — take your time.

[vincbeck]

Something we might want to leverage? #65590

[o-nikolas]

Something we might want to leverage? #65590

Yes, let's definitely make use of this if we can here!

[potiuk]

@aurangzaib048 — Your unresolved review thread(s) from @o-nikolas, @vincbeck appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@o-nikolas, @vincbeck — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @aurangzaib048 will follow up.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Still waiting this comment to be addressed: https://github.com/apache/airflow/pull/64770/changes#r3124747034

[aurangzaib048]

Adopted the BaseTrigger.on_kill() pattern from #65590, this removed safe_to_cancel / get_task_state / get_task_instance from EmrContainerTrigger entirely.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.