Enforce execution_timeout in deferrable KubernetesPodOperator

[paultmathew]

Why + What

KubernetesPodOperator(deferrable=True) does not enforce execution_timeout. Once the operator defers, the synchronous execute() returns and the signal.alarm-based timeout context wrapping it exits cleanly — there is no further execution_timeout enforcement for the lifetime of the deferral. Pods continue running well past execution_timeout, bounded only by active_deadline_seconds (which defaults to ~1h or whatever the operator passed).

The framework gap is acknowledged by # TODO: handle timeout in case of deferral at task-sdk/.../task_runner.py:1782.

This PR fixes the symptom for KubernetesPodOperator, mirroring the pattern already merged for AirbyteTriggerSyncOperator (PR #64051) and DbtCloudRunJobOperator (PR #66449).

Approach

1. Operator (pod.py): in invoke_defer_method, translate execution_timeout into an absolute deadline anchored on ti.start_date:

   • execution_deadline = ti.start_date.timestamp() + execution_timeout.total_seconds()
   • Pass execution_deadline to KubernetesPodTrigger.
   • Pass timeout=remaining (timedelta) to self.defer() so the framework's trigger_timeout also bounds the trigger lifetime as a backstop.
   • Anchoring on ti.start_date keeps the deadline stable across re-deferrals (e.g. logging_interval re-entries), since Airflow preserves the original start_date when a task resumes from defer.
   • Re-pass context from trigger_reentry → invoke_defer_method so the deadline is recomputed correctly on each re-defer.

2. Trigger (pod.py): at the top of _wait_for_container_completion, check time.time() >= execution_deadline and emit a status="timeout" event when the deadline is crossed. The operator's existing trigger_reentry terminal-event path already handles status in ("error", "failed", "timeout", "success") — the operator fails the task and _clean() runs on_finish_action (default: delete pod).

Impact

• Existing behaviour preserved: execution_timeout was previously a no-op for deferred KPO tasks, and remains a no-op when not set. Tasks without execution_timeout see no behaviour change (execution_deadline=None, defer.timeout=None).
• No public API changes: the new execution_deadline parameter on KubernetesPodTrigger is keyword-only with a None default. Trigger serialization adds the field but defaults preserve back-compat for existing serialized triggers (the trigger's __init__ accepts the kwarg as optional).
• Pod cleanup: the existing on_finish_action path handles pod deletion (default delete_pod) when the operator fails on a timeout event. _clean() already special-cases event["status"] == "timeout" to skip await_pod_completion (the pod may hang on ErrImagePull/ContainerCreating).

Tests

• Trigger (tests/unit/cncf/kubernetes/triggers/test_pod.py):
  • Updated test_serialize to include the new execution_deadline key.
  • Added test_serialize_with_execution_deadline — round-trips a non-None deadline.
  • Added test_run_loop_emits_timeout_event_when_execution_deadline_reached — past-deadline → first iteration emits status="timeout" event.
  • Added test_run_loop_does_not_emit_timeout_when_execution_deadline_not_reached — far-future deadline → trigger keeps polling normally.
• Operator (tests/unit/cncf/kubernetes/operators/test_pod.py):
  • Added test_invoke_defer_method_passes_execution_deadline_when_execution_timeout_set — operator with execution_timeout=300s passes a deadline ≈ ti.start_date + 300s to the trigger; defer.timeout is set.
  • Added test_invoke_defer_method_passes_no_deadline_when_execution_timeout_not_set — operator without execution_timeout passes None (no enforcement, no behaviour change).

Backwards Compatibility

No public API changes. New execution_deadline parameter on KubernetesPodTrigger is optional with default None. Behaviour change: execution_timeout-equipped deferred KPO tasks now actually fail at the configured timeout instead of running indefinitely; this is the documented contract.

Closes

Closes: #67227

[jscheffl]

Thanks for the extension, looks good to me. Except some comments.

[Copilot]

Pull request overview

This PR enforces execution_timeout for KubernetesPodOperator(deferrable=True) by translating the timeout into an absolute deadline, passing it to KubernetesPodTrigger, and adding trigger-side logic to emit a terminal timeout event when the deadline is exceeded (with accompanying unit tests). It also updates the EKS-specific trigger subclass to forward the new parameter.

Changes:

• Add execution_deadline plumbing from KubernetesPodOperator.invoke_defer_method() to KubernetesPodTrigger and pass a timeout= to defer() based on remaining budget.
• Add trigger-side deadline enforcement that emits a status="timeout" event once the deadline is crossed.
• Extend/adjust unit tests for trigger serialization and timeout behavior, plus operator deferral plumbing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py	Compute an absolute execution deadline from ti.start_date and execution_timeout, pass it to the trigger, and set defer(timeout=…).
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py	Add execution_deadline to trigger init/serialization and emit a timeout TriggerEvent when the deadline is exceeded.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py	Add tests asserting the operator passes execution_deadline (or None) into the trigger and sets defer.timeout appropriately.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py	Update serialization expectations and add trigger run-loop tests for deadline timeout vs. continued polling.
providers/amazon/src/airflow/providers/amazon/aws/triggers/eks.py	Forward the new execution_deadline parameter through EksPodTrigger to the base Kubernetes trigger.

[paultmathew]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

[jscheffl]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

Can you please resolve the comments addressed? And in the ones not being addressed reply a comment?

[SameerMesiah97]

Looks fine overall. I have left some comments.

Also, I would urge you to verify your implementation using a live cluster to ensure that it fixes the issue. Passing a non-None value as an argument for the timeout parameter in the defer method resulted in the exact same bug remaining unresolved in the DBT Cloud and Airbyte providers. Arguably, your approach is more robust but it still needs live verification to be sure.

[paultmathew]

@jscheffl@SameerMesiah97 thank you both for the reviews.

Pulled the **kwargs revert and any new surface on KubernetesPodTrigger. The deadline now rides inside the existing trigger_kwargs dict under the reserved key _execution_deadline. The leading-underscore convention follows the _redefer_count precedent in the same file.

@SameerMesiah97 you were right — the previous 60s minimum clamp didn't account for poll_interval. With execution_timeout=30s, poll_interval=60s, the framework would have cancelled the trigger at T+60 before its next deadline check at T+65.

The latest commit guarantees defer.timeout always covers remaining + ≥ 2 poll cycles. New unit test pins it: test_invoke_defer_method_pads_defer_timeout_for_slow_poll_interval verifies defer.timeout = 30 + 120 = 150s for a poll_interval=60 setup. I also did a smoke test against a live Kubernetes cluster (EKS) in our sandbox environment.

Smoke test summary

Task	Configuration	Result	What it proved
t1 happy path	execution_timeout=5m, sleep 20s	✓ success at ~26s, pod deleted	Deferred path works end-to-end on real K8s; deadline doesn't trip prematurely
t2 timeout default poll	execution_timeout=30s, poll_interval=2s, sleep 600s	✓ failed at ~64s with "Execution deadline reached for pod ... emitting timeout event", pod deleted	New code path fires; _clean() runs; trigger emits soft-timeout event
t3 timeout slow poll	execution_timeout=30s, poll_interval=30s, sleep 600s	✓ failed at ~70s, timeout event at +38s, pod deleted	The reviewer's exact concern — slow poll doesn't break cleanup; new poll_buffer = max(60, poll_interval * 2) gives the trigger sufficient runway
t4 no timeout	no execution_timeout, sleep 20s	✓ success at ~26s, no deadline messages	Opt-out path preserved; tasks without execution_timeout continue to work as before

[jscheffl]

Looks good to me. @SameerMesiah97 another pass of review or any other maintainer feedback?

[SameerMesiah97]

Looks great. I must mention that your approach to add a buffer is a good compromise (which I will implement myself going forwards). Unfortunately, there is a gap at the framework level regarding the implementation of execution timeouts that has not been addressed as of yet so we must use temporary workarounds like this.

[paultmathew]

Why + What

KubernetesPodOperator(deferrable=True) does not enforce execution_timeout. Once the operator defers, the synchronous execute() returns and the signal.alarm-based timeout context wrapping it exits cleanly — there is no further execution_timeout enforcement for the lifetime of the deferral. Pods continue running well past execution_timeout, bounded only by active_deadline_seconds (which defaults to ~1h or whatever the operator passed).

The framework gap is acknowledged by # TODO: handle timeout in case of deferral at task-sdk/.../task_runner.py:1782.

This PR fixes the symptom for KubernetesPodOperator, mirroring the pattern already merged for AirbyteTriggerSyncOperator (PR #64051) and DbtCloudRunJobOperator (PR #66449).

Approach

1. Operator (pod.py): in invoke_defer_method, translate execution_timeout into an absolute deadline anchored on ti.start_date:

   • execution_deadline = ti.start_date.timestamp() + execution_timeout.total_seconds()
   • Pass execution_deadline to KubernetesPodTrigger.
   • Pass timeout=remaining (timedelta) to self.defer() so the framework's trigger_timeout also bounds the trigger lifetime as a backstop.
   • Anchoring on ti.start_date keeps the deadline stable across re-deferrals (e.g. logging_interval re-entries), since Airflow preserves the original start_date when a task resumes from defer.
   • Re-pass context from trigger_reentry → invoke_defer_method so the deadline is recomputed correctly on each re-defer.

2. Trigger (pod.py): at the top of _wait_for_container_completion, check time.time() >= execution_deadline and emit a status="timeout" event when the deadline is crossed. The operator's existing trigger_reentry terminal-event path already handles status in ("error", "failed", "timeout", "success") — the operator fails the task and _clean() runs on_finish_action (default: delete pod).

Impact

• Existing behaviour preserved: execution_timeout was previously a no-op for deferred KPO tasks, and remains a no-op when not set. Tasks without execution_timeout see no behaviour change (execution_deadline=None, defer.timeout=None).
• No public API changes: the new execution_deadline parameter on KubernetesPodTrigger is keyword-only with a None default. Trigger serialization adds the field but defaults preserve back-compat for existing serialized triggers (the trigger's __init__ accepts the kwarg as optional).
• Pod cleanup: the existing on_finish_action path handles pod deletion (default delete_pod) when the operator fails on a timeout event. _clean() already special-cases event["status"] == "timeout" to skip await_pod_completion (the pod may hang on ErrImagePull/ContainerCreating).

Tests

• Trigger (tests/unit/cncf/kubernetes/triggers/test_pod.py):
  • Updated test_serialize to include the new execution_deadline key.
  • Added test_serialize_with_execution_deadline — round-trips a non-None deadline.
  • Added test_run_loop_emits_timeout_event_when_execution_deadline_reached — past-deadline → first iteration emits status="timeout" event.
  • Added test_run_loop_does_not_emit_timeout_when_execution_deadline_not_reached — far-future deadline → trigger keeps polling normally.
• Operator (tests/unit/cncf/kubernetes/operators/test_pod.py):
  • Added test_invoke_defer_method_passes_execution_deadline_when_execution_timeout_set — operator with execution_timeout=300s passes a deadline ≈ ti.start_date + 300s to the trigger; defer.timeout is set.
  • Added test_invoke_defer_method_passes_no_deadline_when_execution_timeout_not_set — operator without execution_timeout passes None (no enforcement, no behaviour change).

Backwards Compatibility

No public API changes. New execution_deadline parameter on KubernetesPodTrigger is optional with default None. Behaviour change: execution_timeout-equipped deferred KPO tasks now actually fail at the configured timeout instead of running indefinitely; this is the documented contract.

Closes

Closes: #67227

[jscheffl]

Thanks for the extension, looks good to me. Except some comments.

[Copilot]

Pull request overview

This PR enforces execution_timeout for KubernetesPodOperator(deferrable=True) by translating the timeout into an absolute deadline, passing it to KubernetesPodTrigger, and adding trigger-side logic to emit a terminal timeout event when the deadline is exceeded (with accompanying unit tests). It also updates the EKS-specific trigger subclass to forward the new parameter.

Changes:

• Add execution_deadline plumbing from KubernetesPodOperator.invoke_defer_method() to KubernetesPodTrigger and pass a timeout= to defer() based on remaining budget.
• Add trigger-side deadline enforcement that emits a status="timeout" event once the deadline is crossed.
• Extend/adjust unit tests for trigger serialization and timeout behavior, plus operator deferral plumbing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py	Compute an absolute execution deadline from ti.start_date and execution_timeout, pass it to the trigger, and set defer(timeout=…).
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py	Add execution_deadline to trigger init/serialization and emit a timeout TriggerEvent when the deadline is exceeded.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py	Add tests asserting the operator passes execution_deadline (or None) into the trigger and sets defer.timeout appropriately.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py	Update serialization expectations and add trigger run-loop tests for deadline timeout vs. continued polling.
providers/amazon/src/airflow/providers/amazon/aws/triggers/eks.py	Forward the new execution_deadline parameter through EksPodTrigger to the base Kubernetes trigger.

[paultmathew]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

[jscheffl]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

Can you please resolve the comments addressed? And in the ones not being addressed reply a comment?

[SameerMesiah97]

Looks fine overall. I have left some comments.

Also, I would urge you to verify your implementation using a live cluster to ensure that it fixes the issue. Passing a non-None value as an argument for the timeout parameter in the defer method resulted in the exact same bug remaining unresolved in the DBT Cloud and Airbyte providers. Arguably, your approach is more robust but it still needs live verification to be sure.

[paultmathew]

@jscheffl@SameerMesiah97 thank you both for the reviews.

Pulled the **kwargs revert and any new surface on KubernetesPodTrigger. The deadline now rides inside the existing trigger_kwargs dict under the reserved key _execution_deadline. The leading-underscore convention follows the _redefer_count precedent in the same file.

@SameerMesiah97 you were right — the previous 60s minimum clamp didn't account for poll_interval. With execution_timeout=30s, poll_interval=60s, the framework would have cancelled the trigger at T+60 before its next deadline check at T+65.

The latest commit guarantees defer.timeout always covers remaining + ≥ 2 poll cycles. New unit test pins it: test_invoke_defer_method_pads_defer_timeout_for_slow_poll_interval verifies defer.timeout = 30 + 120 = 150s for a poll_interval=60 setup. I also did a smoke test against a live Kubernetes cluster (EKS) in our sandbox environment.

Smoke test summary

Task	Configuration	Result	What it proved
t1 happy path	execution_timeout=5m, sleep 20s	✓ success at ~26s, pod deleted	Deferred path works end-to-end on real K8s; deadline doesn't trip prematurely
t2 timeout default poll	execution_timeout=30s, poll_interval=2s, sleep 600s	✓ failed at ~64s with "Execution deadline reached for pod ... emitting timeout event", pod deleted	New code path fires; _clean() runs; trigger emits soft-timeout event
t3 timeout slow poll	execution_timeout=30s, poll_interval=30s, sleep 600s	✓ failed at ~70s, timeout event at +38s, pod deleted	The reviewer's exact concern — slow poll doesn't break cleanup; new poll_buffer = max(60, poll_interval * 2) gives the trigger sufficient runway
t4 no timeout	no execution_timeout, sleep 20s	✓ success at ~26s, no deadline messages	Opt-out path preserved; tasks without execution_timeout continue to work as before

[jscheffl]

Looks good to me. @SameerMesiah97 another pass of review or any other maintainer feedback?

[SameerMesiah97]

Looks great. I must mention that your approach to add a buffer is a good compromise (which I will implement myself going forwards). Unfortunately, there is a gap at the framework level regarding the implementation of execution timeouts that has not been addressed as of yet so we must use temporary workarounds like this.

[paultmathew]

Why + What

KubernetesPodOperator(deferrable=True) does not enforce execution_timeout. Once the operator defers, the synchronous execute() returns and the signal.alarm-based timeout context wrapping it exits cleanly — there is no further execution_timeout enforcement for the lifetime of the deferral. Pods continue running well past execution_timeout, bounded only by active_deadline_seconds (which defaults to ~1h or whatever the operator passed).

The framework gap is acknowledged by # TODO: handle timeout in case of deferral at task-sdk/.../task_runner.py:1782.

This PR fixes the symptom for KubernetesPodOperator, mirroring the pattern already merged for AirbyteTriggerSyncOperator (PR #64051) and DbtCloudRunJobOperator (PR #66449).

Approach

1. Operator (pod.py): in invoke_defer_method, translate execution_timeout into an absolute deadline anchored on ti.start_date:

   • execution_deadline = ti.start_date.timestamp() + execution_timeout.total_seconds()
   • Pass execution_deadline to KubernetesPodTrigger.
   • Pass timeout=remaining (timedelta) to self.defer() so the framework's trigger_timeout also bounds the trigger lifetime as a backstop.
   • Anchoring on ti.start_date keeps the deadline stable across re-deferrals (e.g. logging_interval re-entries), since Airflow preserves the original start_date when a task resumes from defer.
   • Re-pass context from trigger_reentry → invoke_defer_method so the deadline is recomputed correctly on each re-defer.

2. Trigger (pod.py): at the top of _wait_for_container_completion, check time.time() >= execution_deadline and emit a status="timeout" event when the deadline is crossed. The operator's existing trigger_reentry terminal-event path already handles status in ("error", "failed", "timeout", "success") — the operator fails the task and _clean() runs on_finish_action (default: delete pod).

Impact

• Existing behaviour preserved: execution_timeout was previously a no-op for deferred KPO tasks, and remains a no-op when not set. Tasks without execution_timeout see no behaviour change (execution_deadline=None, defer.timeout=None).
• No public API changes: the new execution_deadline parameter on KubernetesPodTrigger is keyword-only with a None default. Trigger serialization adds the field but defaults preserve back-compat for existing serialized triggers (the trigger's __init__ accepts the kwarg as optional).
• Pod cleanup: the existing on_finish_action path handles pod deletion (default delete_pod) when the operator fails on a timeout event. _clean() already special-cases event["status"] == "timeout" to skip await_pod_completion (the pod may hang on ErrImagePull/ContainerCreating).

Tests

• Trigger (tests/unit/cncf/kubernetes/triggers/test_pod.py):
  • Updated test_serialize to include the new execution_deadline key.
  • Added test_serialize_with_execution_deadline — round-trips a non-None deadline.
  • Added test_run_loop_emits_timeout_event_when_execution_deadline_reached — past-deadline → first iteration emits status="timeout" event.
  • Added test_run_loop_does_not_emit_timeout_when_execution_deadline_not_reached — far-future deadline → trigger keeps polling normally.
• Operator (tests/unit/cncf/kubernetes/operators/test_pod.py):
  • Added test_invoke_defer_method_passes_execution_deadline_when_execution_timeout_set — operator with execution_timeout=300s passes a deadline ≈ ti.start_date + 300s to the trigger; defer.timeout is set.
  • Added test_invoke_defer_method_passes_no_deadline_when_execution_timeout_not_set — operator without execution_timeout passes None (no enforcement, no behaviour change).

Backwards Compatibility

No public API changes. New execution_deadline parameter on KubernetesPodTrigger is optional with default None. Behaviour change: execution_timeout-equipped deferred KPO tasks now actually fail at the configured timeout instead of running indefinitely; this is the documented contract.

Closes

Closes: #67227

[jscheffl]

Thanks for the extension, looks good to me. Except some comments.

[Copilot]

Pull request overview

This PR enforces execution_timeout for KubernetesPodOperator(deferrable=True) by translating the timeout into an absolute deadline, passing it to KubernetesPodTrigger, and adding trigger-side logic to emit a terminal timeout event when the deadline is exceeded (with accompanying unit tests). It also updates the EKS-specific trigger subclass to forward the new parameter.

Changes:

• Add execution_deadline plumbing from KubernetesPodOperator.invoke_defer_method() to KubernetesPodTrigger and pass a timeout= to defer() based on remaining budget.
• Add trigger-side deadline enforcement that emits a status="timeout" event once the deadline is crossed.
• Extend/adjust unit tests for trigger serialization and timeout behavior, plus operator deferral plumbing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py	Compute an absolute execution deadline from ti.start_date and execution_timeout, pass it to the trigger, and set defer(timeout=…).
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py	Add execution_deadline to trigger init/serialization and emit a timeout TriggerEvent when the deadline is exceeded.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py	Add tests asserting the operator passes execution_deadline (or None) into the trigger and sets defer.timeout appropriately.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py	Update serialization expectations and add trigger run-loop tests for deadline timeout vs. continued polling.
providers/amazon/src/airflow/providers/amazon/aws/triggers/eks.py	Forward the new execution_deadline parameter through EksPodTrigger to the base Kubernetes trigger.

[paultmathew]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

[jscheffl]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

Can you please resolve the comments addressed? And in the ones not being addressed reply a comment?

[SameerMesiah97]

Looks fine overall. I have left some comments.

Also, I would urge you to verify your implementation using a live cluster to ensure that it fixes the issue. Passing a non-None value as an argument for the timeout parameter in the defer method resulted in the exact same bug remaining unresolved in the DBT Cloud and Airbyte providers. Arguably, your approach is more robust but it still needs live verification to be sure.

[paultmathew]

@jscheffl@SameerMesiah97 thank you both for the reviews.

Pulled the **kwargs revert and any new surface on KubernetesPodTrigger. The deadline now rides inside the existing trigger_kwargs dict under the reserved key _execution_deadline. The leading-underscore convention follows the _redefer_count precedent in the same file.

@SameerMesiah97 you were right — the previous 60s minimum clamp didn't account for poll_interval. With execution_timeout=30s, poll_interval=60s, the framework would have cancelled the trigger at T+60 before its next deadline check at T+65.

The latest commit guarantees defer.timeout always covers remaining + ≥ 2 poll cycles. New unit test pins it: test_invoke_defer_method_pads_defer_timeout_for_slow_poll_interval verifies defer.timeout = 30 + 120 = 150s for a poll_interval=60 setup. I also did a smoke test against a live Kubernetes cluster (EKS) in our sandbox environment.

Smoke test summary

Task	Configuration	Result	What it proved
t1 happy path	execution_timeout=5m, sleep 20s	✓ success at ~26s, pod deleted	Deferred path works end-to-end on real K8s; deadline doesn't trip prematurely
t2 timeout default poll	execution_timeout=30s, poll_interval=2s, sleep 600s	✓ failed at ~64s with "Execution deadline reached for pod ... emitting timeout event", pod deleted	New code path fires; _clean() runs; trigger emits soft-timeout event
t3 timeout slow poll	execution_timeout=30s, poll_interval=30s, sleep 600s	✓ failed at ~70s, timeout event at +38s, pod deleted	The reviewer's exact concern — slow poll doesn't break cleanup; new poll_buffer = max(60, poll_interval * 2) gives the trigger sufficient runway
t4 no timeout	no execution_timeout, sleep 20s	✓ success at ~26s, no deadline messages	Opt-out path preserved; tasks without execution_timeout continue to work as before

[jscheffl]

Looks good to me. @SameerMesiah97 another pass of review or any other maintainer feedback?

[SameerMesiah97]

Looks great. I must mention that your approach to add a buffer is a good compromise (which I will implement myself going forwards). Unfortunately, there is a gap at the framework level regarding the implementation of execution timeouts that has not been addressed as of yet so we must use temporary workarounds like this.

[paultmathew]

Why + What

KubernetesPodOperator(deferrable=True) does not enforce execution_timeout. Once the operator defers, the synchronous execute() returns and the signal.alarm-based timeout context wrapping it exits cleanly — there is no further execution_timeout enforcement for the lifetime of the deferral. Pods continue running well past execution_timeout, bounded only by active_deadline_seconds (which defaults to ~1h or whatever the operator passed).

The framework gap is acknowledged by # TODO: handle timeout in case of deferral at task-sdk/.../task_runner.py:1782.

This PR fixes the symptom for KubernetesPodOperator, mirroring the pattern already merged for AirbyteTriggerSyncOperator (PR #64051) and DbtCloudRunJobOperator (PR #66449).

Approach

1. Operator (pod.py): in invoke_defer_method, translate execution_timeout into an absolute deadline anchored on ti.start_date:

   • execution_deadline = ti.start_date.timestamp() + execution_timeout.total_seconds()
   • Pass execution_deadline to KubernetesPodTrigger.
   • Pass timeout=remaining (timedelta) to self.defer() so the framework's trigger_timeout also bounds the trigger lifetime as a backstop.
   • Anchoring on ti.start_date keeps the deadline stable across re-deferrals (e.g. logging_interval re-entries), since Airflow preserves the original start_date when a task resumes from defer.
   • Re-pass context from trigger_reentry → invoke_defer_method so the deadline is recomputed correctly on each re-defer.

2. Trigger (pod.py): at the top of _wait_for_container_completion, check time.time() >= execution_deadline and emit a status="timeout" event when the deadline is crossed. The operator's existing trigger_reentry terminal-event path already handles status in ("error", "failed", "timeout", "success") — the operator fails the task and _clean() runs on_finish_action (default: delete pod).

Impact

• Existing behaviour preserved: execution_timeout was previously a no-op for deferred KPO tasks, and remains a no-op when not set. Tasks without execution_timeout see no behaviour change (execution_deadline=None, defer.timeout=None).
• No public API changes: the new execution_deadline parameter on KubernetesPodTrigger is keyword-only with a None default. Trigger serialization adds the field but defaults preserve back-compat for existing serialized triggers (the trigger's __init__ accepts the kwarg as optional).
• Pod cleanup: the existing on_finish_action path handles pod deletion (default delete_pod) when the operator fails on a timeout event. _clean() already special-cases event["status"] == "timeout" to skip await_pod_completion (the pod may hang on ErrImagePull/ContainerCreating).

Tests

• Trigger (tests/unit/cncf/kubernetes/triggers/test_pod.py):
  • Updated test_serialize to include the new execution_deadline key.
  • Added test_serialize_with_execution_deadline — round-trips a non-None deadline.
  • Added test_run_loop_emits_timeout_event_when_execution_deadline_reached — past-deadline → first iteration emits status="timeout" event.
  • Added test_run_loop_does_not_emit_timeout_when_execution_deadline_not_reached — far-future deadline → trigger keeps polling normally.
• Operator (tests/unit/cncf/kubernetes/operators/test_pod.py):
  • Added test_invoke_defer_method_passes_execution_deadline_when_execution_timeout_set — operator with execution_timeout=300s passes a deadline ≈ ti.start_date + 300s to the trigger; defer.timeout is set.
  • Added test_invoke_defer_method_passes_no_deadline_when_execution_timeout_not_set — operator without execution_timeout passes None (no enforcement, no behaviour change).

Backwards Compatibility

No public API changes. New execution_deadline parameter on KubernetesPodTrigger is optional with default None. Behaviour change: execution_timeout-equipped deferred KPO tasks now actually fail at the configured timeout instead of running indefinitely; this is the documented contract.

Closes

Closes: #67227

[jscheffl]

Thanks for the extension, looks good to me. Except some comments.

[Copilot]

Pull request overview

This PR enforces execution_timeout for KubernetesPodOperator(deferrable=True) by translating the timeout into an absolute deadline, passing it to KubernetesPodTrigger, and adding trigger-side logic to emit a terminal timeout event when the deadline is exceeded (with accompanying unit tests). It also updates the EKS-specific trigger subclass to forward the new parameter.

Changes:

• Add execution_deadline plumbing from KubernetesPodOperator.invoke_defer_method() to KubernetesPodTrigger and pass a timeout= to defer() based on remaining budget.
• Add trigger-side deadline enforcement that emits a status="timeout" event once the deadline is crossed.
• Extend/adjust unit tests for trigger serialization and timeout behavior, plus operator deferral plumbing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py	Compute an absolute execution deadline from ti.start_date and execution_timeout, pass it to the trigger, and set defer(timeout=…).
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py	Add execution_deadline to trigger init/serialization and emit a timeout TriggerEvent when the deadline is exceeded.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py	Add tests asserting the operator passes execution_deadline (or None) into the trigger and sets defer.timeout appropriately.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py	Update serialization expectations and add trigger run-loop tests for deadline timeout vs. continued polling.
providers/amazon/src/airflow/providers/amazon/aws/triggers/eks.py	Forward the new execution_deadline parameter through EksPodTrigger to the base Kubernetes trigger.

[paultmathew]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

[jscheffl]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

Can you please resolve the comments addressed? And in the ones not being addressed reply a comment?

[SameerMesiah97]

Looks fine overall. I have left some comments.

Also, I would urge you to verify your implementation using a live cluster to ensure that it fixes the issue. Passing a non-None value as an argument for the timeout parameter in the defer method resulted in the exact same bug remaining unresolved in the DBT Cloud and Airbyte providers. Arguably, your approach is more robust but it still needs live verification to be sure.

[paultmathew]

@jscheffl@SameerMesiah97 thank you both for the reviews.

Pulled the **kwargs revert and any new surface on KubernetesPodTrigger. The deadline now rides inside the existing trigger_kwargs dict under the reserved key _execution_deadline. The leading-underscore convention follows the _redefer_count precedent in the same file.

@SameerMesiah97 you were right — the previous 60s minimum clamp didn't account for poll_interval. With execution_timeout=30s, poll_interval=60s, the framework would have cancelled the trigger at T+60 before its next deadline check at T+65.

The latest commit guarantees defer.timeout always covers remaining + ≥ 2 poll cycles. New unit test pins it: test_invoke_defer_method_pads_defer_timeout_for_slow_poll_interval verifies defer.timeout = 30 + 120 = 150s for a poll_interval=60 setup. I also did a smoke test against a live Kubernetes cluster (EKS) in our sandbox environment.

Smoke test summary

Task	Configuration	Result	What it proved
t1 happy path	execution_timeout=5m, sleep 20s	✓ success at ~26s, pod deleted	Deferred path works end-to-end on real K8s; deadline doesn't trip prematurely
t2 timeout default poll	execution_timeout=30s, poll_interval=2s, sleep 600s	✓ failed at ~64s with "Execution deadline reached for pod ... emitting timeout event", pod deleted	New code path fires; _clean() runs; trigger emits soft-timeout event
t3 timeout slow poll	execution_timeout=30s, poll_interval=30s, sleep 600s	✓ failed at ~70s, timeout event at +38s, pod deleted	The reviewer's exact concern — slow poll doesn't break cleanup; new poll_buffer = max(60, poll_interval * 2) gives the trigger sufficient runway
t4 no timeout	no execution_timeout, sleep 20s	✓ success at ~26s, no deadline messages	Opt-out path preserved; tasks without execution_timeout continue to work as before

[jscheffl]

Looks good to me. @SameerMesiah97 another pass of review or any other maintainer feedback?

[SameerMesiah97]

Looks great. I must mention that your approach to add a buffer is a good compromise (which I will implement myself going forwards). Unfortunately, there is a gap at the framework level regarding the implementation of execution timeouts that has not been addressed as of yet so we must use temporary workarounds like this.

[paultmathew]

Why + What

KubernetesPodOperator(deferrable=True) does not enforce execution_timeout. Once the operator defers, the synchronous execute() returns and the signal.alarm-based timeout context wrapping it exits cleanly — there is no further execution_timeout enforcement for the lifetime of the deferral. Pods continue running well past execution_timeout, bounded only by active_deadline_seconds (which defaults to ~1h or whatever the operator passed).

The framework gap is acknowledged by # TODO: handle timeout in case of deferral at task-sdk/.../task_runner.py:1782.

This PR fixes the symptom for KubernetesPodOperator, mirroring the pattern already merged for AirbyteTriggerSyncOperator (PR #64051) and DbtCloudRunJobOperator (PR #66449).

Approach

1. Operator (pod.py): in invoke_defer_method, translate execution_timeout into an absolute deadline anchored on ti.start_date:

   • execution_deadline = ti.start_date.timestamp() + execution_timeout.total_seconds()
   • Pass execution_deadline to KubernetesPodTrigger.
   • Pass timeout=remaining (timedelta) to self.defer() so the framework's trigger_timeout also bounds the trigger lifetime as a backstop.
   • Anchoring on ti.start_date keeps the deadline stable across re-deferrals (e.g. logging_interval re-entries), since Airflow preserves the original start_date when a task resumes from defer.
   • Re-pass context from trigger_reentry → invoke_defer_method so the deadline is recomputed correctly on each re-defer.

2. Trigger (pod.py): at the top of _wait_for_container_completion, check time.time() >= execution_deadline and emit a status="timeout" event when the deadline is crossed. The operator's existing trigger_reentry terminal-event path already handles status in ("error", "failed", "timeout", "success") — the operator fails the task and _clean() runs on_finish_action (default: delete pod).

Impact

• Existing behaviour preserved: execution_timeout was previously a no-op for deferred KPO tasks, and remains a no-op when not set. Tasks without execution_timeout see no behaviour change (execution_deadline=None, defer.timeout=None).
• No public API changes: the new execution_deadline parameter on KubernetesPodTrigger is keyword-only with a None default. Trigger serialization adds the field but defaults preserve back-compat for existing serialized triggers (the trigger's __init__ accepts the kwarg as optional).
• Pod cleanup: the existing on_finish_action path handles pod deletion (default delete_pod) when the operator fails on a timeout event. _clean() already special-cases event["status"] == "timeout" to skip await_pod_completion (the pod may hang on ErrImagePull/ContainerCreating).

Tests

• Trigger (tests/unit/cncf/kubernetes/triggers/test_pod.py):
  • Updated test_serialize to include the new execution_deadline key.
  • Added test_serialize_with_execution_deadline — round-trips a non-None deadline.
  • Added test_run_loop_emits_timeout_event_when_execution_deadline_reached — past-deadline → first iteration emits status="timeout" event.
  • Added test_run_loop_does_not_emit_timeout_when_execution_deadline_not_reached — far-future deadline → trigger keeps polling normally.
• Operator (tests/unit/cncf/kubernetes/operators/test_pod.py):
  • Added test_invoke_defer_method_passes_execution_deadline_when_execution_timeout_set — operator with execution_timeout=300s passes a deadline ≈ ti.start_date + 300s to the trigger; defer.timeout is set.
  • Added test_invoke_defer_method_passes_no_deadline_when_execution_timeout_not_set — operator without execution_timeout passes None (no enforcement, no behaviour change).

Backwards Compatibility

No public API changes. New execution_deadline parameter on KubernetesPodTrigger is optional with default None. Behaviour change: execution_timeout-equipped deferred KPO tasks now actually fail at the configured timeout instead of running indefinitely; this is the documented contract.

Closes

Closes: #67227

[jscheffl]

Thanks for the extension, looks good to me. Except some comments.

[Copilot]

Pull request overview

This PR enforces execution_timeout for KubernetesPodOperator(deferrable=True) by translating the timeout into an absolute deadline, passing it to KubernetesPodTrigger, and adding trigger-side logic to emit a terminal timeout event when the deadline is exceeded (with accompanying unit tests). It also updates the EKS-specific trigger subclass to forward the new parameter.

Changes:

• Add execution_deadline plumbing from KubernetesPodOperator.invoke_defer_method() to KubernetesPodTrigger and pass a timeout= to defer() based on remaining budget.
• Add trigger-side deadline enforcement that emits a status="timeout" event once the deadline is crossed.
• Extend/adjust unit tests for trigger serialization and timeout behavior, plus operator deferral plumbing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py	Compute an absolute execution deadline from ti.start_date and execution_timeout, pass it to the trigger, and set defer(timeout=…).
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py	Add execution_deadline to trigger init/serialization and emit a timeout TriggerEvent when the deadline is exceeded.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py	Add tests asserting the operator passes execution_deadline (or None) into the trigger and sets defer.timeout appropriately.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py	Update serialization expectations and add trigger run-loop tests for deadline timeout vs. continued polling.
providers/amazon/src/airflow/providers/amazon/aws/triggers/eks.py	Forward the new execution_deadline parameter through EksPodTrigger to the base Kubernetes trigger.

[paultmathew]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

[jscheffl]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

Can you please resolve the comments addressed? And in the ones not being addressed reply a comment?

[SameerMesiah97]

Looks fine overall. I have left some comments.

Also, I would urge you to verify your implementation using a live cluster to ensure that it fixes the issue. Passing a non-None value as an argument for the timeout parameter in the defer method resulted in the exact same bug remaining unresolved in the DBT Cloud and Airbyte providers. Arguably, your approach is more robust but it still needs live verification to be sure.

[paultmathew]

@jscheffl@SameerMesiah97 thank you both for the reviews.

Pulled the **kwargs revert and any new surface on KubernetesPodTrigger. The deadline now rides inside the existing trigger_kwargs dict under the reserved key _execution_deadline. The leading-underscore convention follows the _redefer_count precedent in the same file.

@SameerMesiah97 you were right — the previous 60s minimum clamp didn't account for poll_interval. With execution_timeout=30s, poll_interval=60s, the framework would have cancelled the trigger at T+60 before its next deadline check at T+65.

The latest commit guarantees defer.timeout always covers remaining + ≥ 2 poll cycles. New unit test pins it: test_invoke_defer_method_pads_defer_timeout_for_slow_poll_interval verifies defer.timeout = 30 + 120 = 150s for a poll_interval=60 setup. I also did a smoke test against a live Kubernetes cluster (EKS) in our sandbox environment.

Smoke test summary

Task	Configuration	Result	What it proved
t1 happy path	execution_timeout=5m, sleep 20s	✓ success at ~26s, pod deleted	Deferred path works end-to-end on real K8s; deadline doesn't trip prematurely
t2 timeout default poll	execution_timeout=30s, poll_interval=2s, sleep 600s	✓ failed at ~64s with "Execution deadline reached for pod ... emitting timeout event", pod deleted	New code path fires; _clean() runs; trigger emits soft-timeout event
t3 timeout slow poll	execution_timeout=30s, poll_interval=30s, sleep 600s	✓ failed at ~70s, timeout event at +38s, pod deleted	The reviewer's exact concern — slow poll doesn't break cleanup; new poll_buffer = max(60, poll_interval * 2) gives the trigger sufficient runway
t4 no timeout	no execution_timeout, sleep 20s	✓ success at ~26s, no deadline messages	Opt-out path preserved; tasks without execution_timeout continue to work as before

[jscheffl]

Looks good to me. @SameerMesiah97 another pass of review or any other maintainer feedback?

[SameerMesiah97]

Looks great. I must mention that your approach to add a buffer is a good compromise (which I will implement myself going forwards). Unfortunately, there is a gap at the framework level regarding the implementation of execution timeouts that has not been addressed as of yet so we must use temporary workarounds like this.

[paultmathew]

Why + What

KubernetesPodOperator(deferrable=True) does not enforce execution_timeout. Once the operator defers, the synchronous execute() returns and the signal.alarm-based timeout context wrapping it exits cleanly — there is no further execution_timeout enforcement for the lifetime of the deferral. Pods continue running well past execution_timeout, bounded only by active_deadline_seconds (which defaults to ~1h or whatever the operator passed).

The framework gap is acknowledged by # TODO: handle timeout in case of deferral at task-sdk/.../task_runner.py:1782.

This PR fixes the symptom for KubernetesPodOperator, mirroring the pattern already merged for AirbyteTriggerSyncOperator (PR #64051) and DbtCloudRunJobOperator (PR #66449).

Approach

1. Operator (pod.py): in invoke_defer_method, translate execution_timeout into an absolute deadline anchored on ti.start_date:

   • execution_deadline = ti.start_date.timestamp() + execution_timeout.total_seconds()
   • Pass execution_deadline to KubernetesPodTrigger.
   • Pass timeout=remaining (timedelta) to self.defer() so the framework's trigger_timeout also bounds the trigger lifetime as a backstop.
   • Anchoring on ti.start_date keeps the deadline stable across re-deferrals (e.g. logging_interval re-entries), since Airflow preserves the original start_date when a task resumes from defer.
   • Re-pass context from trigger_reentry → invoke_defer_method so the deadline is recomputed correctly on each re-defer.

2. Trigger (pod.py): at the top of _wait_for_container_completion, check time.time() >= execution_deadline and emit a status="timeout" event when the deadline is crossed. The operator's existing trigger_reentry terminal-event path already handles status in ("error", "failed", "timeout", "success") — the operator fails the task and _clean() runs on_finish_action (default: delete pod).

Impact

• Existing behaviour preserved: execution_timeout was previously a no-op for deferred KPO tasks, and remains a no-op when not set. Tasks without execution_timeout see no behaviour change (execution_deadline=None, defer.timeout=None).
• No public API changes: the new execution_deadline parameter on KubernetesPodTrigger is keyword-only with a None default. Trigger serialization adds the field but defaults preserve back-compat for existing serialized triggers (the trigger's __init__ accepts the kwarg as optional).
• Pod cleanup: the existing on_finish_action path handles pod deletion (default delete_pod) when the operator fails on a timeout event. _clean() already special-cases event["status"] == "timeout" to skip await_pod_completion (the pod may hang on ErrImagePull/ContainerCreating).

Tests

• Trigger (tests/unit/cncf/kubernetes/triggers/test_pod.py):
  • Updated test_serialize to include the new execution_deadline key.
  • Added test_serialize_with_execution_deadline — round-trips a non-None deadline.
  • Added test_run_loop_emits_timeout_event_when_execution_deadline_reached — past-deadline → first iteration emits status="timeout" event.
  • Added test_run_loop_does_not_emit_timeout_when_execution_deadline_not_reached — far-future deadline → trigger keeps polling normally.
• Operator (tests/unit/cncf/kubernetes/operators/test_pod.py):
  • Added test_invoke_defer_method_passes_execution_deadline_when_execution_timeout_set — operator with execution_timeout=300s passes a deadline ≈ ti.start_date + 300s to the trigger; defer.timeout is set.
  • Added test_invoke_defer_method_passes_no_deadline_when_execution_timeout_not_set — operator without execution_timeout passes None (no enforcement, no behaviour change).

Backwards Compatibility

No public API changes. New execution_deadline parameter on KubernetesPodTrigger is optional with default None. Behaviour change: execution_timeout-equipped deferred KPO tasks now actually fail at the configured timeout instead of running indefinitely; this is the documented contract.

Closes

Closes: #67227

[jscheffl]

Thanks for the extension, looks good to me. Except some comments.

[Copilot]

Pull request overview

This PR enforces execution_timeout for KubernetesPodOperator(deferrable=True) by translating the timeout into an absolute deadline, passing it to KubernetesPodTrigger, and adding trigger-side logic to emit a terminal timeout event when the deadline is exceeded (with accompanying unit tests). It also updates the EKS-specific trigger subclass to forward the new parameter.

Changes:

• Add execution_deadline plumbing from KubernetesPodOperator.invoke_defer_method() to KubernetesPodTrigger and pass a timeout= to defer() based on remaining budget.
• Add trigger-side deadline enforcement that emits a status="timeout" event once the deadline is crossed.
• Extend/adjust unit tests for trigger serialization and timeout behavior, plus operator deferral plumbing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py	Compute an absolute execution deadline from ti.start_date and execution_timeout, pass it to the trigger, and set defer(timeout=…).
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py	Add execution_deadline to trigger init/serialization and emit a timeout TriggerEvent when the deadline is exceeded.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py	Add tests asserting the operator passes execution_deadline (or None) into the trigger and sets defer.timeout appropriately.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py	Update serialization expectations and add trigger run-loop tests for deadline timeout vs. continued polling.
providers/amazon/src/airflow/providers/amazon/aws/triggers/eks.py	Forward the new execution_deadline parameter through EksPodTrigger to the base Kubernetes trigger.

[paultmathew]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

[jscheffl]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

Can you please resolve the comments addressed? And in the ones not being addressed reply a comment?

[SameerMesiah97]

Looks fine overall. I have left some comments.

Also, I would urge you to verify your implementation using a live cluster to ensure that it fixes the issue. Passing a non-None value as an argument for the timeout parameter in the defer method resulted in the exact same bug remaining unresolved in the DBT Cloud and Airbyte providers. Arguably, your approach is more robust but it still needs live verification to be sure.

[paultmathew]

@jscheffl@SameerMesiah97 thank you both for the reviews.

Pulled the **kwargs revert and any new surface on KubernetesPodTrigger. The deadline now rides inside the existing trigger_kwargs dict under the reserved key _execution_deadline. The leading-underscore convention follows the _redefer_count precedent in the same file.

@SameerMesiah97 you were right — the previous 60s minimum clamp didn't account for poll_interval. With execution_timeout=30s, poll_interval=60s, the framework would have cancelled the trigger at T+60 before its next deadline check at T+65.

The latest commit guarantees defer.timeout always covers remaining + ≥ 2 poll cycles. New unit test pins it: test_invoke_defer_method_pads_defer_timeout_for_slow_poll_interval verifies defer.timeout = 30 + 120 = 150s for a poll_interval=60 setup. I also did a smoke test against a live Kubernetes cluster (EKS) in our sandbox environment.

Smoke test summary

Task	Configuration	Result	What it proved
t1 happy path	execution_timeout=5m, sleep 20s	✓ success at ~26s, pod deleted	Deferred path works end-to-end on real K8s; deadline doesn't trip prematurely
t2 timeout default poll	execution_timeout=30s, poll_interval=2s, sleep 600s	✓ failed at ~64s with "Execution deadline reached for pod ... emitting timeout event", pod deleted	New code path fires; _clean() runs; trigger emits soft-timeout event
t3 timeout slow poll	execution_timeout=30s, poll_interval=30s, sleep 600s	✓ failed at ~70s, timeout event at +38s, pod deleted	The reviewer's exact concern — slow poll doesn't break cleanup; new poll_buffer = max(60, poll_interval * 2) gives the trigger sufficient runway
t4 no timeout	no execution_timeout, sleep 20s	✓ success at ~26s, no deadline messages	Opt-out path preserved; tasks without execution_timeout continue to work as before

[jscheffl]

Looks good to me. @SameerMesiah97 another pass of review or any other maintainer feedback?

[SameerMesiah97]

Looks great. I must mention that your approach to add a buffer is a good compromise (which I will implement myself going forwards). Unfortunately, there is a gap at the framework level regarding the implementation of execution timeouts that has not been addressed as of yet so we must use temporary workarounds like this.

[paultmathew]

Why + What

KubernetesPodOperator(deferrable=True) does not enforce execution_timeout. Once the operator defers, the synchronous execute() returns and the signal.alarm-based timeout context wrapping it exits cleanly — there is no further execution_timeout enforcement for the lifetime of the deferral. Pods continue running well past execution_timeout, bounded only by active_deadline_seconds (which defaults to ~1h or whatever the operator passed).

The framework gap is acknowledged by # TODO: handle timeout in case of deferral at task-sdk/.../task_runner.py:1782.

This PR fixes the symptom for KubernetesPodOperator, mirroring the pattern already merged for AirbyteTriggerSyncOperator (PR #64051) and DbtCloudRunJobOperator (PR #66449).

Approach

1. Operator (pod.py): in invoke_defer_method, translate execution_timeout into an absolute deadline anchored on ti.start_date:

   • execution_deadline = ti.start_date.timestamp() + execution_timeout.total_seconds()
   • Pass execution_deadline to KubernetesPodTrigger.
   • Pass timeout=remaining (timedelta) to self.defer() so the framework's trigger_timeout also bounds the trigger lifetime as a backstop.
   • Anchoring on ti.start_date keeps the deadline stable across re-deferrals (e.g. logging_interval re-entries), since Airflow preserves the original start_date when a task resumes from defer.
   • Re-pass context from trigger_reentry → invoke_defer_method so the deadline is recomputed correctly on each re-defer.

2. Trigger (pod.py): at the top of _wait_for_container_completion, check time.time() >= execution_deadline and emit a status="timeout" event when the deadline is crossed. The operator's existing trigger_reentry terminal-event path already handles status in ("error", "failed", "timeout", "success") — the operator fails the task and _clean() runs on_finish_action (default: delete pod).

Impact

• Existing behaviour preserved: execution_timeout was previously a no-op for deferred KPO tasks, and remains a no-op when not set. Tasks without execution_timeout see no behaviour change (execution_deadline=None, defer.timeout=None).
• No public API changes: the new execution_deadline parameter on KubernetesPodTrigger is keyword-only with a None default. Trigger serialization adds the field but defaults preserve back-compat for existing serialized triggers (the trigger's __init__ accepts the kwarg as optional).
• Pod cleanup: the existing on_finish_action path handles pod deletion (default delete_pod) when the operator fails on a timeout event. _clean() already special-cases event["status"] == "timeout" to skip await_pod_completion (the pod may hang on ErrImagePull/ContainerCreating).

Tests

• Trigger (tests/unit/cncf/kubernetes/triggers/test_pod.py):
  • Updated test_serialize to include the new execution_deadline key.
  • Added test_serialize_with_execution_deadline — round-trips a non-None deadline.
  • Added test_run_loop_emits_timeout_event_when_execution_deadline_reached — past-deadline → first iteration emits status="timeout" event.
  • Added test_run_loop_does_not_emit_timeout_when_execution_deadline_not_reached — far-future deadline → trigger keeps polling normally.
• Operator (tests/unit/cncf/kubernetes/operators/test_pod.py):
  • Added test_invoke_defer_method_passes_execution_deadline_when_execution_timeout_set — operator with execution_timeout=300s passes a deadline ≈ ti.start_date + 300s to the trigger; defer.timeout is set.
  • Added test_invoke_defer_method_passes_no_deadline_when_execution_timeout_not_set — operator without execution_timeout passes None (no enforcement, no behaviour change).

Backwards Compatibility

No public API changes. New execution_deadline parameter on KubernetesPodTrigger is optional with default None. Behaviour change: execution_timeout-equipped deferred KPO tasks now actually fail at the configured timeout instead of running indefinitely; this is the documented contract.

Closes

Closes: #67227

[jscheffl]

Thanks for the extension, looks good to me. Except some comments.

[Copilot]

Pull request overview

This PR enforces execution_timeout for KubernetesPodOperator(deferrable=True) by translating the timeout into an absolute deadline, passing it to KubernetesPodTrigger, and adding trigger-side logic to emit a terminal timeout event when the deadline is exceeded (with accompanying unit tests). It also updates the EKS-specific trigger subclass to forward the new parameter.

Changes:

• Add execution_deadline plumbing from KubernetesPodOperator.invoke_defer_method() to KubernetesPodTrigger and pass a timeout= to defer() based on remaining budget.
• Add trigger-side deadline enforcement that emits a status="timeout" event once the deadline is crossed.
• Extend/adjust unit tests for trigger serialization and timeout behavior, plus operator deferral plumbing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py	Compute an absolute execution deadline from ti.start_date and execution_timeout, pass it to the trigger, and set defer(timeout=…).
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py	Add execution_deadline to trigger init/serialization and emit a timeout TriggerEvent when the deadline is exceeded.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py	Add tests asserting the operator passes execution_deadline (or None) into the trigger and sets defer.timeout appropriately.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py	Update serialization expectations and add trigger run-loop tests for deadline timeout vs. continued polling.
providers/amazon/src/airflow/providers/amazon/aws/triggers/eks.py	Forward the new execution_deadline parameter through EksPodTrigger to the base Kubernetes trigger.

[paultmathew]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

[jscheffl]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

Can you please resolve the comments addressed? And in the ones not being addressed reply a comment?

[SameerMesiah97]

Looks fine overall. I have left some comments.

Also, I would urge you to verify your implementation using a live cluster to ensure that it fixes the issue. Passing a non-None value as an argument for the timeout parameter in the defer method resulted in the exact same bug remaining unresolved in the DBT Cloud and Airbyte providers. Arguably, your approach is more robust but it still needs live verification to be sure.

[paultmathew]

@jscheffl@SameerMesiah97 thank you both for the reviews.

Pulled the **kwargs revert and any new surface on KubernetesPodTrigger. The deadline now rides inside the existing trigger_kwargs dict under the reserved key _execution_deadline. The leading-underscore convention follows the _redefer_count precedent in the same file.

@SameerMesiah97 you were right — the previous 60s minimum clamp didn't account for poll_interval. With execution_timeout=30s, poll_interval=60s, the framework would have cancelled the trigger at T+60 before its next deadline check at T+65.

The latest commit guarantees defer.timeout always covers remaining + ≥ 2 poll cycles. New unit test pins it: test_invoke_defer_method_pads_defer_timeout_for_slow_poll_interval verifies defer.timeout = 30 + 120 = 150s for a poll_interval=60 setup. I also did a smoke test against a live Kubernetes cluster (EKS) in our sandbox environment.

Smoke test summary

Task	Configuration	Result	What it proved
t1 happy path	execution_timeout=5m, sleep 20s	✓ success at ~26s, pod deleted	Deferred path works end-to-end on real K8s; deadline doesn't trip prematurely
t2 timeout default poll	execution_timeout=30s, poll_interval=2s, sleep 600s	✓ failed at ~64s with "Execution deadline reached for pod ... emitting timeout event", pod deleted	New code path fires; _clean() runs; trigger emits soft-timeout event
t3 timeout slow poll	execution_timeout=30s, poll_interval=30s, sleep 600s	✓ failed at ~70s, timeout event at +38s, pod deleted	The reviewer's exact concern — slow poll doesn't break cleanup; new poll_buffer = max(60, poll_interval * 2) gives the trigger sufficient runway
t4 no timeout	no execution_timeout, sleep 20s	✓ success at ~26s, no deadline messages	Opt-out path preserved; tasks without execution_timeout continue to work as before

[jscheffl]

Looks good to me. @SameerMesiah97 another pass of review or any other maintainer feedback?

[SameerMesiah97]

Looks great. I must mention that your approach to add a buffer is a good compromise (which I will implement myself going forwards). Unfortunately, there is a gap at the framework level regarding the implementation of execution timeouts that has not been addressed as of yet so we must use temporary workarounds like this.

[paultmathew]

Why + What

KubernetesPodOperator(deferrable=True) does not enforce execution_timeout. Once the operator defers, the synchronous execute() returns and the signal.alarm-based timeout context wrapping it exits cleanly — there is no further execution_timeout enforcement for the lifetime of the deferral. Pods continue running well past execution_timeout, bounded only by active_deadline_seconds (which defaults to ~1h or whatever the operator passed).

The framework gap is acknowledged by # TODO: handle timeout in case of deferral at task-sdk/.../task_runner.py:1782.

This PR fixes the symptom for KubernetesPodOperator, mirroring the pattern already merged for AirbyteTriggerSyncOperator (PR #64051) and DbtCloudRunJobOperator (PR #66449).

Approach

1. Operator (pod.py): in invoke_defer_method, translate execution_timeout into an absolute deadline anchored on ti.start_date:

   • execution_deadline = ti.start_date.timestamp() + execution_timeout.total_seconds()
   • Pass execution_deadline to KubernetesPodTrigger.
   • Pass timeout=remaining (timedelta) to self.defer() so the framework's trigger_timeout also bounds the trigger lifetime as a backstop.
   • Anchoring on ti.start_date keeps the deadline stable across re-deferrals (e.g. logging_interval re-entries), since Airflow preserves the original start_date when a task resumes from defer.
   • Re-pass context from trigger_reentry → invoke_defer_method so the deadline is recomputed correctly on each re-defer.

2. Trigger (pod.py): at the top of _wait_for_container_completion, check time.time() >= execution_deadline and emit a status="timeout" event when the deadline is crossed. The operator's existing trigger_reentry terminal-event path already handles status in ("error", "failed", "timeout", "success") — the operator fails the task and _clean() runs on_finish_action (default: delete pod).

Impact

• Existing behaviour preserved: execution_timeout was previously a no-op for deferred KPO tasks, and remains a no-op when not set. Tasks without execution_timeout see no behaviour change (execution_deadline=None, defer.timeout=None).
• No public API changes: the new execution_deadline parameter on KubernetesPodTrigger is keyword-only with a None default. Trigger serialization adds the field but defaults preserve back-compat for existing serialized triggers (the trigger's __init__ accepts the kwarg as optional).
• Pod cleanup: the existing on_finish_action path handles pod deletion (default delete_pod) when the operator fails on a timeout event. _clean() already special-cases event["status"] == "timeout" to skip await_pod_completion (the pod may hang on ErrImagePull/ContainerCreating).

Tests

• Trigger (tests/unit/cncf/kubernetes/triggers/test_pod.py):
  • Updated test_serialize to include the new execution_deadline key.
  • Added test_serialize_with_execution_deadline — round-trips a non-None deadline.
  • Added test_run_loop_emits_timeout_event_when_execution_deadline_reached — past-deadline → first iteration emits status="timeout" event.
  • Added test_run_loop_does_not_emit_timeout_when_execution_deadline_not_reached — far-future deadline → trigger keeps polling normally.
• Operator (tests/unit/cncf/kubernetes/operators/test_pod.py):
  • Added test_invoke_defer_method_passes_execution_deadline_when_execution_timeout_set — operator with execution_timeout=300s passes a deadline ≈ ti.start_date + 300s to the trigger; defer.timeout is set.
  • Added test_invoke_defer_method_passes_no_deadline_when_execution_timeout_not_set — operator without execution_timeout passes None (no enforcement, no behaviour change).

Backwards Compatibility

No public API changes. New execution_deadline parameter on KubernetesPodTrigger is optional with default None. Behaviour change: execution_timeout-equipped deferred KPO tasks now actually fail at the configured timeout instead of running indefinitely; this is the documented contract.

Closes

Closes: #67227

[jscheffl]

Thanks for the extension, looks good to me. Except some comments.

[Copilot]

Pull request overview

This PR enforces execution_timeout for KubernetesPodOperator(deferrable=True) by translating the timeout into an absolute deadline, passing it to KubernetesPodTrigger, and adding trigger-side logic to emit a terminal timeout event when the deadline is exceeded (with accompanying unit tests). It also updates the EKS-specific trigger subclass to forward the new parameter.

Changes:

• Add execution_deadline plumbing from KubernetesPodOperator.invoke_defer_method() to KubernetesPodTrigger and pass a timeout= to defer() based on remaining budget.
• Add trigger-side deadline enforcement that emits a status="timeout" event once the deadline is crossed.
• Extend/adjust unit tests for trigger serialization and timeout behavior, plus operator deferral plumbing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py	Compute an absolute execution deadline from ti.start_date and execution_timeout, pass it to the trigger, and set defer(timeout=…).
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py	Add execution_deadline to trigger init/serialization and emit a timeout TriggerEvent when the deadline is exceeded.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py	Add tests asserting the operator passes execution_deadline (or None) into the trigger and sets defer.timeout appropriately.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py	Update serialization expectations and add trigger run-loop tests for deadline timeout vs. continued polling.
providers/amazon/src/airflow/providers/amazon/aws/triggers/eks.py	Forward the new execution_deadline parameter through EksPodTrigger to the base Kubernetes trigger.

[paultmathew]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

[jscheffl]

@jscheffl Thanks for the review. I pushed a change and addressed the comments.

Can you please resolve the comments addressed? And in the ones not being addressed reply a comment?

[SameerMesiah97]

Looks fine overall. I have left some comments.

Also, I would urge you to verify your implementation using a live cluster to ensure that it fixes the issue. Passing a non-None value as an argument for the timeout parameter in the defer method resulted in the exact same bug remaining unresolved in the DBT Cloud and Airbyte providers. Arguably, your approach is more robust but it still needs live verification to be sure.

[paultmathew]

@jscheffl@SameerMesiah97 thank you both for the reviews.

Pulled the **kwargs revert and any new surface on KubernetesPodTrigger. The deadline now rides inside the existing trigger_kwargs dict under the reserved key _execution_deadline. The leading-underscore convention follows the _redefer_count precedent in the same file.

@SameerMesiah97 you were right — the previous 60s minimum clamp didn't account for poll_interval. With execution_timeout=30s, poll_interval=60s, the framework would have cancelled the trigger at T+60 before its next deadline check at T+65.

The latest commit guarantees defer.timeout always covers remaining + ≥ 2 poll cycles. New unit test pins it: test_invoke_defer_method_pads_defer_timeout_for_slow_poll_interval verifies defer.timeout = 30 + 120 = 150s for a poll_interval=60 setup. I also did a smoke test against a live Kubernetes cluster (EKS) in our sandbox environment.

Smoke test summary

Task	Configuration	Result	What it proved
t1 happy path	execution_timeout=5m, sleep 20s	✓ success at ~26s, pod deleted	Deferred path works end-to-end on real K8s; deadline doesn't trip prematurely
t2 timeout default poll	execution_timeout=30s, poll_interval=2s, sleep 600s	✓ failed at ~64s with "Execution deadline reached for pod ... emitting timeout event", pod deleted	New code path fires; _clean() runs; trigger emits soft-timeout event
t3 timeout slow poll	execution_timeout=30s, poll_interval=30s, sleep 600s	✓ failed at ~70s, timeout event at +38s, pod deleted	The reviewer's exact concern — slow poll doesn't break cleanup; new poll_buffer = max(60, poll_interval * 2) gives the trigger sufficient runway
t4 no timeout	no execution_timeout, sleep 20s	✓ success at ~26s, no deadline messages	Opt-out path preserved; tasks without execution_timeout continue to work as before

[jscheffl]

Looks good to me. @SameerMesiah97 another pass of review or any other maintainer feedback?

[SameerMesiah97]

Looks great. I must mention that your approach to add a buffer is a good compromise (which I will implement myself going forwards). Unfortunately, there is a gap at the framework level regarding the implementation of execution timeouts that has not been addressed as of yet so we must use temporary workarounds like this.