Skip to content

Fail closed when supervisor IPC fails on a non-success terminal state - #66573

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state
May 19, 2026
Merged

Fail closed when supervisor IPC fails on a non-success terminal state#66573
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

When a task FAILED / SKIPPED / etc. and the IPC send of the terminal-state message to the supervisor itself raised, the existing finally block logged the exception and let run() return normally. The task subprocess then exited with code 0, which the supervisor's final_state property maps to SUCCESS for an exit_code == 0 process without a _terminal_state (the supervisor never received the message).

A genuine task FAILURE was silently being upgraded to SUCCESS on transient IPC failures, breaking downstream pipeline correctness without any signal.

Fix

Exit non-zero from the finally block when the terminal state is anything other than SUCCESS, so the supervisor's final_state correctly classifies as FAILED (or UP_FOR_RETRY when retries are configured).

The SUCCESS exemption preserves the existing softening for the legitimate scenario where the supervisor rejects the terminal-state send with a 409 because the server already terminalised the TI — covered by test_run_swallows_supervisor_terminal_send_failure, which continues to pass.

Test

New regression test: when the task fails and the supervisor IPC send raises (BrokenPipeError simulating a dead Unix socket), run() raises SystemExit(1).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-006).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from 607e582 to 2acf7f0CompareMay 17, 2026 19:33
@potiuk

Copy link
Copy Markdown
MemberAuthor

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with nit

Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's a good idea to add the early exit for this case.

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@potiuk can you address comments?

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
potiukand others added 3 commits May 19, 2026 13:27
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from be3187f to d1a32abCompareMay 19, 2026 11:55
@vatsrahul1001
vatsrahul1001 merged commit 7e91517 into apache:mainMay 19, 2026
113 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-2-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 19, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@vatsrahul1001@Lee-W@kaxil@jason810496
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Fail closed when supervisor IPC fails on a non-success terminal state by potiuk · Pull Request #66573 · apache/airflow · GitHub
Skip to content

Fail closed when supervisor IPC fails on a non-success terminal state - #66573

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state
May 19, 2026
Merged

Fail closed when supervisor IPC fails on a non-success terminal state#66573
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

When a task FAILED / SKIPPED / etc. and the IPC send of the terminal-state message to the supervisor itself raised, the existing finally block logged the exception and let run() return normally. The task subprocess then exited with code 0, which the supervisor's final_state property maps to SUCCESS for an exit_code == 0 process without a _terminal_state (the supervisor never received the message).

A genuine task FAILURE was silently being upgraded to SUCCESS on transient IPC failures, breaking downstream pipeline correctness without any signal.

Fix

Exit non-zero from the finally block when the terminal state is anything other than SUCCESS, so the supervisor's final_state correctly classifies as FAILED (or UP_FOR_RETRY when retries are configured).

The SUCCESS exemption preserves the existing softening for the legitimate scenario where the supervisor rejects the terminal-state send with a 409 because the server already terminalised the TI — covered by test_run_swallows_supervisor_terminal_send_failure, which continues to pass.

Test

New regression test: when the task fails and the supervisor IPC send raises (BrokenPipeError simulating a dead Unix socket), run() raises SystemExit(1).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-006).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from 607e582 to 2acf7f0CompareMay 17, 2026 19:33
@potiuk

Copy link
Copy Markdown
MemberAuthor

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with nit

Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's a good idea to add the early exit for this case.

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@potiuk can you address comments?

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
potiukand others added 3 commits May 19, 2026 13:27
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from be3187f to d1a32abCompareMay 19, 2026 11:55
@vatsrahul1001
vatsrahul1001 merged commit 7e91517 into apache:mainMay 19, 2026
113 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-2-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 19, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@vatsrahul1001@Lee-W@kaxil@jason810496
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Fail closed when supervisor IPC fails on a non-success terminal state by potiuk · Pull Request #66573 · apache/airflow · GitHub
Skip to content

Fail closed when supervisor IPC fails on a non-success terminal state - #66573

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state
May 19, 2026
Merged

Fail closed when supervisor IPC fails on a non-success terminal state#66573
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

When a task FAILED / SKIPPED / etc. and the IPC send of the terminal-state message to the supervisor itself raised, the existing finally block logged the exception and let run() return normally. The task subprocess then exited with code 0, which the supervisor's final_state property maps to SUCCESS for an exit_code == 0 process without a _terminal_state (the supervisor never received the message).

A genuine task FAILURE was silently being upgraded to SUCCESS on transient IPC failures, breaking downstream pipeline correctness without any signal.

Fix

Exit non-zero from the finally block when the terminal state is anything other than SUCCESS, so the supervisor's final_state correctly classifies as FAILED (or UP_FOR_RETRY when retries are configured).

The SUCCESS exemption preserves the existing softening for the legitimate scenario where the supervisor rejects the terminal-state send with a 409 because the server already terminalised the TI — covered by test_run_swallows_supervisor_terminal_send_failure, which continues to pass.

Test

New regression test: when the task fails and the supervisor IPC send raises (BrokenPipeError simulating a dead Unix socket), run() raises SystemExit(1).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-006).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from 607e582 to 2acf7f0CompareMay 17, 2026 19:33
@potiuk

Copy link
Copy Markdown
MemberAuthor

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with nit

Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's a good idea to add the early exit for this case.

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@potiuk can you address comments?

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
potiukand others added 3 commits May 19, 2026 13:27
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from be3187f to d1a32abCompareMay 19, 2026 11:55
@vatsrahul1001
vatsrahul1001 merged commit 7e91517 into apache:mainMay 19, 2026
113 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-2-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 19, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@vatsrahul1001@Lee-W@kaxil@jason810496
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Fail closed when supervisor IPC fails on a non-success terminal state by potiuk · Pull Request #66573 · apache/airflow · GitHub
Skip to content

Fail closed when supervisor IPC fails on a non-success terminal state - #66573

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state
May 19, 2026
Merged

Fail closed when supervisor IPC fails on a non-success terminal state#66573
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

When a task FAILED / SKIPPED / etc. and the IPC send of the terminal-state message to the supervisor itself raised, the existing finally block logged the exception and let run() return normally. The task subprocess then exited with code 0, which the supervisor's final_state property maps to SUCCESS for an exit_code == 0 process without a _terminal_state (the supervisor never received the message).

A genuine task FAILURE was silently being upgraded to SUCCESS on transient IPC failures, breaking downstream pipeline correctness without any signal.

Fix

Exit non-zero from the finally block when the terminal state is anything other than SUCCESS, so the supervisor's final_state correctly classifies as FAILED (or UP_FOR_RETRY when retries are configured).

The SUCCESS exemption preserves the existing softening for the legitimate scenario where the supervisor rejects the terminal-state send with a 409 because the server already terminalised the TI — covered by test_run_swallows_supervisor_terminal_send_failure, which continues to pass.

Test

New regression test: when the task fails and the supervisor IPC send raises (BrokenPipeError simulating a dead Unix socket), run() raises SystemExit(1).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-006).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from 607e582 to 2acf7f0CompareMay 17, 2026 19:33
@potiuk

Copy link
Copy Markdown
MemberAuthor

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with nit

Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's a good idea to add the early exit for this case.

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@potiuk can you address comments?

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
potiukand others added 3 commits May 19, 2026 13:27
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from be3187f to d1a32abCompareMay 19, 2026 11:55
@vatsrahul1001
vatsrahul1001 merged commit 7e91517 into apache:mainMay 19, 2026
113 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-2-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 19, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@vatsrahul1001@Lee-W@kaxil@jason810496
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Fail closed when supervisor IPC fails on a non-success terminal state by potiuk · Pull Request #66573 · apache/airflow · GitHub
Skip to content

Fail closed when supervisor IPC fails on a non-success terminal state - #66573

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state
May 19, 2026
Merged

Fail closed when supervisor IPC fails on a non-success terminal state#66573
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

When a task FAILED / SKIPPED / etc. and the IPC send of the terminal-state message to the supervisor itself raised, the existing finally block logged the exception and let run() return normally. The task subprocess then exited with code 0, which the supervisor's final_state property maps to SUCCESS for an exit_code == 0 process without a _terminal_state (the supervisor never received the message).

A genuine task FAILURE was silently being upgraded to SUCCESS on transient IPC failures, breaking downstream pipeline correctness without any signal.

Fix

Exit non-zero from the finally block when the terminal state is anything other than SUCCESS, so the supervisor's final_state correctly classifies as FAILED (or UP_FOR_RETRY when retries are configured).

The SUCCESS exemption preserves the existing softening for the legitimate scenario where the supervisor rejects the terminal-state send with a 409 because the server already terminalised the TI — covered by test_run_swallows_supervisor_terminal_send_failure, which continues to pass.

Test

New regression test: when the task fails and the supervisor IPC send raises (BrokenPipeError simulating a dead Unix socket), run() raises SystemExit(1).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-006).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from 607e582 to 2acf7f0CompareMay 17, 2026 19:33
@potiuk

Copy link
Copy Markdown
MemberAuthor

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with nit

Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's a good idea to add the early exit for this case.

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@potiuk can you address comments?

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
potiukand others added 3 commits May 19, 2026 13:27
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from be3187f to d1a32abCompareMay 19, 2026 11:55
@vatsrahul1001
vatsrahul1001 merged commit 7e91517 into apache:mainMay 19, 2026
113 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-2-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 19, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@vatsrahul1001@Lee-W@kaxil@jason810496
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Fail closed when supervisor IPC fails on a non-success terminal state by potiuk · Pull Request #66573 · apache/airflow · GitHub
Skip to content

Fail closed when supervisor IPC fails on a non-success terminal state - #66573

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state
May 19, 2026
Merged

Fail closed when supervisor IPC fails on a non-success terminal state#66573
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

When a task FAILED / SKIPPED / etc. and the IPC send of the terminal-state message to the supervisor itself raised, the existing finally block logged the exception and let run() return normally. The task subprocess then exited with code 0, which the supervisor's final_state property maps to SUCCESS for an exit_code == 0 process without a _terminal_state (the supervisor never received the message).

A genuine task FAILURE was silently being upgraded to SUCCESS on transient IPC failures, breaking downstream pipeline correctness without any signal.

Fix

Exit non-zero from the finally block when the terminal state is anything other than SUCCESS, so the supervisor's final_state correctly classifies as FAILED (or UP_FOR_RETRY when retries are configured).

The SUCCESS exemption preserves the existing softening for the legitimate scenario where the supervisor rejects the terminal-state send with a 409 because the server already terminalised the TI — covered by test_run_swallows_supervisor_terminal_send_failure, which continues to pass.

Test

New regression test: when the task fails and the supervisor IPC send raises (BrokenPipeError simulating a dead Unix socket), run() raises SystemExit(1).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-006).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from 607e582 to 2acf7f0CompareMay 17, 2026 19:33
@potiuk

Copy link
Copy Markdown
MemberAuthor

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with nit

Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's a good idea to add the early exit for this case.

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@potiuk can you address comments?

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
potiukand others added 3 commits May 19, 2026 13:27
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from be3187f to d1a32abCompareMay 19, 2026 11:55
@vatsrahul1001
vatsrahul1001 merged commit 7e91517 into apache:mainMay 19, 2026
113 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-2-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 19, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@vatsrahul1001@Lee-W@kaxil@jason810496
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Fail closed when supervisor IPC fails on a non-success terminal state by potiuk · Pull Request #66573 · apache/airflow · GitHub
Skip to content

Fail closed when supervisor IPC fails on a non-success terminal state - #66573

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state
May 19, 2026
Merged

Fail closed when supervisor IPC fails on a non-success terminal state#66573
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

When a task FAILED / SKIPPED / etc. and the IPC send of the terminal-state message to the supervisor itself raised, the existing finally block logged the exception and let run() return normally. The task subprocess then exited with code 0, which the supervisor's final_state property maps to SUCCESS for an exit_code == 0 process without a _terminal_state (the supervisor never received the message).

A genuine task FAILURE was silently being upgraded to SUCCESS on transient IPC failures, breaking downstream pipeline correctness without any signal.

Fix

Exit non-zero from the finally block when the terminal state is anything other than SUCCESS, so the supervisor's final_state correctly classifies as FAILED (or UP_FOR_RETRY when retries are configured).

The SUCCESS exemption preserves the existing softening for the legitimate scenario where the supervisor rejects the terminal-state send with a 409 because the server already terminalised the TI — covered by test_run_swallows_supervisor_terminal_send_failure, which continues to pass.

Test

New regression test: when the task fails and the supervisor IPC send raises (BrokenPipeError simulating a dead Unix socket), run() raises SystemExit(1).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-006).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from 607e582 to 2acf7f0CompareMay 17, 2026 19:33
@potiuk

Copy link
Copy Markdown
MemberAuthor

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with nit

Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's a good idea to add the early exit for this case.

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@potiuk can you address comments?

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
potiukand others added 3 commits May 19, 2026 13:27
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from be3187f to d1a32abCompareMay 19, 2026 11:55
@vatsrahul1001
vatsrahul1001 merged commit 7e91517 into apache:mainMay 19, 2026
113 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-2-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 19, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@vatsrahul1001@Lee-W@kaxil@jason810496
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Fail closed when supervisor IPC fails on a non-success terminal state by potiuk · Pull Request #66573 · apache/airflow · GitHub
Skip to content

Fail closed when supervisor IPC fails on a non-success terminal state - #66573

Merged
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state
May 19, 2026
Merged

Fail closed when supervisor IPC fails on a non-success terminal state#66573
vatsrahul1001 merged 3 commits into
apache:mainfrom
potiuk:fix/tasksdk-fail-closed-terminal-state

Conversation

@potiuk

Copy link
Copy Markdown
Member

Summary

When a task FAILED / SKIPPED / etc. and the IPC send of the terminal-state message to the supervisor itself raised, the existing finally block logged the exception and let run() return normally. The task subprocess then exited with code 0, which the supervisor's final_state property maps to SUCCESS for an exit_code == 0 process without a _terminal_state (the supervisor never received the message).

A genuine task FAILURE was silently being upgraded to SUCCESS on transient IPC failures, breaking downstream pipeline correctness without any signal.

Fix

Exit non-zero from the finally block when the terminal state is anything other than SUCCESS, so the supervisor's final_state correctly classifies as FAILED (or UP_FOR_RETRY when retries are configured).

The SUCCESS exemption preserves the existing softening for the legitimate scenario where the supervisor rejects the terminal-state send with a 409 because the server already terminalised the TI — covered by test_run_swallows_supervisor_terminal_send_failure, which continues to pass.

Test

New regression test: when the task fails and the supervisor IPC send raises (BrokenPipeError simulating a dead Unix socket), run() raises SystemExit(1).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-006).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from 607e582 to 2acf7f0CompareMay 17, 2026 19:33
@potiuk

Copy link
Copy Markdown
MemberAuthor

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)


Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with nit

Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's a good idea to add the early exit for this case.

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@potiuk can you address comments?

Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
potiukand others added 3 commits May 19, 2026 13:27
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
@potiuk
potiukforce-pushed the fix/tasksdk-fail-closed-terminal-state branch from be3187f to d1a32abCompareMay 19, 2026 11:55
@vatsrahul1001
vatsrahul1001 merged commit 7e91517 into apache:mainMay 19, 2026
113 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-2-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 19, 2026
…rminal state (apache#66573)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 19, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rminal state (#66573) (#67183)
* Fail closed when supervisor IPC fails on a non-success terminal state
When a task FAILED / SKIPPED / etc. and the IPC send of the
terminal-state message to the supervisor itself raised, the existing
finally block logged the exception and let run() return normally.
The task subprocess then exited with code 0, which the supervisor
final_state property maps to SUCCESS for an exit_code-0 process
without a _terminal_state (the supervisor never received the
message). A genuine task FAILURE was silently being upgraded to
SUCCESS on transient IPC failures, breaking downstream pipeline
correctness without any signal.
Exit non-zero from the finally block when the terminal state is
anything other than SUCCESS, so the supervisor's final_state
correctly classifies as FAILED (or UP_FOR_RETRY when retries are
configured). The SUCCESS exemption preserves the existing softening
for the legitimate scenario where the supervisor rejects the
terminal-state send with a 409 because the server already
terminalised the TI -- covered by
test_run_swallows_supervisor_terminal_send_failure, which continues
to pass.
New regression test: when the task fails and the supervisor IPC send
raises (BrokenPipeError simulating a dead Unix socket), run() now
raises SystemExit(1).
Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-006).
* Address review comments: defer fail-closed exit, narrow guard
- Narrow the fail-closed guard from `state != SUCCESS` to FAILED /
UP_FOR_RETRY only (kaxil). SKIPPED / UP_FOR_RESCHEDULE / DEFERRED
would otherwise be mismapped to FAILED by supervisor's final_state,
which is strictly worse than the default mapping.
- Stop calling sys.exit(1) inside run()'s finally — SystemExit is a
BaseException so main()'s `except Exception:` would not catch it
and finalize() would be skipped, silently dropping
on_failure_callback / on_retry_callback / listener hooks /
email_on_failure / email_on_retry on the same IPC failure (kaxil).
Signal via `_terminal_state_send_failed` on the ti and let main()
sys.exit(1) after finalize() has run.
- Remove the redundant inline `from ... import run` in the test
(Lee-W, kaxil) — `run` is already imported at module level.
- Rework the existing regression test to assert the new contract
(run() returns FAILED + sets the flag) and add a listener-based
test that locks in callbacks/listeners still firing on the
IPC-broken path (kaxil).
* Declare _terminal_state_send_failed on RuntimeTaskInstance for mypy
The fail-closed path in _handle_current_task_failure set ti._terminal_state_send_failed = True
dynamically without a class-level declaration, so mypy raised [attr-defined].
Add the field as a Pydantic PrivateAttr (bool, default False) matching the
existing _cached_template_context pattern. No runtime behavior change —
getattr(ti, '_terminal_state_send_failed', False) still returns False for
the unset case, now via the PrivateAttr default instead of the getattr fallback.
---------
(cherry picked from commit 7e91517)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@vatsrahul1001@Lee-W@kaxil@jason810496