Don't crash supervisor IPC loop on transient network errors by potiuk · Pull Request #66572 · apache/airflow

potiuk · 2026-05-07T22:16:30Z

Summary

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor↔task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come.

Fix

Add a catch-all except Exception after the ServerResponseError handler that:

Logs the unhandled exception with type info.
Sends a best-effort ErrorResponse(API_SERVER_ERROR, …) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault).
Lets the request loop continue to the next request.

Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-005).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005).

potiuk · 2026-05-17T19:45:08Z

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

jason810496

Thanks, LGTM overall.

vatsrahul1001 · 2026-05-18T17:01:05Z

@potiuk can you address comments?

amoghrajesh

Non blocking

- Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback).

github-actions · 2026-05-19T11:22:43Z

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 1e5d799 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…67177) * Don't crash supervisor IPC loop on transient network errors handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005). * Address review comments: shorten comment and use exc_info - Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback). (cherry picked from commit 1e5d799) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-05-07T22:16:30Z

Summary

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor↔task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come.

Fix

Add a catch-all except Exception after the ServerResponseError handler that:

Logs the unhandled exception with type info.
Sends a best-effort ErrorResponse(API_SERVER_ERROR, …) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault).
Lets the request loop continue to the next request.

Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-005).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005).

potiuk · 2026-05-17T19:45:08Z

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

jason810496

Thanks, LGTM overall.

vatsrahul1001 · 2026-05-18T17:01:05Z

@potiuk can you address comments?

amoghrajesh

Non blocking

- Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback).

github-actions · 2026-05-19T11:22:43Z

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 1e5d799 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…67177) * Don't crash supervisor IPC loop on transient network errors handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005). * Address review comments: shorten comment and use exc_info - Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback). (cherry picked from commit 1e5d799) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-05-07T22:16:30Z

Summary

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor↔task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come.

Fix

Add a catch-all except Exception after the ServerResponseError handler that:

Logs the unhandled exception with type info.
Sends a best-effort ErrorResponse(API_SERVER_ERROR, …) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault).
Lets the request loop continue to the next request.

Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-005).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005).

potiuk · 2026-05-17T19:45:08Z

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

jason810496

Thanks, LGTM overall.

vatsrahul1001 · 2026-05-18T17:01:05Z

@potiuk can you address comments?

amoghrajesh

Non blocking

- Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback).

github-actions · 2026-05-19T11:22:43Z

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 1e5d799 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…67177) * Don't crash supervisor IPC loop on transient network errors handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005). * Address review comments: shorten comment and use exc_info - Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback). (cherry picked from commit 1e5d799) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-05-07T22:16:30Z

Summary

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor↔task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come.

Fix

Add a catch-all except Exception after the ServerResponseError handler that:

Logs the unhandled exception with type info.
Sends a best-effort ErrorResponse(API_SERVER_ERROR, …) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault).
Lets the request loop continue to the next request.

Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-005).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005).

potiuk · 2026-05-17T19:45:08Z

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

jason810496

Thanks, LGTM overall.

vatsrahul1001 · 2026-05-18T17:01:05Z

@potiuk can you address comments?

amoghrajesh

Non blocking

- Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback).

github-actions · 2026-05-19T11:22:43Z

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 1e5d799 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…67177) * Don't crash supervisor IPC loop on transient network errors handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005). * Address review comments: shorten comment and use exc_info - Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback). (cherry picked from commit 1e5d799) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-05-07T22:16:30Z

Summary

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor↔task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come.

Fix

Add a catch-all except Exception after the ServerResponseError handler that:

Logs the unhandled exception with type info.
Sends a best-effort ErrorResponse(API_SERVER_ERROR, …) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault).
Lets the request loop continue to the next request.

Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-005).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005).

potiuk · 2026-05-17T19:45:08Z

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

jason810496

Thanks, LGTM overall.

vatsrahul1001 · 2026-05-18T17:01:05Z

@potiuk can you address comments?

amoghrajesh

Non blocking

- Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback).

github-actions · 2026-05-19T11:22:43Z

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 1e5d799 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…67177) * Don't crash supervisor IPC loop on transient network errors handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005). * Address review comments: shorten comment and use exc_info - Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback). (cherry picked from commit 1e5d799) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-05-07T22:16:30Z

Summary

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor↔task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come.

Fix

Add a catch-all except Exception after the ServerResponseError handler that:

Logs the unhandled exception with type info.
Sends a best-effort ErrorResponse(API_SERVER_ERROR, …) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault).
Lets the request loop continue to the next request.

Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-005).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005).

potiuk · 2026-05-17T19:45:08Z

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

jason810496

Thanks, LGTM overall.

vatsrahul1001 · 2026-05-18T17:01:05Z

@potiuk can you address comments?

amoghrajesh

Non blocking

- Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback).

github-actions · 2026-05-19T11:22:43Z

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 1e5d799 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…67177) * Don't crash supervisor IPC loop on transient network errors handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005). * Address review comments: shorten comment and use exc_info - Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback). (cherry picked from commit 1e5d799) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-05-07T22:16:30Z

Summary

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor↔task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come.

Fix

Add a catch-all except Exception after the ServerResponseError handler that:

Logs the unhandled exception with type info.
Sends a best-effort ErrorResponse(API_SERVER_ERROR, …) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault).
Lets the request loop continue to the next request.

Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-005).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005).

potiuk · 2026-05-17T19:45:08Z

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

jason810496

Thanks, LGTM overall.

vatsrahul1001 · 2026-05-18T17:01:05Z

@potiuk can you address comments?

amoghrajesh

Non blocking

- Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback).

github-actions · 2026-05-19T11:22:43Z

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 1e5d799 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…67177) * Don't crash supervisor IPC loop on transient network errors handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005). * Address review comments: shorten comment and use exc_info - Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback). (cherry picked from commit 1e5d799) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-05-07T22:16:30Z

Summary

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor↔task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come.

Fix

Add a catch-all except Exception after the ServerResponseError handler that:

Logs the unhandled exception with type info.
Sends a best-effort ErrorResponse(API_SERVER_ERROR, …) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault).
Lets the request loop continue to the next request.

Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead).

Reported by

L3 ASVS sweep — apache/tooling-agents#24 (FINDING-005).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005).

potiuk · 2026-05-17T19:45:08Z

I'd love to get this one merged — and would love it in 3.2.2 if it's not too late. cc @vatsrahul1001 (3.2.2 RM)

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

jason810496

Thanks, LGTM overall.

vatsrahul1001 · 2026-05-18T17:01:05Z

@potiuk can you address comments?

amoghrajesh

Non blocking

- Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback).

github-actions · 2026-05-19T11:22:43Z

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 1e5d799 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…67177) * Don't crash supervisor IPC loop on transient network errors handle_requests in the supervisor only caught ServerResponseError. Any non-HTTP exception (httpx.ConnectError, httpx.TimeoutException, socket timeouts, etc.) would propagate, terminate the generator, and permanently break the supervisor-to-task IPC channel. The task subprocess would then get EOFError on every subsequent send, and the worker would be stuck waiting for replies that never come. Add a catch-all except Exception after the ServerResponseError handler that logs the unhandled exception with type info, sends a best-effort ErrorResponse(API_SERVER_ERROR, ...) back to the task so the failure surfaces in task logs (wrapped in suppress(Exception) because if we can't reach the task subprocess via stdin we shouldn't double-fault), and lets the request loop continue to the next request. Test added: a fake httpx.ConnectError on the first call produces an ErrorResponse, the generator stays alive, and a second request is processed normally (the loop is not dead). Reported by the L3 ASVS sweep at apache/tooling-agents#24 (FINDING-005). * Address review comments: shorten comment and use exc_info - Shorten the catch-all comment per amoghrajesh's suggestion. - Use exc_info=e in log.exception instead of exception_type field per jason810496's suggestion (exception type is redundant since the exception itself is logged with full type info and traceback). (cherry picked from commit 1e5d799) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk requested review from amoghrajesh, ashb and kaxil as code owners May 7, 2026 22:16

boring-cyborgBot added the area:task-sdk label May 7, 2026

potiuk force-pushed the fix/tasksdk-handle-requests-network-errors branch from 0f0a6ff to 067a4eeCompare May 17, 2026 19:34

potiuk added this to the Airflow 3.2.2 milestone May 17, 2026

potiuk added the backport-to-v3-2-test label May 17, 2026

Lee-W approved these changes May 18, 2026
View reviewed changes

eladkal approved these changes May 18, 2026
View reviewed changes

eladkal added the type:bug-fix Changelog: Bug Fixes label May 18, 2026

eladkal requested a review from vatsrahul1001 May 18, 2026 08:51

jason810496 reviewed May 18, 2026
View reviewed changes

Comment threadtask-sdk/src/airflow/sdk/execution_time/supervisor.py

amoghrajesh reviewed May 19, 2026
View reviewed changes

Comment threadtask-sdk/src/airflow/sdk/execution_time/supervisor.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/supervisor.py

amoghrajesh approved these changes May 19, 2026
View reviewed changes

vatsrahul1001 approved these changes May 19, 2026
View reviewed changes

vatsrahul1001 merged commit 1e5d799 into apache:mainMay 19, 2026
113 checks passed

vatsrahul1001 mentioned this pull request May 19, 2026
[v3-2-test] Don't crash supervisor IPC loop on transient network errors (#66572) #67177
Merged

vatsrahul1001 mentioned this pull request May 21, 2026
Status of testing of Apache Airflow 3.2.2rc3 #67282
Closed

mobuchowski mentioned this pull request Jun 18, 2026
OpenLineage: add execute_in_thread to emit task events without forking #68708
Merged
1 task

Andrushika mentioned this pull request Jul 10, 2026
Webserver timeout randomly #64212
Open
2 tasks

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

This was referenced Aug 3, 2026
Status of testing Providers that were prepared on August 01, 2026 #70953
Closed
Status of testing Providers that were prepared on August 06, 2026 #71246
Closed