Cap default httpx connection pool on the task-sdk Client by potiuk · Pull Request #67506 · apache/airflow

potiuk · 2026-05-25T22:04:55Z

Client.__init__ did not set explicit httpx.Limits, so each Client inherited httpx's defaults of max_connections=100 and max_keepalive_connections=10. The supervisor's own _ensure_client capped at max_connections=10, max_keepalive_connections=1 — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.

Reported as F-011 in the apache/tooling-agents L3 task-sdk sweep 0920c77.

Change

Standardise on a single bounded default at the Client level via kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20)). The supervisor's lower-cap path is unchanged (it sets limits explicitly before passing to the Client), and any caller that needs different limits can still pass limits=... via kwargs.

Choosing (5, 20) rather than (1, 10): a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.

Test plan

test_default_pool_limits_are_bounded — instantiates a real (non-mock-transport) Client and asserts pool._max_connections == 20 and pool._max_keepalive_connections == 5.
test_pool_limits_can_be_overridden — limits=httpx.Limits(...) kwarg overrides the default.
prek run ruff clean.
prek run mypy-task-sdk clean.
Full test_client.py suite: 117 passed.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

``Client.__init__`` did not set explicit ``httpx.Limits``, so each Client inherited httpx's defaults of ``max_connections=100`` and ``max_keepalive_connections=10``. The supervisor's own ``_ensure_client`` capped at ``max_connections=10, max_keepalive_connections=1`` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Standardise on a single bounded default at the Client level: ``httpx.Limits(max_keepalive_connections=5, max_connections=20)``. The supervisor's lower-cap path is unchanged (it sets ``limits`` explicitly), and any caller that needs different limits can still pass ``limits=...`` via kwargs.

ashb

Ai hallucination.

ashb · 2026-06-04T04:36:18Z


+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default


This code is only run in the supervisor, so this comment makes no sense

potiuk · 2026-06-05T01:17:26Z

You're right — closing this. Traced it through: the only real-network instantiation is _ensure_client in the supervisor, which already sets limits=httpx.Limits(max_connections=10) explicitly, so the setdefault("limits", …) here never fires on that path. The other two call sites (dag_processing/manager.py, triggerer_job_runner.py) omit limits but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

potiuk · 2026-05-25T22:04:55Z

Client.__init__ did not set explicit httpx.Limits, so each Client inherited httpx's defaults of max_connections=100 and max_keepalive_connections=10. The supervisor's own _ensure_client capped at max_connections=10, max_keepalive_connections=1 — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.

Reported as F-011 in the apache/tooling-agents L3 task-sdk sweep 0920c77.

Change

Standardise on a single bounded default at the Client level via kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20)). The supervisor's lower-cap path is unchanged (it sets limits explicitly before passing to the Client), and any caller that needs different limits can still pass limits=... via kwargs.

Choosing (5, 20) rather than (1, 10): a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.

Test plan

test_default_pool_limits_are_bounded — instantiates a real (non-mock-transport) Client and asserts pool._max_connections == 20 and pool._max_keepalive_connections == 5.
test_pool_limits_can_be_overridden — limits=httpx.Limits(...) kwarg overrides the default.
prek run ruff clean.
prek run mypy-task-sdk clean.
Full test_client.py suite: 117 passed.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

``Client.__init__`` did not set explicit ``httpx.Limits``, so each Client inherited httpx's defaults of ``max_connections=100`` and ``max_keepalive_connections=10``. The supervisor's own ``_ensure_client`` capped at ``max_connections=10, max_keepalive_connections=1`` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Standardise on a single bounded default at the Client level: ``httpx.Limits(max_keepalive_connections=5, max_connections=20)``. The supervisor's lower-cap path is unchanged (it sets ``limits`` explicitly), and any caller that needs different limits can still pass ``limits=...`` via kwargs.

ashb

Ai hallucination.

ashb · 2026-06-04T04:36:18Z


+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default


This code is only run in the supervisor, so this comment makes no sense

potiuk · 2026-06-05T01:17:26Z

You're right — closing this. Traced it through: the only real-network instantiation is _ensure_client in the supervisor, which already sets limits=httpx.Limits(max_connections=10) explicitly, so the setdefault("limits", …) here never fires on that path. The other two call sites (dag_processing/manager.py, triggerer_job_runner.py) omit limits but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

potiuk · 2026-05-25T22:04:55Z

Client.__init__ did not set explicit httpx.Limits, so each Client inherited httpx's defaults of max_connections=100 and max_keepalive_connections=10. The supervisor's own _ensure_client capped at max_connections=10, max_keepalive_connections=1 — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.

Reported as F-011 in the apache/tooling-agents L3 task-sdk sweep 0920c77.

Change

Standardise on a single bounded default at the Client level via kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20)). The supervisor's lower-cap path is unchanged (it sets limits explicitly before passing to the Client), and any caller that needs different limits can still pass limits=... via kwargs.

Choosing (5, 20) rather than (1, 10): a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.

Test plan

test_default_pool_limits_are_bounded — instantiates a real (non-mock-transport) Client and asserts pool._max_connections == 20 and pool._max_keepalive_connections == 5.
test_pool_limits_can_be_overridden — limits=httpx.Limits(...) kwarg overrides the default.
prek run ruff clean.
prek run mypy-task-sdk clean.
Full test_client.py suite: 117 passed.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

``Client.__init__`` did not set explicit ``httpx.Limits``, so each Client inherited httpx's defaults of ``max_connections=100`` and ``max_keepalive_connections=10``. The supervisor's own ``_ensure_client`` capped at ``max_connections=10, max_keepalive_connections=1`` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Standardise on a single bounded default at the Client level: ``httpx.Limits(max_keepalive_connections=5, max_connections=20)``. The supervisor's lower-cap path is unchanged (it sets ``limits`` explicitly), and any caller that needs different limits can still pass ``limits=...`` via kwargs.

ashb

Ai hallucination.

ashb · 2026-06-04T04:36:18Z


+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default


This code is only run in the supervisor, so this comment makes no sense

potiuk · 2026-06-05T01:17:26Z

You're right — closing this. Traced it through: the only real-network instantiation is _ensure_client in the supervisor, which already sets limits=httpx.Limits(max_connections=10) explicitly, so the setdefault("limits", …) here never fires on that path. The other two call sites (dag_processing/manager.py, triggerer_job_runner.py) omit limits but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

potiuk · 2026-05-25T22:04:55Z

Client.__init__ did not set explicit httpx.Limits, so each Client inherited httpx's defaults of max_connections=100 and max_keepalive_connections=10. The supervisor's own _ensure_client capped at max_connections=10, max_keepalive_connections=1 — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.

Reported as F-011 in the apache/tooling-agents L3 task-sdk sweep 0920c77.

Change

Standardise on a single bounded default at the Client level via kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20)). The supervisor's lower-cap path is unchanged (it sets limits explicitly before passing to the Client), and any caller that needs different limits can still pass limits=... via kwargs.

Choosing (5, 20) rather than (1, 10): a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.

Test plan

test_default_pool_limits_are_bounded — instantiates a real (non-mock-transport) Client and asserts pool._max_connections == 20 and pool._max_keepalive_connections == 5.
test_pool_limits_can_be_overridden — limits=httpx.Limits(...) kwarg overrides the default.
prek run ruff clean.
prek run mypy-task-sdk clean.
Full test_client.py suite: 117 passed.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

``Client.__init__`` did not set explicit ``httpx.Limits``, so each Client inherited httpx's defaults of ``max_connections=100`` and ``max_keepalive_connections=10``. The supervisor's own ``_ensure_client`` capped at ``max_connections=10, max_keepalive_connections=1`` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Standardise on a single bounded default at the Client level: ``httpx.Limits(max_keepalive_connections=5, max_connections=20)``. The supervisor's lower-cap path is unchanged (it sets ``limits`` explicitly), and any caller that needs different limits can still pass ``limits=...`` via kwargs.

ashb

Ai hallucination.

ashb · 2026-06-04T04:36:18Z


+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default


This code is only run in the supervisor, so this comment makes no sense

potiuk · 2026-06-05T01:17:26Z

You're right — closing this. Traced it through: the only real-network instantiation is _ensure_client in the supervisor, which already sets limits=httpx.Limits(max_connections=10) explicitly, so the setdefault("limits", …) here never fires on that path. The other two call sites (dag_processing/manager.py, triggerer_job_runner.py) omit limits but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

potiuk · 2026-05-25T22:04:55Z

Client.__init__ did not set explicit httpx.Limits, so each Client inherited httpx's defaults of max_connections=100 and max_keepalive_connections=10. The supervisor's own _ensure_client capped at max_connections=10, max_keepalive_connections=1 — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.

Reported as F-011 in the apache/tooling-agents L3 task-sdk sweep 0920c77.

Change

Standardise on a single bounded default at the Client level via kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20)). The supervisor's lower-cap path is unchanged (it sets limits explicitly before passing to the Client), and any caller that needs different limits can still pass limits=... via kwargs.

Choosing (5, 20) rather than (1, 10): a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.

Test plan

test_default_pool_limits_are_bounded — instantiates a real (non-mock-transport) Client and asserts pool._max_connections == 20 and pool._max_keepalive_connections == 5.
test_pool_limits_can_be_overridden — limits=httpx.Limits(...) kwarg overrides the default.
prek run ruff clean.
prek run mypy-task-sdk clean.
Full test_client.py suite: 117 passed.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

``Client.__init__`` did not set explicit ``httpx.Limits``, so each Client inherited httpx's defaults of ``max_connections=100`` and ``max_keepalive_connections=10``. The supervisor's own ``_ensure_client`` capped at ``max_connections=10, max_keepalive_connections=1`` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Standardise on a single bounded default at the Client level: ``httpx.Limits(max_keepalive_connections=5, max_connections=20)``. The supervisor's lower-cap path is unchanged (it sets ``limits`` explicitly), and any caller that needs different limits can still pass ``limits=...`` via kwargs.

ashb

Ai hallucination.

ashb · 2026-06-04T04:36:18Z


+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default


This code is only run in the supervisor, so this comment makes no sense

potiuk · 2026-06-05T01:17:26Z

You're right — closing this. Traced it through: the only real-network instantiation is _ensure_client in the supervisor, which already sets limits=httpx.Limits(max_connections=10) explicitly, so the setdefault("limits", …) here never fires on that path. The other two call sites (dag_processing/manager.py, triggerer_job_runner.py) omit limits but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

potiuk · 2026-05-25T22:04:55Z

Client.__init__ did not set explicit httpx.Limits, so each Client inherited httpx's defaults of max_connections=100 and max_keepalive_connections=10. The supervisor's own _ensure_client capped at max_connections=10, max_keepalive_connections=1 — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.

Reported as F-011 in the apache/tooling-agents L3 task-sdk sweep 0920c77.

Change

Standardise on a single bounded default at the Client level via kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20)). The supervisor's lower-cap path is unchanged (it sets limits explicitly before passing to the Client), and any caller that needs different limits can still pass limits=... via kwargs.

Choosing (5, 20) rather than (1, 10): a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.

Test plan

test_default_pool_limits_are_bounded — instantiates a real (non-mock-transport) Client and asserts pool._max_connections == 20 and pool._max_keepalive_connections == 5.
test_pool_limits_can_be_overridden — limits=httpx.Limits(...) kwarg overrides the default.
prek run ruff clean.
prek run mypy-task-sdk clean.
Full test_client.py suite: 117 passed.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

``Client.__init__`` did not set explicit ``httpx.Limits``, so each Client inherited httpx's defaults of ``max_connections=100`` and ``max_keepalive_connections=10``. The supervisor's own ``_ensure_client`` capped at ``max_connections=10, max_keepalive_connections=1`` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Standardise on a single bounded default at the Client level: ``httpx.Limits(max_keepalive_connections=5, max_connections=20)``. The supervisor's lower-cap path is unchanged (it sets ``limits`` explicitly), and any caller that needs different limits can still pass ``limits=...`` via kwargs.

ashb

Ai hallucination.

ashb · 2026-06-04T04:36:18Z


+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default


This code is only run in the supervisor, so this comment makes no sense

potiuk · 2026-06-05T01:17:26Z

You're right — closing this. Traced it through: the only real-network instantiation is _ensure_client in the supervisor, which already sets limits=httpx.Limits(max_connections=10) explicitly, so the setdefault("limits", …) here never fires on that path. The other two call sites (dag_processing/manager.py, triggerer_job_runner.py) omit limits but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

potiuk · 2026-05-25T22:04:55Z

Client.__init__ did not set explicit httpx.Limits, so each Client inherited httpx's defaults of max_connections=100 and max_keepalive_connections=10. The supervisor's own _ensure_client capped at max_connections=10, max_keepalive_connections=1 — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.

Reported as F-011 in the apache/tooling-agents L3 task-sdk sweep 0920c77.

Change

Standardise on a single bounded default at the Client level via kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20)). The supervisor's lower-cap path is unchanged (it sets limits explicitly before passing to the Client), and any caller that needs different limits can still pass limits=... via kwargs.

Choosing (5, 20) rather than (1, 10): a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.

Test plan

test_default_pool_limits_are_bounded — instantiates a real (non-mock-transport) Client and asserts pool._max_connections == 20 and pool._max_keepalive_connections == 5.
test_pool_limits_can_be_overridden — limits=httpx.Limits(...) kwarg overrides the default.
prek run ruff clean.
prek run mypy-task-sdk clean.
Full test_client.py suite: 117 passed.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

``Client.__init__`` did not set explicit ``httpx.Limits``, so each Client inherited httpx's defaults of ``max_connections=100`` and ``max_keepalive_connections=10``. The supervisor's own ``_ensure_client`` capped at ``max_connections=10, max_keepalive_connections=1`` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Standardise on a single bounded default at the Client level: ``httpx.Limits(max_keepalive_connections=5, max_connections=20)``. The supervisor's lower-cap path is unchanged (it sets ``limits`` explicitly), and any caller that needs different limits can still pass ``limits=...`` via kwargs.

ashb

Ai hallucination.

ashb · 2026-06-04T04:36:18Z


+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default


This code is only run in the supervisor, so this comment makes no sense

potiuk · 2026-06-05T01:17:26Z

You're right — closing this. Traced it through: the only real-network instantiation is _ensure_client in the supervisor, which already sets limits=httpx.Limits(max_connections=10) explicitly, so the setdefault("limits", …) here never fires on that path. The other two call sites (dag_processing/manager.py, triggerer_job_runner.py) omit limits but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

potiuk · 2026-05-25T22:04:55Z

Client.__init__ did not set explicit httpx.Limits, so each Client inherited httpx's defaults of max_connections=100 and max_keepalive_connections=10. The supervisor's own _ensure_client capped at max_connections=10, max_keepalive_connections=1 — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency.

Reported as F-011 in the apache/tooling-agents L3 task-sdk sweep 0920c77.

Change

Standardise on a single bounded default at the Client level via kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20)). The supervisor's lower-cap path is unchanged (it sets limits explicitly before passing to the Client), and any caller that needs different limits can still pass limits=... via kwargs.

Choosing (5, 20) rather than (1, 10): a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default.

Test plan

test_default_pool_limits_are_bounded — instantiates a real (non-mock-transport) Client and asserts pool._max_connections == 20 and pool._max_keepalive_connections == 5.
test_pool_limits_can_be_overridden — limits=httpx.Limits(...) kwarg overrides the default.
prek run ruff clean.
prek run mypy-task-sdk clean.
Full test_client.py suite: 117 passed.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

``Client.__init__`` did not set explicit ``httpx.Limits``, so each Client inherited httpx's defaults of ``max_connections=100`` and ``max_keepalive_connections=10``. The supervisor's own ``_ensure_client`` capped at ``max_connections=10, max_keepalive_connections=1`` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Standardise on a single bounded default at the Client level: ``httpx.Limits(max_keepalive_connections=5, max_connections=20)``. The supervisor's lower-cap path is unchanged (it sets ``limits`` explicitly), and any caller that needs different limits can still pass ``limits=...`` via kwargs.

ashb

Ai hallucination.

ashb · 2026-06-04T04:36:18Z


+ # Cap the httpx connection pool. Without an explicit value httpx defaults to 100 max
+ # connections, which is far higher than a single task subprocess ever needs (the
+ # supervisor sets ``max_connections=10`` in ``_ensure_client``). A bounded default


This code is only run in the supervisor, so this comment makes no sense

potiuk · 2026-06-05T01:17:26Z

You're right — closing this. Traced it through: the only real-network instantiation is _ensure_client in the supervisor, which already sets limits=httpx.Limits(max_connections=10) explicitly, so the setdefault("limits", …) here never fires on that path. The other two call sites (dag_processing/manager.py, triggerer_job_runner.py) omit limits but use in-process dry-run transports, where pool limits are moot. So the cap is effectively a no-op and the comment's "task subprocess / both code paths" framing was wrong. Thanks for catching it.

Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

potiuk requested review from amoghrajesh, ashb and kaxil as code owners May 25, 2026 22:04

boring-cyborgBot added the area:task-sdk label May 25, 2026

potiuk mentioned this pull request Jun 3, 2026
Warn when supervisor-side mask_secret IPC send fails #67503
Closed
7 tasks