Move jaeger CI host ports below ephemeral range to fix flake by potiuk · Pull Request #65867 · apache/airflow

potiuk · 2026-04-26T11:03:50Z

The Integration core otel job intermittently fails at docker compose
startup with:

failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp:
address already in use

The four jaeger host ports in scripts/ci/docker-compose/integration-otel.yml
(34317, 34318, 36686, 39422) sit inside Linux's default ephemeral port
range (32768-60999). The kernel can transiently grab any of these as
the source port of an outbound TCP connection from another process on
the runner, which races with docker's bind. The existing retry-with-
docker-restart logic doesn't help — the holder is not a docker process.

Move the four jaeger host ports below 32768, matching the convention
already used by the other services in the same compose file (24318,
28889, 29090, 23000). Also updates the matching jaeger-UI URL in the
otel test docstring.

Example failure:
https://github.com/apache/airflow/actions/runs/24954132155/job/73070212907

Was generative AI tooling used to co-author this PR?

Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port.

github-actions · 2026-04-26T11:55:07Z

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

…ake (apache#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) (#65868) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-04-26T11:03:50Z

The Integration core otel job intermittently fails at docker compose
startup with:

failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp:
address already in use

The four jaeger host ports in scripts/ci/docker-compose/integration-otel.yml
(34317, 34318, 36686, 39422) sit inside Linux's default ephemeral port
range (32768-60999). The kernel can transiently grab any of these as
the source port of an outbound TCP connection from another process on
the runner, which races with docker's bind. The existing retry-with-
docker-restart logic doesn't help — the holder is not a docker process.

Move the four jaeger host ports below 32768, matching the convention
already used by the other services in the same compose file (24318,
28889, 29090, 23000). Also updates the matching jaeger-UI URL in the
otel test docstring.

Example failure:
https://github.com/apache/airflow/actions/runs/24954132155/job/73070212907

Was generative AI tooling used to co-author this PR?

Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port.

github-actions · 2026-04-26T11:55:07Z

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

…ake (apache#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) (#65868) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-04-26T11:03:50Z

The Integration core otel job intermittently fails at docker compose
startup with:

failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp:
address already in use

The four jaeger host ports in scripts/ci/docker-compose/integration-otel.yml
(34317, 34318, 36686, 39422) sit inside Linux's default ephemeral port
range (32768-60999). The kernel can transiently grab any of these as
the source port of an outbound TCP connection from another process on
the runner, which races with docker's bind. The existing retry-with-
docker-restart logic doesn't help — the holder is not a docker process.

Move the four jaeger host ports below 32768, matching the convention
already used by the other services in the same compose file (24318,
28889, 29090, 23000). Also updates the matching jaeger-UI URL in the
otel test docstring.

Example failure:
https://github.com/apache/airflow/actions/runs/24954132155/job/73070212907

Was generative AI tooling used to co-author this PR?

Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port.

github-actions · 2026-04-26T11:55:07Z

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

…ake (apache#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) (#65868) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-04-26T11:03:50Z

The Integration core otel job intermittently fails at docker compose
startup with:

failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp:
address already in use

The four jaeger host ports in scripts/ci/docker-compose/integration-otel.yml
(34317, 34318, 36686, 39422) sit inside Linux's default ephemeral port
range (32768-60999). The kernel can transiently grab any of these as
the source port of an outbound TCP connection from another process on
the runner, which races with docker's bind. The existing retry-with-
docker-restart logic doesn't help — the holder is not a docker process.

Move the four jaeger host ports below 32768, matching the convention
already used by the other services in the same compose file (24318,
28889, 29090, 23000). Also updates the matching jaeger-UI URL in the
otel test docstring.

Example failure:
https://github.com/apache/airflow/actions/runs/24954132155/job/73070212907

Was generative AI tooling used to co-author this PR?

Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port.

github-actions · 2026-04-26T11:55:07Z

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

…ake (apache#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) (#65868) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-04-26T11:03:50Z

The Integration core otel job intermittently fails at docker compose
startup with:

failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp:
address already in use

The four jaeger host ports in scripts/ci/docker-compose/integration-otel.yml
(34317, 34318, 36686, 39422) sit inside Linux's default ephemeral port
range (32768-60999). The kernel can transiently grab any of these as
the source port of an outbound TCP connection from another process on
the runner, which races with docker's bind. The existing retry-with-
docker-restart logic doesn't help — the holder is not a docker process.

Move the four jaeger host ports below 32768, matching the convention
already used by the other services in the same compose file (24318,
28889, 29090, 23000). Also updates the matching jaeger-UI URL in the
otel test docstring.

Example failure:
https://github.com/apache/airflow/actions/runs/24954132155/job/73070212907

Was generative AI tooling used to co-author this PR?

Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port.

github-actions · 2026-04-26T11:55:07Z

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

…ake (apache#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) (#65868) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-04-26T11:03:50Z

The Integration core otel job intermittently fails at docker compose
startup with:

failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp:
address already in use

The four jaeger host ports in scripts/ci/docker-compose/integration-otel.yml
(34317, 34318, 36686, 39422) sit inside Linux's default ephemeral port
range (32768-60999). The kernel can transiently grab any of these as
the source port of an outbound TCP connection from another process on
the runner, which races with docker's bind. The existing retry-with-
docker-restart logic doesn't help — the holder is not a docker process.

Move the four jaeger host ports below 32768, matching the convention
already used by the other services in the same compose file (24318,
28889, 29090, 23000). Also updates the matching jaeger-UI URL in the
otel test docstring.

Example failure:
https://github.com/apache/airflow/actions/runs/24954132155/job/73070212907

Was generative AI tooling used to co-author this PR?

Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port.

github-actions · 2026-04-26T11:55:07Z

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

…ake (apache#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) (#65868) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-04-26T11:03:50Z

The Integration core otel job intermittently fails at docker compose
startup with:

failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp:
address already in use

The four jaeger host ports in scripts/ci/docker-compose/integration-otel.yml
(34317, 34318, 36686, 39422) sit inside Linux's default ephemeral port
range (32768-60999). The kernel can transiently grab any of these as
the source port of an outbound TCP connection from another process on
the runner, which races with docker's bind. The existing retry-with-
docker-restart logic doesn't help — the holder is not a docker process.

Move the four jaeger host ports below 32768, matching the convention
already used by the other services in the same compose file (24318,
28889, 29090, 23000). Also updates the matching jaeger-UI URL in the
otel test docstring.

Example failure:
https://github.com/apache/airflow/actions/runs/24954132155/job/73070212907

Was generative AI tooling used to co-author this PR?

Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port.

github-actions · 2026-04-26T11:55:07Z

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

…ake (apache#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) (#65868) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk · 2026-04-26T11:03:50Z

The Integration core otel job intermittently fails at docker compose
startup with:

failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp:
address already in use

The four jaeger host ports in scripts/ci/docker-compose/integration-otel.yml
(34317, 34318, 36686, 39422) sit inside Linux's default ephemeral port
range (32768-60999). The kernel can transiently grab any of these as
the source port of an outbound TCP connection from another process on
the runner, which races with docker's bind. The existing retry-with-
docker-restart logic doesn't help — the holder is not a docker process.

Move the four jaeger host ports below 32768, matching the convention
already used by the other services in the same compose file (24318,
28889, 29090, 23000). Also updates the matching jaeger-UI URL in the
otel test docstring.

Example failure:
https://github.com/apache/airflow/actions/runs/24954132155/job/73070212907

Was generative AI tooling used to co-author this PR?

Yes — Claude Opus 4.7 (1M context)

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port.

github-actions · 2026-04-26T11:55:07Z

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

…ake (apache#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

…ake (#65867) (#65868) The jaeger container in the otel integration compose file mapped four host ports (34317, 34318, 36686, 39422) inside the Linux default ephemeral port range (32768-60999). The kernel can transiently grab a port in that range as the source port of an outbound connection from any other process on the CI runner, which then races with the docker bind and surfaces as: failed to bind host port for 0.0.0.0:34318:172.18.0.3:4318/tcp: address already in use The retry mechanism (which restarts docker between attempts) does not help, because the holder is a non-docker process. Move the four jaeger host ports below 32768, matching the convention the other integration services in the same file already follow, and update the docstring in test_otel.py that pointed at the old jaeger UI port. (cherry picked from commit c29cc61) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk requested review from amoghrajesh, ashb, bugraoz93, gopidesupavan, jason810496 and jscheffl as code owners April 26, 2026 11:03

boring-cyborgBot added area:dev-tools backport-to-v3-2-test labels Apr 26, 2026

jscheffl approved these changes Apr 26, 2026
View reviewed changes

potiuk merged commit c29cc61 into apache:mainApr 26, 2026
135 of 136 checks passed

potiuk deleted the fix-otel-jaeger-port-flake branch April 26, 2026 11:53

hkc-8010 mentioned this pull request May 15, 2026
Reduce task success asset registration lock contention #66854
Closed
5 tasks