Fix flaky OTel integration tests by bounding scheduler shutdown wait by shubhamraj-git · Pull Request #67455 · apache/airflow

shubhamraj-git · 2026-05-25T06:11:11Z

The OTel integration test methods (test_export_legacy_metric_names) failed intermittently with Failed: Timeout >90.0s at scheduler_process.wait().

Root cause: The execution_timeout(90) was sized to match wait_for_dag_run(max_wait_time=90) but ignored the fixed overheads around it:

10s startup sleep (start_scheduler) 90s dag run wait (wait_for_dag_run max) 10s post-run sleep (span_status propagation) Xs scheduler shutdown — includes OTel atexit flush (force_flush, up to 10s)
─── 110s+ worst case > 90s limit

When the dag run took ~65–70s, only 5–10s remained for scheduler shutdown — not enough for force_flush(). The subprocess.wait() call had no timeout, blocking in os.waitpid() until SIGALRM fired. The 60→90 bump in #67170 was still too small for the same reason.

Changes:

Replace bare subprocess.wait() with wait(timeout=30) + TimeoutExpired / kill() / wait() in both finally blocks.
Raise execution_timeout from 90→160 on all three test methods.
Budget: 10 + 90 + 10 + 30 = 140s worst case + 20s CI buffer = 160s.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (claude-sonnet-4-6)

Generated-by: Claude Code (claude-sonnet-4-6) following the guidelines

amoghrajesh

Non blockers, but pre-emptive approval

Replace unbounded subprocess.wait() calls with a 30-second grace period followed by SIGKILL, and raise execution_timeout from 90s to 160s on all three OTel integration test methods. The 90s limit did not account for the fixed 10s startup sleep, 10s post-run sleep, and OTel atexit metric flush (up to 10s via force_flush), leaving no budget for shutdown on slow CI runs. The new 160s ceiling covers the 140s worst-case path with a 20s buffer.

potiuk · 2026-05-25T15:49:15Z

That's the third time we try it :)

shubhamraj-git · 2026-05-25T06:11:11Z

The OTel integration test methods (test_export_legacy_metric_names) failed intermittently with Failed: Timeout >90.0s at scheduler_process.wait().

Root cause: The execution_timeout(90) was sized to match wait_for_dag_run(max_wait_time=90) but ignored the fixed overheads around it:

10s startup sleep (start_scheduler) 90s dag run wait (wait_for_dag_run max) 10s post-run sleep (span_status propagation) Xs scheduler shutdown — includes OTel atexit flush (force_flush, up to 10s)
─── 110s+ worst case > 90s limit

When the dag run took ~65–70s, only 5–10s remained for scheduler shutdown — not enough for force_flush(). The subprocess.wait() call had no timeout, blocking in os.waitpid() until SIGALRM fired. The 60→90 bump in #67170 was still too small for the same reason.

Changes:

Replace bare subprocess.wait() with wait(timeout=30) + TimeoutExpired / kill() / wait() in both finally blocks.
Raise execution_timeout from 90→160 on all three test methods.
Budget: 10 + 90 + 10 + 30 = 140s worst case + 20s CI buffer = 160s.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (claude-sonnet-4-6)

Generated-by: Claude Code (claude-sonnet-4-6) following the guidelines

amoghrajesh

Non blockers, but pre-emptive approval

Replace unbounded subprocess.wait() calls with a 30-second grace period followed by SIGKILL, and raise execution_timeout from 90s to 160s on all three OTel integration test methods. The 90s limit did not account for the fixed 10s startup sleep, 10s post-run sleep, and OTel atexit metric flush (up to 10s via force_flush), leaving no budget for shutdown on slow CI runs. The new 160s ceiling covers the 140s worst-case path with a 20s buffer.

potiuk · 2026-05-25T15:49:15Z

That's the third time we try it :)

shubhamraj-git · 2026-05-25T06:11:11Z

The OTel integration test methods (test_export_legacy_metric_names) failed intermittently with Failed: Timeout >90.0s at scheduler_process.wait().

Root cause: The execution_timeout(90) was sized to match wait_for_dag_run(max_wait_time=90) but ignored the fixed overheads around it:

10s startup sleep (start_scheduler) 90s dag run wait (wait_for_dag_run max) 10s post-run sleep (span_status propagation) Xs scheduler shutdown — includes OTel atexit flush (force_flush, up to 10s)
─── 110s+ worst case > 90s limit

When the dag run took ~65–70s, only 5–10s remained for scheduler shutdown — not enough for force_flush(). The subprocess.wait() call had no timeout, blocking in os.waitpid() until SIGALRM fired. The 60→90 bump in #67170 was still too small for the same reason.

Changes:

Replace bare subprocess.wait() with wait(timeout=30) + TimeoutExpired / kill() / wait() in both finally blocks.
Raise execution_timeout from 90→160 on all three test methods.
Budget: 10 + 90 + 10 + 30 = 140s worst case + 20s CI buffer = 160s.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (claude-sonnet-4-6)

Generated-by: Claude Code (claude-sonnet-4-6) following the guidelines

amoghrajesh

Non blockers, but pre-emptive approval

Replace unbounded subprocess.wait() calls with a 30-second grace period followed by SIGKILL, and raise execution_timeout from 90s to 160s on all three OTel integration test methods. The 90s limit did not account for the fixed 10s startup sleep, 10s post-run sleep, and OTel atexit metric flush (up to 10s via force_flush), leaving no budget for shutdown on slow CI runs. The new 160s ceiling covers the 140s worst-case path with a 20s buffer.

potiuk · 2026-05-25T15:49:15Z

That's the third time we try it :)

shubhamraj-git · 2026-05-25T06:11:11Z

The OTel integration test methods (test_export_legacy_metric_names) failed intermittently with Failed: Timeout >90.0s at scheduler_process.wait().

Root cause: The execution_timeout(90) was sized to match wait_for_dag_run(max_wait_time=90) but ignored the fixed overheads around it:

10s startup sleep (start_scheduler) 90s dag run wait (wait_for_dag_run max) 10s post-run sleep (span_status propagation) Xs scheduler shutdown — includes OTel atexit flush (force_flush, up to 10s)
─── 110s+ worst case > 90s limit

When the dag run took ~65–70s, only 5–10s remained for scheduler shutdown — not enough for force_flush(). The subprocess.wait() call had no timeout, blocking in os.waitpid() until SIGALRM fired. The 60→90 bump in #67170 was still too small for the same reason.

Changes:

Replace bare subprocess.wait() with wait(timeout=30) + TimeoutExpired / kill() / wait() in both finally blocks.
Raise execution_timeout from 90→160 on all three test methods.
Budget: 10 + 90 + 10 + 30 = 140s worst case + 20s CI buffer = 160s.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (claude-sonnet-4-6)

Generated-by: Claude Code (claude-sonnet-4-6) following the guidelines

amoghrajesh

Non blockers, but pre-emptive approval

Replace unbounded subprocess.wait() calls with a 30-second grace period followed by SIGKILL, and raise execution_timeout from 90s to 160s on all three OTel integration test methods. The 90s limit did not account for the fixed 10s startup sleep, 10s post-run sleep, and OTel atexit metric flush (up to 10s via force_flush), leaving no budget for shutdown on slow CI runs. The new 160s ceiling covers the 140s worst-case path with a 20s buffer.

potiuk · 2026-05-25T15:49:15Z

That's the third time we try it :)

shubhamraj-git · 2026-05-25T06:11:11Z

The OTel integration test methods (test_export_legacy_metric_names) failed intermittently with Failed: Timeout >90.0s at scheduler_process.wait().

Root cause: The execution_timeout(90) was sized to match wait_for_dag_run(max_wait_time=90) but ignored the fixed overheads around it:

10s startup sleep (start_scheduler) 90s dag run wait (wait_for_dag_run max) 10s post-run sleep (span_status propagation) Xs scheduler shutdown — includes OTel atexit flush (force_flush, up to 10s)
─── 110s+ worst case > 90s limit

When the dag run took ~65–70s, only 5–10s remained for scheduler shutdown — not enough for force_flush(). The subprocess.wait() call had no timeout, blocking in os.waitpid() until SIGALRM fired. The 60→90 bump in #67170 was still too small for the same reason.

Changes:

Replace bare subprocess.wait() with wait(timeout=30) + TimeoutExpired / kill() / wait() in both finally blocks.
Raise execution_timeout from 90→160 on all three test methods.
Budget: 10 + 90 + 10 + 30 = 140s worst case + 20s CI buffer = 160s.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (claude-sonnet-4-6)

Generated-by: Claude Code (claude-sonnet-4-6) following the guidelines

amoghrajesh

Non blockers, but pre-emptive approval

Replace unbounded subprocess.wait() calls with a 30-second grace period followed by SIGKILL, and raise execution_timeout from 90s to 160s on all three OTel integration test methods. The 90s limit did not account for the fixed 10s startup sleep, 10s post-run sleep, and OTel atexit metric flush (up to 10s via force_flush), leaving no budget for shutdown on slow CI runs. The new 160s ceiling covers the 140s worst-case path with a 20s buffer.

potiuk · 2026-05-25T15:49:15Z

That's the third time we try it :)

shubhamraj-git · 2026-05-25T06:11:11Z

The OTel integration test methods (test_export_legacy_metric_names) failed intermittently with Failed: Timeout >90.0s at scheduler_process.wait().

Root cause: The execution_timeout(90) was sized to match wait_for_dag_run(max_wait_time=90) but ignored the fixed overheads around it:

10s startup sleep (start_scheduler) 90s dag run wait (wait_for_dag_run max) 10s post-run sleep (span_status propagation) Xs scheduler shutdown — includes OTel atexit flush (force_flush, up to 10s)
─── 110s+ worst case > 90s limit

When the dag run took ~65–70s, only 5–10s remained for scheduler shutdown — not enough for force_flush(). The subprocess.wait() call had no timeout, blocking in os.waitpid() until SIGALRM fired. The 60→90 bump in #67170 was still too small for the same reason.

Changes:

Replace bare subprocess.wait() with wait(timeout=30) + TimeoutExpired / kill() / wait() in both finally blocks.
Raise execution_timeout from 90→160 on all three test methods.
Budget: 10 + 90 + 10 + 30 = 140s worst case + 20s CI buffer = 160s.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (claude-sonnet-4-6)

Generated-by: Claude Code (claude-sonnet-4-6) following the guidelines

amoghrajesh

Non blockers, but pre-emptive approval

Replace unbounded subprocess.wait() calls with a 30-second grace period followed by SIGKILL, and raise execution_timeout from 90s to 160s on all three OTel integration test methods. The 90s limit did not account for the fixed 10s startup sleep, 10s post-run sleep, and OTel atexit metric flush (up to 10s via force_flush), leaving no budget for shutdown on slow CI runs. The new 160s ceiling covers the 140s worst-case path with a 20s buffer.

potiuk · 2026-05-25T15:49:15Z

That's the third time we try it :)

shubhamraj-git · 2026-05-25T06:11:11Z

The OTel integration test methods (test_export_legacy_metric_names) failed intermittently with Failed: Timeout >90.0s at scheduler_process.wait().

Root cause: The execution_timeout(90) was sized to match wait_for_dag_run(max_wait_time=90) but ignored the fixed overheads around it:

10s startup sleep (start_scheduler) 90s dag run wait (wait_for_dag_run max) 10s post-run sleep (span_status propagation) Xs scheduler shutdown — includes OTel atexit flush (force_flush, up to 10s)
─── 110s+ worst case > 90s limit

When the dag run took ~65–70s, only 5–10s remained for scheduler shutdown — not enough for force_flush(). The subprocess.wait() call had no timeout, blocking in os.waitpid() until SIGALRM fired. The 60→90 bump in #67170 was still too small for the same reason.

Changes:

Replace bare subprocess.wait() with wait(timeout=30) + TimeoutExpired / kill() / wait() in both finally blocks.
Raise execution_timeout from 90→160 on all three test methods.
Budget: 10 + 90 + 10 + 30 = 140s worst case + 20s CI buffer = 160s.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (claude-sonnet-4-6)

Generated-by: Claude Code (claude-sonnet-4-6) following the guidelines

amoghrajesh

Non blockers, but pre-emptive approval

Replace unbounded subprocess.wait() calls with a 30-second grace period followed by SIGKILL, and raise execution_timeout from 90s to 160s on all three OTel integration test methods. The 90s limit did not account for the fixed 10s startup sleep, 10s post-run sleep, and OTel atexit metric flush (up to 10s via force_flush), leaving no budget for shutdown on slow CI runs. The new 160s ceiling covers the 140s worst-case path with a 20s buffer.

potiuk · 2026-05-25T15:49:15Z

That's the third time we try it :)

shubhamraj-git · 2026-05-25T06:11:11Z

The OTel integration test methods (test_export_legacy_metric_names) failed intermittently with Failed: Timeout >90.0s at scheduler_process.wait().

Root cause: The execution_timeout(90) was sized to match wait_for_dag_run(max_wait_time=90) but ignored the fixed overheads around it:

10s startup sleep (start_scheduler) 90s dag run wait (wait_for_dag_run max) 10s post-run sleep (span_status propagation) Xs scheduler shutdown — includes OTel atexit flush (force_flush, up to 10s)
─── 110s+ worst case > 90s limit

When the dag run took ~65–70s, only 5–10s remained for scheduler shutdown — not enough for force_flush(). The subprocess.wait() call had no timeout, blocking in os.waitpid() until SIGALRM fired. The 60→90 bump in #67170 was still too small for the same reason.

Changes:

Replace bare subprocess.wait() with wait(timeout=30) + TimeoutExpired / kill() / wait() in both finally blocks.
Raise execution_timeout from 90→160 on all three test methods.
Budget: 10 + 90 + 10 + 30 = 140s worst case + 20s CI buffer = 160s.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (claude-sonnet-4-6)

Generated-by: Claude Code (claude-sonnet-4-6) following the guidelines

amoghrajesh

Non blockers, but pre-emptive approval

Replace unbounded subprocess.wait() calls with a 30-second grace period followed by SIGKILL, and raise execution_timeout from 90s to 160s on all three OTel integration test methods. The 90s limit did not account for the fixed 10s startup sleep, 10s post-run sleep, and OTel atexit metric flush (up to 10s via force_flush), leaving no budget for shutdown on slow CI runs. The new 160s ceiling covers the 140s worst-case path with a 20s buffer.

potiuk · 2026-05-25T15:49:15Z

That's the third time we try it :)

shubhamraj-git requested review from amoghrajesh, jscheffl and potiuk May 25, 2026 06:11

amoghrajesh approved these changes May 25, 2026
View reviewed changes

Comment threadairflow-core/tests/integration/otel/test_otel.py Outdated
Comment threadairflow-core/tests/integration/otel/test_otel.py

shubhamraj-git force-pushed the fix/otel-integration-test-timeout branch from 402902a to 6207e28Compare May 25, 2026 08:13

potiuk force-pushed the fix/otel-integration-test-timeout branch from 80193dc to 67a1667Compare May 25, 2026 15:48

potiuk approved these changes May 25, 2026
View reviewed changes

potiuk merged commit 40da90b into apache:mainMay 25, 2026
78 checks passed

omkhar mentioned this pull request May 26, 2026
edge3: bind team_name into worker JWT (defense-in-depth for experimental multi-team) #67397
Closed