Uh oh!
There was an error while loading. Please reload this page.
Fix scheduler firing on_failure_callback for heartbeat-timed-out retries - #66767
Merged
vatsrahul1001 merged 1 commit intoJul 13, 2026
Merged
Conversation
Contributor
Hi @1fanwang, thanks for picking this up. Since this builds directly on #65404 (the diff is essentially identical), could you add this trailer to the commit message? @potiuk, could you make sure the trailer is preserved when this is squash-merged? Appreciate it. |
1fanwangforce-pushed
the
fix/65400-heartbeat-callback-type
branch
from
May 13, 2026 06:32
4656c17 to
608231dCompareContributorAuthor
Added — squashed + force-pushed with the trailer. Separately: open to closing this in favor of #65404 if you'd like to take it forward yourself. You opened first, you're the original reporter, and your version was in good shape — the credit should land with you. Let me know if you'd like to reopen #65404 and I'll close this one out. |
2 tasks
kimhaggie
commented
May 13, 2026
Contributor
Thanks @1fanwang, really appreciate both the trailer and the offer. I'll let this PR proceed since you've already put in the work. Happy with co-author attribution — looking forward to seeing this merged. |
1fanwangforce-pushed
the
fix/65400-heartbeat-callback-type
branch
2 times, most recently
from
June 19, 2026 18:42
2f039f2 to
b760421CompareUh oh!
There was an error while loading. Please reload this page.
1fanwangforce-pushed
the
fix/65400-heartbeat-callback-type
branch
from
June 29, 2026 06:50
b760421 to
70eef3cCompareWhen a worker stops heartbeating (OOMKill, node eviction), the scheduler's ``_purge_task_instances_without_heartbeats`` built a ``TaskCallbackRequest`` without ``task_callback_type``. The Dag processor's task-callback dispatch branches on that field: ``UP_FOR_RETRY`` runs ``on_retry_callback``, anything else (including ``None``) runs ``on_failure_callback``. So heartbeat-timeout cleanup always fired ``on_failure_callback`` even when the task still had retries remaining, producing spurious failure alerts for tasks that ultimately succeeded on retry. Set ``task_callback_type`` from ``ti.is_eligible_to_retry()``, the canonical retry-eligibility predicate, guarded by ``max_tries > 0``. The guard covers the one gap the predicate has here: this path doesn't load ``ti.task``, so the predicate falls back to ``try_number <= max_tries`` and drops the retries-configured check its task-loaded branch applies. Deferring to the predicate also keeps a ``RESTARTING`` task (cleared while running) retry- eligible past ``max_tries``, where a hand-rolled ``try_number <= max_tries`` check would have fired ``on_failure_callback``. closes: apache#65400 Co-authored-by: kimhaggie <kimhaggie@gmail.com> Signed-off-by: 1fanwang <1fannnw@gmail.com>
1fanwangforce-pushed
the
fix/65400-heartbeat-callback-type
branch
from
June 29, 2026 16:31
70eef3c to
bd1f186Compareuranusjr
approved these changes
Jul 1, 2026
jason810496
approved these changes
Jul 13, 2026
eladkal
approved these changes
Jul 13, 2026
vatsrahul1001
approved these changes
Jul 13, 2026
Uh oh!
There was an error while loading. Please reload this page.
Contributor
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 13, 2026
…med-out retries (apache#66767) When a worker stops heartbeating (OOMKill, node eviction), the scheduler's ``_purge_task_instances_without_heartbeats`` built a ``TaskCallbackRequest`` without ``task_callback_type``. The Dag processor's task-callback dispatch branches on that field: ``UP_FOR_RETRY`` runs ``on_retry_callback``, anything else (including ``None``) runs ``on_failure_callback``. So heartbeat-timeout cleanup always fired ``on_failure_callback`` even when the task still had retries remaining, producing spurious failure alerts for tasks that ultimately succeeded on retry. Set ``task_callback_type`` from ``ti.is_eligible_to_retry()``, the canonical retry-eligibility predicate, guarded by ``max_tries > 0``. The guard covers the one gap the predicate has here: this path doesn't load ``ti.task``, so the predicate falls back to ``try_number <= max_tries`` and drops the retries-configured check its task-loaded branch applies. Deferring to the predicate also keeps a ``RESTARTING`` task (cleared while running) retry- eligible past ``max_tries``, where a hand-rolled ``try_number <= max_tries`` check would have fired ``on_failure_callback``. closes: apache#65400 (cherry picked from commit f2403cc) Co-authored-by: Stefan Wang <1fannnw@gmail.com> Signed-off-by: 1fanwang <1fannnw@gmail.com> Co-authored-by: kimhaggie <kimhaggie@gmail.com>
eladkal pushed a commit
that referenced
this pull request
Jul 13, 2026
…med-out retries (#66767) (#69824) When a worker stops heartbeating (OOMKill, node eviction), the scheduler's ``_purge_task_instances_without_heartbeats`` built a ``TaskCallbackRequest`` without ``task_callback_type``. The Dag processor's task-callback dispatch branches on that field: ``UP_FOR_RETRY`` runs ``on_retry_callback``, anything else (including ``None``) runs ``on_failure_callback``. So heartbeat-timeout cleanup always fired ``on_failure_callback`` even when the task still had retries remaining, producing spurious failure alerts for tasks that ultimately succeeded on retry. Set ``task_callback_type`` from ``ti.is_eligible_to_retry()``, the canonical retry-eligibility predicate, guarded by ``max_tries > 0``. The guard covers the one gap the predicate has here: this path doesn't load ``ti.task``, so the predicate falls back to ``try_number <= max_tries`` and drops the retries-configured check its task-loaded branch applies. Deferring to the predicate also keeps a ``RESTARTING`` task (cleared while running) retry- eligible past ``max_tries``, where a hand-rolled ``try_number <= max_tries`` check would have fired ``on_failure_callback``. closes: #65400 (cherry picked from commit f2403cc) Signed-off-by: 1fanwang <1fannnw@gmail.com> Co-authored-by: Stefan Wang <1fannnw@gmail.com> Co-authored-by: kimhaggie <kimhaggie@gmail.com>
joshuabvarghese pushed a commit
to joshuabvarghese/airflow
that referenced
this pull request
Jul 16, 2026
…ies (apache#66767) When a worker stops heartbeating (OOMKill, node eviction), the scheduler's ``_purge_task_instances_without_heartbeats`` built a ``TaskCallbackRequest`` without ``task_callback_type``. The Dag processor's task-callback dispatch branches on that field: ``UP_FOR_RETRY`` runs ``on_retry_callback``, anything else (including ``None``) runs ``on_failure_callback``. So heartbeat-timeout cleanup always fired ``on_failure_callback`` even when the task still had retries remaining, producing spurious failure alerts for tasks that ultimately succeeded on retry. Set ``task_callback_type`` from ``ti.is_eligible_to_retry()``, the canonical retry-eligibility predicate, guarded by ``max_tries > 0``. The guard covers the one gap the predicate has here: this path doesn't load ``ti.task``, so the predicate falls back to ``try_number <= max_tries`` and drops the retries-configured check its task-loaded branch applies. Deferring to the predicate also keeps a ``RESTARTING`` task (cleared while running) retry- eligible past ``max_tries``, where a hand-rolled ``try_number <= max_tries`` check would have fired ``on_failure_callback``. closes: apache#65400 Signed-off-by: 1fanwang <1fannnw@gmail.com> Co-authored-by: kimhaggie <kimhaggie@gmail.com>
vatsrahul1001 pushed a commit
that referenced
this pull request
Aug 5, 2026
…med-out retries (#66767) (#69824) When a worker stops heartbeating (OOMKill, node eviction), the scheduler's ``_purge_task_instances_without_heartbeats`` built a ``TaskCallbackRequest`` without ``task_callback_type``. The Dag processor's task-callback dispatch branches on that field: ``UP_FOR_RETRY`` runs ``on_retry_callback``, anything else (including ``None``) runs ``on_failure_callback``. So heartbeat-timeout cleanup always fired ``on_failure_callback`` even when the task still had retries remaining, producing spurious failure alerts for tasks that ultimately succeeded on retry. Set ``task_callback_type`` from ``ti.is_eligible_to_retry()``, the canonical retry-eligibility predicate, guarded by ``max_tries > 0``. The guard covers the one gap the predicate has here: this path doesn't load ``ti.task``, so the predicate falls back to ``try_number <= max_tries`` and drops the retries-configured check its task-loaded branch applies. Deferring to the predicate also keeps a ``RESTARTING`` task (cleared while running) retry- eligible past ``max_tries``, where a hand-rolled ``try_number <= max_tries`` check would have fired ``on_failure_callback``. closes: #65400 (cherry picked from commit f2403cc) Signed-off-by: 1fanwang <1fannnw@gmail.com> Co-authored-by: Stefan Wang <1fannnw@gmail.com> Co-authored-by: kimhaggie <kimhaggie@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a worker stops heartbeating mid-retry (OOM kill, eviction, node drain), the scheduler's heartbeat-timeout cleanup enqueued a callback request without marking it as a retry, so the Dag processor fired
on_failure_callbackeven when the task still had retries left — producing spurious failure alerts for tasks that ultimately succeeded on retry.closes: #65400
What
The cleanup now marks the callback as a retry when the task is still retry-eligible, so
on_retry_callbackfires instead ofon_failure_callbackfor heartbeat-timed-out retries.Tests
Parametrized regression
test_heartbeat_timeout_sets_callback_type_by_retry_eligibilitydrives the real purge path and asserts both the callback type and the dispatched callback across the full retry matrix (no retries, retries available, mid-chain, exhausted). Each case fails onmainand passes here.Picking up from #65404 by @kimhaggie (closed for inactivity).