Fix duplicate task execution when running multiple schedulers (HA) by ephraimbuddy · Pull Request #60330 · apache/airflow

ephraimbuddy · 2026-01-09T14:18:43Z

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued.

This change makes scheduling idempotent under HA races by:

Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly).
Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path.

Adds regression tests covering:

TI already queued by another scheduler.
EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING.
UP_FOR_RESCHEDULE scheduling keeps try_number unchanged.
Only one “scheduler” update succeeds when competing.

Closes: #57618

Note: The reproduction of this issue was based on unit tests

ashb

LGTM once we've done some perf testing

dabla · 2026-01-14T14:09:15Z

@ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator.

dabla · 2026-01-14T16:23:43Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

ephraimbuddy · 2026-01-15T13:59:50Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

dabla · 2026-01-21T10:41:47Z

Now we also experience the same issue with HttpOperator:

[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class 'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI widgets will be missing), because the 'flask_appbuilder' package is not installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!

yennysu · 2026-02-02T14:50:52Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.
Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running.

dstandish · 2026-03-02T16:30:24Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

sam-dumont · 2026-03-14T08:53:28Z

This PR fixed most of our issues (and we subsequently discovered another race condition). Our investigation is here in the comments #59378 (comment)

Anything we can do to finalize it and see it in airflow 3.1.9?

kaxil · 2026-03-19T03:27:37Z

@dabla @yennysu The error you're seeing ("reason":"not_found","message":"Task Instance not found") is a separate bug from what this PR fixes. This PR fixes the try_number double-increment race (where two schedulers both schedule the same TI). Your error is a 404, not a 409, caused by UUID reassignment during orphan adoption: when a scheduler crashes, another scheduler resets the orphaned TI via prepare_db_for_next_try() which assigns a new UUID. The worker that's still running heartbeats with the old UUID and gets a 404.

We're tracking that as a separate issue. Scaling with scheduler count matches this theory since more schedulers means more adoption cycles.

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

github-actions · 2026-03-19T04:31:35Z

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker a31db1f v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla · 2026-03-19T06:51:28Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

Sorry, missed your message @dstandish, we where experiencing this on kunernetes with 2 pod instances of the scheduler. Since we installed 3.1.8 I haven’t experienced this error anymore, so I might assume it is indeed fixed.

i’ve tried this fix through monkey patching though on earlier versions and then the issue persisted, so weird.

…#60330) In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Closes: apache#57618 Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

pdellarciprete · 2026-04-15T06:52:55Z

Sorry, I’m not sure I understood — is this fix included in 3.1.8 or 3.1.9?
If it’s planned for 3.1.9, is there an estimated release timeline?

cc: @ephraimbuddy @dabla @dstandish

anormalpersonBE · 2026-04-20T09:08:26Z

@pdellarciprete I was confused as well about the state of this fix.
As far as I can see on the main branch: your fix is in there.

fabbuc-gyg · 2026-04-21T17:54:12Z

@kaxil I just read your comment about the 404 error and "Task Instance not Found". I have been getting this a lot on our Airflow 3.1.8 deployment. Are you aware of any PR on this matter, or any workaround to get a grip on this?

pdellarciprete · 2026-05-05T04:52:04Z

Unfortunately, in our case this fix mitigated but didn't solve the problem.

squ1b3r · 2026-06-02T16:34:45Z

Currently running version 3.2.1 and using PartitionedAssetTimetable(assets=Asset("my-asset")) still causes duplicated dags runs when running multiple schedulers.

ephraimbuddy · 2026-01-09T14:18:43Z

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued.

This change makes scheduling idempotent under HA races by:

Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly).
Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path.

Adds regression tests covering:

TI already queued by another scheduler.
EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING.
UP_FOR_RESCHEDULE scheduling keeps try_number unchanged.
Only one “scheduler” update succeeds when competing.

Closes: #57618

Note: The reproduction of this issue was based on unit tests

ashb

LGTM once we've done some perf testing

dabla · 2026-01-14T14:09:15Z

@ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator.

dabla · 2026-01-14T16:23:43Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

ephraimbuddy · 2026-01-15T13:59:50Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

dabla · 2026-01-21T10:41:47Z

Now we also experience the same issue with HttpOperator:

[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class 'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI widgets will be missing), because the 'flask_appbuilder' package is not installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!

yennysu · 2026-02-02T14:50:52Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.
Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running.

dstandish · 2026-03-02T16:30:24Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

sam-dumont · 2026-03-14T08:53:28Z

This PR fixed most of our issues (and we subsequently discovered another race condition). Our investigation is here in the comments #59378 (comment)

Anything we can do to finalize it and see it in airflow 3.1.9?

kaxil · 2026-03-19T03:27:37Z

@dabla @yennysu The error you're seeing ("reason":"not_found","message":"Task Instance not found") is a separate bug from what this PR fixes. This PR fixes the try_number double-increment race (where two schedulers both schedule the same TI). Your error is a 404, not a 409, caused by UUID reassignment during orphan adoption: when a scheduler crashes, another scheduler resets the orphaned TI via prepare_db_for_next_try() which assigns a new UUID. The worker that's still running heartbeats with the old UUID and gets a 404.

We're tracking that as a separate issue. Scaling with scheduler count matches this theory since more schedulers means more adoption cycles.

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

github-actions · 2026-03-19T04:31:35Z

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker a31db1f v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla · 2026-03-19T06:51:28Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

Sorry, missed your message @dstandish, we where experiencing this on kunernetes with 2 pod instances of the scheduler. Since we installed 3.1.8 I haven’t experienced this error anymore, so I might assume it is indeed fixed.

i’ve tried this fix through monkey patching though on earlier versions and then the issue persisted, so weird.

…#60330) In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Closes: apache#57618 Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

pdellarciprete · 2026-04-15T06:52:55Z

Sorry, I’m not sure I understood — is this fix included in 3.1.8 or 3.1.9?
If it’s planned for 3.1.9, is there an estimated release timeline?

cc: @ephraimbuddy @dabla @dstandish

anormalpersonBE · 2026-04-20T09:08:26Z

@pdellarciprete I was confused as well about the state of this fix.
As far as I can see on the main branch: your fix is in there.

fabbuc-gyg · 2026-04-21T17:54:12Z

@kaxil I just read your comment about the 404 error and "Task Instance not Found". I have been getting this a lot on our Airflow 3.1.8 deployment. Are you aware of any PR on this matter, or any workaround to get a grip on this?

pdellarciprete · 2026-05-05T04:52:04Z

Unfortunately, in our case this fix mitigated but didn't solve the problem.

squ1b3r · 2026-06-02T16:34:45Z

Currently running version 3.2.1 and using PartitionedAssetTimetable(assets=Asset("my-asset")) still causes duplicated dags runs when running multiple schedulers.

ephraimbuddy · 2026-01-09T14:18:43Z

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued.

This change makes scheduling idempotent under HA races by:

Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly).
Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path.

Adds regression tests covering:

TI already queued by another scheduler.
EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING.
UP_FOR_RESCHEDULE scheduling keeps try_number unchanged.
Only one “scheduler” update succeeds when competing.

Closes: #57618

Note: The reproduction of this issue was based on unit tests

ashb

LGTM once we've done some perf testing

dabla · 2026-01-14T14:09:15Z

@ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator.

dabla · 2026-01-14T16:23:43Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

ephraimbuddy · 2026-01-15T13:59:50Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

dabla · 2026-01-21T10:41:47Z

Now we also experience the same issue with HttpOperator:

[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class 'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI widgets will be missing), because the 'flask_appbuilder' package is not installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!

yennysu · 2026-02-02T14:50:52Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.
Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running.

dstandish · 2026-03-02T16:30:24Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

sam-dumont · 2026-03-14T08:53:28Z

This PR fixed most of our issues (and we subsequently discovered another race condition). Our investigation is here in the comments #59378 (comment)

Anything we can do to finalize it and see it in airflow 3.1.9?

kaxil · 2026-03-19T03:27:37Z

@dabla @yennysu The error you're seeing ("reason":"not_found","message":"Task Instance not found") is a separate bug from what this PR fixes. This PR fixes the try_number double-increment race (where two schedulers both schedule the same TI). Your error is a 404, not a 409, caused by UUID reassignment during orphan adoption: when a scheduler crashes, another scheduler resets the orphaned TI via prepare_db_for_next_try() which assigns a new UUID. The worker that's still running heartbeats with the old UUID and gets a 404.

We're tracking that as a separate issue. Scaling with scheduler count matches this theory since more schedulers means more adoption cycles.

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

github-actions · 2026-03-19T04:31:35Z

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker a31db1f v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla · 2026-03-19T06:51:28Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

Sorry, missed your message @dstandish, we where experiencing this on kunernetes with 2 pod instances of the scheduler. Since we installed 3.1.8 I haven’t experienced this error anymore, so I might assume it is indeed fixed.

i’ve tried this fix through monkey patching though on earlier versions and then the issue persisted, so weird.

…#60330) In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Closes: apache#57618 Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

pdellarciprete · 2026-04-15T06:52:55Z

Sorry, I’m not sure I understood — is this fix included in 3.1.8 or 3.1.9?
If it’s planned for 3.1.9, is there an estimated release timeline?

cc: @ephraimbuddy @dabla @dstandish

anormalpersonBE · 2026-04-20T09:08:26Z

@pdellarciprete I was confused as well about the state of this fix.
As far as I can see on the main branch: your fix is in there.

fabbuc-gyg · 2026-04-21T17:54:12Z

@kaxil I just read your comment about the 404 error and "Task Instance not Found". I have been getting this a lot on our Airflow 3.1.8 deployment. Are you aware of any PR on this matter, or any workaround to get a grip on this?

pdellarciprete · 2026-05-05T04:52:04Z

Unfortunately, in our case this fix mitigated but didn't solve the problem.

squ1b3r · 2026-06-02T16:34:45Z

Currently running version 3.2.1 and using PartitionedAssetTimetable(assets=Asset("my-asset")) still causes duplicated dags runs when running multiple schedulers.

ephraimbuddy · 2026-01-09T14:18:43Z

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued.

This change makes scheduling idempotent under HA races by:

Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly).
Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path.

Adds regression tests covering:

TI already queued by another scheduler.
EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING.
UP_FOR_RESCHEDULE scheduling keeps try_number unchanged.
Only one “scheduler” update succeeds when competing.

Closes: #57618

Note: The reproduction of this issue was based on unit tests

ashb

LGTM once we've done some perf testing

dabla · 2026-01-14T14:09:15Z

@ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator.

dabla · 2026-01-14T16:23:43Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

ephraimbuddy · 2026-01-15T13:59:50Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

dabla · 2026-01-21T10:41:47Z

Now we also experience the same issue with HttpOperator:

[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class 'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI widgets will be missing), because the 'flask_appbuilder' package is not installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!

yennysu · 2026-02-02T14:50:52Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.
Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running.

dstandish · 2026-03-02T16:30:24Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

sam-dumont · 2026-03-14T08:53:28Z

This PR fixed most of our issues (and we subsequently discovered another race condition). Our investigation is here in the comments #59378 (comment)

Anything we can do to finalize it and see it in airflow 3.1.9?

kaxil · 2026-03-19T03:27:37Z

@dabla @yennysu The error you're seeing ("reason":"not_found","message":"Task Instance not found") is a separate bug from what this PR fixes. This PR fixes the try_number double-increment race (where two schedulers both schedule the same TI). Your error is a 404, not a 409, caused by UUID reassignment during orphan adoption: when a scheduler crashes, another scheduler resets the orphaned TI via prepare_db_for_next_try() which assigns a new UUID. The worker that's still running heartbeats with the old UUID and gets a 404.

We're tracking that as a separate issue. Scaling with scheduler count matches this theory since more schedulers means more adoption cycles.

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

github-actions · 2026-03-19T04:31:35Z

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker a31db1f v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla · 2026-03-19T06:51:28Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

Sorry, missed your message @dstandish, we where experiencing this on kunernetes with 2 pod instances of the scheduler. Since we installed 3.1.8 I haven’t experienced this error anymore, so I might assume it is indeed fixed.

i’ve tried this fix through monkey patching though on earlier versions and then the issue persisted, so weird.

…#60330) In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Closes: apache#57618 Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

pdellarciprete · 2026-04-15T06:52:55Z

Sorry, I’m not sure I understood — is this fix included in 3.1.8 or 3.1.9?
If it’s planned for 3.1.9, is there an estimated release timeline?

cc: @ephraimbuddy @dabla @dstandish

anormalpersonBE · 2026-04-20T09:08:26Z

@pdellarciprete I was confused as well about the state of this fix.
As far as I can see on the main branch: your fix is in there.

fabbuc-gyg · 2026-04-21T17:54:12Z

@kaxil I just read your comment about the 404 error and "Task Instance not Found". I have been getting this a lot on our Airflow 3.1.8 deployment. Are you aware of any PR on this matter, or any workaround to get a grip on this?

pdellarciprete · 2026-05-05T04:52:04Z

Unfortunately, in our case this fix mitigated but didn't solve the problem.

squ1b3r · 2026-06-02T16:34:45Z

Currently running version 3.2.1 and using PartitionedAssetTimetable(assets=Asset("my-asset")) still causes duplicated dags runs when running multiple schedulers.

ephraimbuddy · 2026-01-09T14:18:43Z

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued.

This change makes scheduling idempotent under HA races by:

Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly).
Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path.

Adds regression tests covering:

TI already queued by another scheduler.
EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING.
UP_FOR_RESCHEDULE scheduling keeps try_number unchanged.
Only one “scheduler” update succeeds when competing.

Closes: #57618

Note: The reproduction of this issue was based on unit tests

ashb

LGTM once we've done some perf testing

dabla · 2026-01-14T14:09:15Z

@ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator.

dabla · 2026-01-14T16:23:43Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

ephraimbuddy · 2026-01-15T13:59:50Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

dabla · 2026-01-21T10:41:47Z

Now we also experience the same issue with HttpOperator:

[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class 'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI widgets will be missing), because the 'flask_appbuilder' package is not installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!

yennysu · 2026-02-02T14:50:52Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.
Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running.

dstandish · 2026-03-02T16:30:24Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

sam-dumont · 2026-03-14T08:53:28Z

This PR fixed most of our issues (and we subsequently discovered another race condition). Our investigation is here in the comments #59378 (comment)

Anything we can do to finalize it and see it in airflow 3.1.9?

kaxil · 2026-03-19T03:27:37Z

@dabla @yennysu The error you're seeing ("reason":"not_found","message":"Task Instance not found") is a separate bug from what this PR fixes. This PR fixes the try_number double-increment race (where two schedulers both schedule the same TI). Your error is a 404, not a 409, caused by UUID reassignment during orphan adoption: when a scheduler crashes, another scheduler resets the orphaned TI via prepare_db_for_next_try() which assigns a new UUID. The worker that's still running heartbeats with the old UUID and gets a 404.

We're tracking that as a separate issue. Scaling with scheduler count matches this theory since more schedulers means more adoption cycles.

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

github-actions · 2026-03-19T04:31:35Z

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker a31db1f v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla · 2026-03-19T06:51:28Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

Sorry, missed your message @dstandish, we where experiencing this on kunernetes with 2 pod instances of the scheduler. Since we installed 3.1.8 I haven’t experienced this error anymore, so I might assume it is indeed fixed.

i’ve tried this fix through monkey patching though on earlier versions and then the issue persisted, so weird.

…#60330) In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Closes: apache#57618 Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

pdellarciprete · 2026-04-15T06:52:55Z

Sorry, I’m not sure I understood — is this fix included in 3.1.8 or 3.1.9?
If it’s planned for 3.1.9, is there an estimated release timeline?

cc: @ephraimbuddy @dabla @dstandish

anormalpersonBE · 2026-04-20T09:08:26Z

@pdellarciprete I was confused as well about the state of this fix.
As far as I can see on the main branch: your fix is in there.

fabbuc-gyg · 2026-04-21T17:54:12Z

@kaxil I just read your comment about the 404 error and "Task Instance not Found". I have been getting this a lot on our Airflow 3.1.8 deployment. Are you aware of any PR on this matter, or any workaround to get a grip on this?

pdellarciprete · 2026-05-05T04:52:04Z

Unfortunately, in our case this fix mitigated but didn't solve the problem.

squ1b3r · 2026-06-02T16:34:45Z

Currently running version 3.2.1 and using PartitionedAssetTimetable(assets=Asset("my-asset")) still causes duplicated dags runs when running multiple schedulers.

ephraimbuddy · 2026-01-09T14:18:43Z

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued.

This change makes scheduling idempotent under HA races by:

Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly).
Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path.

Adds regression tests covering:

TI already queued by another scheduler.
EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING.
UP_FOR_RESCHEDULE scheduling keeps try_number unchanged.
Only one “scheduler” update succeeds when competing.

Closes: #57618

Note: The reproduction of this issue was based on unit tests

ashb

LGTM once we've done some perf testing

dabla · 2026-01-14T14:09:15Z

@ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator.

dabla · 2026-01-14T16:23:43Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

ephraimbuddy · 2026-01-15T13:59:50Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

dabla · 2026-01-21T10:41:47Z

Now we also experience the same issue with HttpOperator:

[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class 'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI widgets will be missing), because the 'flask_appbuilder' package is not installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!

yennysu · 2026-02-02T14:50:52Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.
Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running.

dstandish · 2026-03-02T16:30:24Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

sam-dumont · 2026-03-14T08:53:28Z

This PR fixed most of our issues (and we subsequently discovered another race condition). Our investigation is here in the comments #59378 (comment)

Anything we can do to finalize it and see it in airflow 3.1.9?

kaxil · 2026-03-19T03:27:37Z

@dabla @yennysu The error you're seeing ("reason":"not_found","message":"Task Instance not found") is a separate bug from what this PR fixes. This PR fixes the try_number double-increment race (where two schedulers both schedule the same TI). Your error is a 404, not a 409, caused by UUID reassignment during orphan adoption: when a scheduler crashes, another scheduler resets the orphaned TI via prepare_db_for_next_try() which assigns a new UUID. The worker that's still running heartbeats with the old UUID and gets a 404.

We're tracking that as a separate issue. Scaling with scheduler count matches this theory since more schedulers means more adoption cycles.

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

github-actions · 2026-03-19T04:31:35Z

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker a31db1f v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla · 2026-03-19T06:51:28Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

Sorry, missed your message @dstandish, we where experiencing this on kunernetes with 2 pod instances of the scheduler. Since we installed 3.1.8 I haven’t experienced this error anymore, so I might assume it is indeed fixed.

i’ve tried this fix through monkey patching though on earlier versions and then the issue persisted, so weird.

…#60330) In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Closes: apache#57618 Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

pdellarciprete · 2026-04-15T06:52:55Z

Sorry, I’m not sure I understood — is this fix included in 3.1.8 or 3.1.9?
If it’s planned for 3.1.9, is there an estimated release timeline?

cc: @ephraimbuddy @dabla @dstandish

anormalpersonBE · 2026-04-20T09:08:26Z

@pdellarciprete I was confused as well about the state of this fix.
As far as I can see on the main branch: your fix is in there.

fabbuc-gyg · 2026-04-21T17:54:12Z

@kaxil I just read your comment about the 404 error and "Task Instance not Found". I have been getting this a lot on our Airflow 3.1.8 deployment. Are you aware of any PR on this matter, or any workaround to get a grip on this?

pdellarciprete · 2026-05-05T04:52:04Z

Unfortunately, in our case this fix mitigated but didn't solve the problem.

squ1b3r · 2026-06-02T16:34:45Z

Currently running version 3.2.1 and using PartitionedAssetTimetable(assets=Asset("my-asset")) still causes duplicated dags runs when running multiple schedulers.

ephraimbuddy · 2026-01-09T14:18:43Z

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued.

This change makes scheduling idempotent under HA races by:

Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly).
Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path.

Adds regression tests covering:

TI already queued by another scheduler.
EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING.
UP_FOR_RESCHEDULE scheduling keeps try_number unchanged.
Only one “scheduler” update succeeds when competing.

Closes: #57618

Note: The reproduction of this issue was based on unit tests

ashb

LGTM once we've done some perf testing

dabla · 2026-01-14T14:09:15Z

@ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator.

dabla · 2026-01-14T16:23:43Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

ephraimbuddy · 2026-01-15T13:59:50Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

dabla · 2026-01-21T10:41:47Z

Now we also experience the same issue with HttpOperator:

[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class 'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI widgets will be missing), because the 'flask_appbuilder' package is not installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!

yennysu · 2026-02-02T14:50:52Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.
Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running.

dstandish · 2026-03-02T16:30:24Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

sam-dumont · 2026-03-14T08:53:28Z

This PR fixed most of our issues (and we subsequently discovered another race condition). Our investigation is here in the comments #59378 (comment)

Anything we can do to finalize it and see it in airflow 3.1.9?

kaxil · 2026-03-19T03:27:37Z

@dabla @yennysu The error you're seeing ("reason":"not_found","message":"Task Instance not found") is a separate bug from what this PR fixes. This PR fixes the try_number double-increment race (where two schedulers both schedule the same TI). Your error is a 404, not a 409, caused by UUID reassignment during orphan adoption: when a scheduler crashes, another scheduler resets the orphaned TI via prepare_db_for_next_try() which assigns a new UUID. The worker that's still running heartbeats with the old UUID and gets a 404.

We're tracking that as a separate issue. Scaling with scheduler count matches this theory since more schedulers means more adoption cycles.

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

github-actions · 2026-03-19T04:31:35Z

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker a31db1f v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla · 2026-03-19T06:51:28Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

Sorry, missed your message @dstandish, we where experiencing this on kunernetes with 2 pod instances of the scheduler. Since we installed 3.1.8 I haven’t experienced this error anymore, so I might assume it is indeed fixed.

i’ve tried this fix through monkey patching though on earlier versions and then the issue persisted, so weird.

…#60330) In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Closes: apache#57618 Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

pdellarciprete · 2026-04-15T06:52:55Z

Sorry, I’m not sure I understood — is this fix included in 3.1.8 or 3.1.9?
If it’s planned for 3.1.9, is there an estimated release timeline?

cc: @ephraimbuddy @dabla @dstandish

anormalpersonBE · 2026-04-20T09:08:26Z

@pdellarciprete I was confused as well about the state of this fix.
As far as I can see on the main branch: your fix is in there.

fabbuc-gyg · 2026-04-21T17:54:12Z

@kaxil I just read your comment about the 404 error and "Task Instance not Found". I have been getting this a lot on our Airflow 3.1.8 deployment. Are you aware of any PR on this matter, or any workaround to get a grip on this?

pdellarciprete · 2026-05-05T04:52:04Z

Unfortunately, in our case this fix mitigated but didn't solve the problem.

squ1b3r · 2026-06-02T16:34:45Z

Currently running version 3.2.1 and using PartitionedAssetTimetable(assets=Asset("my-asset")) still causes duplicated dags runs when running multiple schedulers.

ephraimbuddy · 2026-01-09T14:18:43Z

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued.

This change makes scheduling idempotent under HA races by:

Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly).
Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path.

Adds regression tests covering:

TI already queued by another scheduler.
EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING.
UP_FOR_RESCHEDULE scheduling keeps try_number unchanged.
Only one “scheduler” update succeeds when competing.

Closes: #57618

Note: The reproduction of this issue was based on unit tests

ashb

LGTM once we've done some perf testing

dabla · 2026-01-14T14:09:15Z

@ephraimbuddy Good job, we are actually testing this fix right now, keep you guys posted if it helped. The weird thing is it seems (we're not sure yet) that we only have this issue with DAG's that have tasks that use the WinRMOperator.

dabla · 2026-01-14T16:23:43Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

ephraimbuddy · 2026-01-15T13:59:50Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.

Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

dabla · 2026-01-21T10:41:47Z

Now we also experience the same issue with HttpOperator:

[2026-01-21 04:44:19] INFO - Calling HTTP method
[2026-01-21 04:44:19] INFO - The hook_class 'airflow.providers.http.hooks.http.HttpHook' is not fully initialized (UI widgets will be missing), because the 'flask_appbuilder' package is not installed, however it is not required for Airflow components to work
[2026-01-21 04:44:25] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"detail":{"reason":"not_found","message":"Task Instance not found"}}
[2026-01-21 04:44:30] ERROR - Task killed!

yennysu · 2026-02-02T14:50:52Z

@ephraimbuddy Unfortunately, even after applying this fix, we still run onto the same issue.
Major problem with the issue is reliability of reproduction path. I have reproduced it once but trying again, I can't. Do you have steps for reliable reproduction?

While we haven’t been able to reproduce this reliably, we’ve observed that the frequency of occurrences seems to scale with the number of schedulers running.

dstandish · 2026-03-02T16:30:24Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

sam-dumont · 2026-03-14T08:53:28Z

This PR fixed most of our issues (and we subsequently discovered another race condition). Our investigation is here in the comments #59378 (comment)

Anything we can do to finalize it and see it in airflow 3.1.9?

kaxil · 2026-03-19T03:27:37Z

@dabla @yennysu The error you're seeing ("reason":"not_found","message":"Task Instance not found") is a separate bug from what this PR fixes. This PR fixes the try_number double-increment race (where two schedulers both schedule the same TI). Your error is a 404, not a 409, caused by UUID reassignment during orphan adoption: when a scheduler crashes, another scheduler resets the orphaned TI via prepare_db_for_next_try() which assigns a new UUID. The worker that's still running heartbeats with the old UUID and gets a 404.

We're tracking that as a separate issue. Scaling with scheduler count matches this theory since more schedulers means more adoption cycles.

In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

github-actions · 2026-03-19T04:31:35Z

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker a31db1f v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla · 2026-03-19T06:51:28Z

@dabla et al, does anyone have a concrete theory about exactly how this happpens?

Sorry, missed your message @dstandish, we where experiencing this on kunernetes with 2 pod instances of the scheduler. Since we installed 3.1.8 I haven’t experienced this error anymore, so I might assume it is indeed fixed.

i’ve tried this fix through monkey patching though on earlier versions and then the issue persisted, so weird.

…#60330) In HA, two scheduler processes can race to schedule the same TaskInstance. Previously DagRun.schedule_tis() updated rows by ti.id alone, so a scheduler could increment try_number and transition state even after another scheduler had already advanced the TI (e.g. to SCHEDULED/QUEUED), resulting in duplicate attempts being queued. This change makes scheduling idempotent under HA races by: - Guarding schedule_tis() DB updates to only apply when the TI is still in schedulable states (derived from SCHEDULEABLE_STATES, handling NULL explicitly). - Using a single CASE (next_try_number) so reschedules (UP_FOR_RESCHEDULE) do not start a new try, and applying this consistently to both normal scheduling and the EmptyOperator fast-path. The CASE uses TI.id (not TI.state) to avoid MySQL SET left-to-right evaluation issues. Adds regression tests covering: - TI already queued by another scheduler. - EmptyOperator fast-path blocked when TI is already QUEUED/RUNNING. - UP_FOR_RESCHEDULE scheduling keeps try_number unchanged. - Only one "scheduler" update succeeds when competing. Closes: apache#57618 Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

pdellarciprete · 2026-04-15T06:52:55Z

Sorry, I’m not sure I understood — is this fix included in 3.1.8 or 3.1.9?
If it’s planned for 3.1.9, is there an estimated release timeline?

cc: @ephraimbuddy @dabla @dstandish

anormalpersonBE · 2026-04-20T09:08:26Z

@pdellarciprete I was confused as well about the state of this fix.
As far as I can see on the main branch: your fix is in there.

fabbuc-gyg · 2026-04-21T17:54:12Z

@kaxil I just read your comment about the 404 error and "Task Instance not Found". I have been getting this a lot on our Airflow 3.1.8 deployment. Are you aware of any PR on this matter, or any workaround to get a grip on this?

pdellarciprete · 2026-05-05T04:52:04Z

Unfortunately, in our case this fix mitigated but didn't solve the problem.

squ1b3r · 2026-06-02T16:34:45Z

Currently running version 3.2.1 and using PartitionedAssetTimetable(assets=Asset("my-asset")) still causes duplicated dags runs when running multiple schedulers.

ephraimbuddy requested review from XD-DENG and ashb as code owners January 9, 2026 14:18

ephraimbuddy mentioned this pull request Jan 9, 2026
Schedulers race condition when using with kubernetes executor #57618
Closed
2 tasks

ashb reviewed Jan 9, 2026
View reviewed changes

Comment threadairflow-core/src/airflow/models/dagrun.py

ashb approved these changes Jan 9, 2026
View reviewed changes

pdellarciprete mentioned this pull request Jan 9, 2026
Prevent Concurrent TaskInstance Tries in Scheduler HA #59234
Closed

dabla mentioned this pull request Jan 16, 2026
Made WinRMOperator deferrable #60651
Merged
1 task

eladkal added this to the Airflow 3.1.7 milestone Jan 21, 2026

eladkal added the backport-to-v3-1-test label Jan 21, 2026

ephraimbuddy modified the milestones: Airflow 3.1.7, Airflow 3.1.8 Jan 28, 2026

vatsrahul1001 modified the milestones: Airflow 3.1.8, Airflow 3.1.9 Mar 2, 2026

jason810496 self-requested a review March 4, 2026 07:05

sam-dumont mentioned this pull request Mar 10, 2026
Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED #63266
Merged
1 task

This was referenced Mar 10, 2026
Task completes successfully but gets killed and marked as failed #63183
Closed
Fix schedule_tis HA race on try_number/state transitions #63367
Closed
Fix task runner failure on duplicate TI success update conflict #63355
Merged

sam-dumont mentioned this pull request Mar 13, 2026
Tasks are immediately restarted after unknown "Server indicated the task shouldn't be running anymore", resulting in HTTP 409 #59378
Closed
2 tasks

kaxil force-pushed the fix-ha-scheduler-race branch 2 times, most recently from d0d020f to d494a67Compare March 19, 2026 03:09

kaxil approved these changes Mar 19, 2026
View reviewed changes

kaxil changed the title ~~Fix HA scheduler try_number double increment~~Fix duplicate task execution when running multiple schedulers (HA)Mar 19, 2026

kaxil force-pushed the fix-ha-scheduler-race branch from 536195e to 5aed519Compare March 19, 2026 04:28

kaxil force-pushed the fix-ha-scheduler-race branch from 5aed519 to 66ffb5aCompare March 19, 2026 04:29

kaxil merged commit a31db1f into apache:mainMar 19, 2026
9 checks passed

kaxil deleted the fix-ha-scheduler-race branch March 19, 2026 04:29

diogoCalladoo mentioned this pull request May 24, 2026
try_number is incremented for Sensor Tasks in Reschedule Mode when using mysql database #45366
Closed
2 tasks