Fix scheduler MySQL task instance index hint

[8silvergun]

closes: #66763

Summary

SchedulerJobRunner._executable_task_instances_to_queued applied USE INDEX (ti_state) twice on MySQL: once on the
inner ranked query that filters by scheduled task instances, and once on the outer rejoin that looks up
task_instance by (dag_id, task_id, run_id, map_index).

The outer query is an exact identity lookup, so forcing ti_state can prevent MySQL from using
task_instance_composite_key. This PR removes only the outer hint and keeps the inner hint that was added for the
original MySQL scheduler optimization.

Regression Coverage

Added a regression test that compiles the scheduler selection query with the MySQL dialect and verifies USE INDEX (ti_state) appears only once.

Tests

• uv run --python 3.12 --group dev pytest airflow-core/tests/unit/jobs/ -xvs
• prek run ruff --from-ref main
• prek run ruff-format --from-ref main

Context

• MySQL Not Using Correct Index for Scheduler Critical Section Query #25627 added the original MySQL ti_state hint for the scheduler state lookup.
• Include the max_active_tasks limit in the query fetching TIs to be queued #54103 reshaped this scheduler query into the ranked subquery plus outer rejoin form; that is where the hint was
  also applied to the outer rejoin.
• Tasks returning from deferred should be prioritized #57210 touched the same scheduling path, but covers deferred-task priority rather than this MySQL hint placement.

---

Was generative AI tooling used to co-author this PR?

• Yes — OpenAI Codex

Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-
docs/05_pull_requests.rst#gen-ai-assisted-contributions)

[xBis7]

Generally it looks good. I need to run some benchmark tests just to be sure.

[xBis7]

I ran the same benchmark tests with and without the hint and I don't see any issues with the change. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[8silvergun]

closes: #66763

Summary

SchedulerJobRunner._executable_task_instances_to_queued applied USE INDEX (ti_state) twice on MySQL: once on the
inner ranked query that filters by scheduled task instances, and once on the outer rejoin that looks up
task_instance by (dag_id, task_id, run_id, map_index).

The outer query is an exact identity lookup, so forcing ti_state can prevent MySQL from using
task_instance_composite_key. This PR removes only the outer hint and keeps the inner hint that was added for the
original MySQL scheduler optimization.

Regression Coverage

Added a regression test that compiles the scheduler selection query with the MySQL dialect and verifies USE INDEX (ti_state) appears only once.

Tests

• uv run --python 3.12 --group dev pytest airflow-core/tests/unit/jobs/ -xvs
• prek run ruff --from-ref main
• prek run ruff-format --from-ref main

Context

• MySQL Not Using Correct Index for Scheduler Critical Section Query #25627 added the original MySQL ti_state hint for the scheduler state lookup.
• Include the max_active_tasks limit in the query fetching TIs to be queued #54103 reshaped this scheduler query into the ranked subquery plus outer rejoin form; that is where the hint was
  also applied to the outer rejoin.
• Tasks returning from deferred should be prioritized #57210 touched the same scheduling path, but covers deferred-task priority rather than this MySQL hint placement.

---

Was generative AI tooling used to co-author this PR?

• Yes — OpenAI Codex

Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-
docs/05_pull_requests.rst#gen-ai-assisted-contributions)

[xBis7]

Generally it looks good. I need to run some benchmark tests just to be sure.

[xBis7]

I ran the same benchmark tests with and without the hint and I don't see any issues with the change. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[8silvergun]

closes: #66763

Summary

SchedulerJobRunner._executable_task_instances_to_queued applied USE INDEX (ti_state) twice on MySQL: once on the
inner ranked query that filters by scheduled task instances, and once on the outer rejoin that looks up
task_instance by (dag_id, task_id, run_id, map_index).

The outer query is an exact identity lookup, so forcing ti_state can prevent MySQL from using
task_instance_composite_key. This PR removes only the outer hint and keeps the inner hint that was added for the
original MySQL scheduler optimization.

Regression Coverage

Added a regression test that compiles the scheduler selection query with the MySQL dialect and verifies USE INDEX (ti_state) appears only once.

Tests

• uv run --python 3.12 --group dev pytest airflow-core/tests/unit/jobs/ -xvs
• prek run ruff --from-ref main
• prek run ruff-format --from-ref main

Context

• MySQL Not Using Correct Index for Scheduler Critical Section Query #25627 added the original MySQL ti_state hint for the scheduler state lookup.
• Include the max_active_tasks limit in the query fetching TIs to be queued #54103 reshaped this scheduler query into the ranked subquery plus outer rejoin form; that is where the hint was
  also applied to the outer rejoin.
• Tasks returning from deferred should be prioritized #57210 touched the same scheduling path, but covers deferred-task priority rather than this MySQL hint placement.

---

Was generative AI tooling used to co-author this PR?

• Yes — OpenAI Codex

Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-
docs/05_pull_requests.rst#gen-ai-assisted-contributions)

[xBis7]

Generally it looks good. I need to run some benchmark tests just to be sure.

[xBis7]

I ran the same benchmark tests with and without the hint and I don't see any issues with the change. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[8silvergun]

closes: #66763

Summary

SchedulerJobRunner._executable_task_instances_to_queued applied USE INDEX (ti_state) twice on MySQL: once on the
inner ranked query that filters by scheduled task instances, and once on the outer rejoin that looks up
task_instance by (dag_id, task_id, run_id, map_index).

The outer query is an exact identity lookup, so forcing ti_state can prevent MySQL from using
task_instance_composite_key. This PR removes only the outer hint and keeps the inner hint that was added for the
original MySQL scheduler optimization.

Regression Coverage

Added a regression test that compiles the scheduler selection query with the MySQL dialect and verifies USE INDEX (ti_state) appears only once.

Tests

• uv run --python 3.12 --group dev pytest airflow-core/tests/unit/jobs/ -xvs
• prek run ruff --from-ref main
• prek run ruff-format --from-ref main

Context

• MySQL Not Using Correct Index for Scheduler Critical Section Query #25627 added the original MySQL ti_state hint for the scheduler state lookup.
• Include the max_active_tasks limit in the query fetching TIs to be queued #54103 reshaped this scheduler query into the ranked subquery plus outer rejoin form; that is where the hint was
  also applied to the outer rejoin.
• Tasks returning from deferred should be prioritized #57210 touched the same scheduling path, but covers deferred-task priority rather than this MySQL hint placement.

---

Was generative AI tooling used to co-author this PR?

• Yes — OpenAI Codex

Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-
docs/05_pull_requests.rst#gen-ai-assisted-contributions)

[xBis7]

Generally it looks good. I need to run some benchmark tests just to be sure.

[xBis7]

I ran the same benchmark tests with and without the hint and I don't see any issues with the change. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[8silvergun]

closes: #66763

Summary

SchedulerJobRunner._executable_task_instances_to_queued applied USE INDEX (ti_state) twice on MySQL: once on the
inner ranked query that filters by scheduled task instances, and once on the outer rejoin that looks up
task_instance by (dag_id, task_id, run_id, map_index).

The outer query is an exact identity lookup, so forcing ti_state can prevent MySQL from using
task_instance_composite_key. This PR removes only the outer hint and keeps the inner hint that was added for the
original MySQL scheduler optimization.

Regression Coverage

Added a regression test that compiles the scheduler selection query with the MySQL dialect and verifies USE INDEX (ti_state) appears only once.

Tests

• uv run --python 3.12 --group dev pytest airflow-core/tests/unit/jobs/ -xvs
• prek run ruff --from-ref main
• prek run ruff-format --from-ref main

Context

• MySQL Not Using Correct Index for Scheduler Critical Section Query #25627 added the original MySQL ti_state hint for the scheduler state lookup.
• Include the max_active_tasks limit in the query fetching TIs to be queued #54103 reshaped this scheduler query into the ranked subquery plus outer rejoin form; that is where the hint was
  also applied to the outer rejoin.
• Tasks returning from deferred should be prioritized #57210 touched the same scheduling path, but covers deferred-task priority rather than this MySQL hint placement.

---

Was generative AI tooling used to co-author this PR?

• Yes — OpenAI Codex

Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-
docs/05_pull_requests.rst#gen-ai-assisted-contributions)

[xBis7]

Generally it looks good. I need to run some benchmark tests just to be sure.

[xBis7]

I ran the same benchmark tests with and without the hint and I don't see any issues with the change. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[8silvergun]

closes: #66763

Summary

SchedulerJobRunner._executable_task_instances_to_queued applied USE INDEX (ti_state) twice on MySQL: once on the
inner ranked query that filters by scheduled task instances, and once on the outer rejoin that looks up
task_instance by (dag_id, task_id, run_id, map_index).

The outer query is an exact identity lookup, so forcing ti_state can prevent MySQL from using
task_instance_composite_key. This PR removes only the outer hint and keeps the inner hint that was added for the
original MySQL scheduler optimization.

Regression Coverage

Added a regression test that compiles the scheduler selection query with the MySQL dialect and verifies USE INDEX (ti_state) appears only once.

Tests

• uv run --python 3.12 --group dev pytest airflow-core/tests/unit/jobs/ -xvs
• prek run ruff --from-ref main
• prek run ruff-format --from-ref main

Context

• MySQL Not Using Correct Index for Scheduler Critical Section Query #25627 added the original MySQL ti_state hint for the scheduler state lookup.
• Include the max_active_tasks limit in the query fetching TIs to be queued #54103 reshaped this scheduler query into the ranked subquery plus outer rejoin form; that is where the hint was
  also applied to the outer rejoin.
• Tasks returning from deferred should be prioritized #57210 touched the same scheduling path, but covers deferred-task priority rather than this MySQL hint placement.

---

Was generative AI tooling used to co-author this PR?

• Yes — OpenAI Codex

Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-
docs/05_pull_requests.rst#gen-ai-assisted-contributions)

[xBis7]

Generally it looks good. I need to run some benchmark tests just to be sure.

[xBis7]

I ran the same benchmark tests with and without the hint and I don't see any issues with the change. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[8silvergun]

closes: #66763

Summary

SchedulerJobRunner._executable_task_instances_to_queued applied USE INDEX (ti_state) twice on MySQL: once on the
inner ranked query that filters by scheduled task instances, and once on the outer rejoin that looks up
task_instance by (dag_id, task_id, run_id, map_index).

The outer query is an exact identity lookup, so forcing ti_state can prevent MySQL from using
task_instance_composite_key. This PR removes only the outer hint and keeps the inner hint that was added for the
original MySQL scheduler optimization.

Regression Coverage

Added a regression test that compiles the scheduler selection query with the MySQL dialect and verifies USE INDEX (ti_state) appears only once.

Tests

• uv run --python 3.12 --group dev pytest airflow-core/tests/unit/jobs/ -xvs
• prek run ruff --from-ref main
• prek run ruff-format --from-ref main

Context

• MySQL Not Using Correct Index for Scheduler Critical Section Query #25627 added the original MySQL ti_state hint for the scheduler state lookup.
• Include the max_active_tasks limit in the query fetching TIs to be queued #54103 reshaped this scheduler query into the ranked subquery plus outer rejoin form; that is where the hint was
  also applied to the outer rejoin.
• Tasks returning from deferred should be prioritized #57210 touched the same scheduling path, but covers deferred-task priority rather than this MySQL hint placement.

---

Was generative AI tooling used to co-author this PR?

• Yes — OpenAI Codex

Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-
docs/05_pull_requests.rst#gen-ai-assisted-contributions)

[xBis7]

Generally it looks good. I need to run some benchmark tests just to be sure.

[xBis7]

I ran the same benchmark tests with and without the hint and I don't see any issues with the change. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[8silvergun]

closes: #66763

Summary

SchedulerJobRunner._executable_task_instances_to_queued applied USE INDEX (ti_state) twice on MySQL: once on the
inner ranked query that filters by scheduled task instances, and once on the outer rejoin that looks up
task_instance by (dag_id, task_id, run_id, map_index).

The outer query is an exact identity lookup, so forcing ti_state can prevent MySQL from using
task_instance_composite_key. This PR removes only the outer hint and keeps the inner hint that was added for the
original MySQL scheduler optimization.

Regression Coverage

Added a regression test that compiles the scheduler selection query with the MySQL dialect and verifies USE INDEX (ti_state) appears only once.

Tests

• uv run --python 3.12 --group dev pytest airflow-core/tests/unit/jobs/ -xvs
• prek run ruff --from-ref main
• prek run ruff-format --from-ref main

Context

• MySQL Not Using Correct Index for Scheduler Critical Section Query #25627 added the original MySQL ti_state hint for the scheduler state lookup.
• Include the max_active_tasks limit in the query fetching TIs to be queued #54103 reshaped this scheduler query into the ranked subquery plus outer rejoin form; that is where the hint was
  also applied to the outer rejoin.
• Tasks returning from deferred should be prioritized #57210 touched the same scheduling path, but covers deferred-task priority rather than this MySQL hint placement.

---

Was generative AI tooling used to co-author this PR?

• Yes — OpenAI Codex

Generated-by: OpenAI Codex following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-
docs/05_pull_requests.rst#gen-ai-assisted-contributions)

[xBis7]

Generally it looks good. I need to run some benchmark tests just to be sure.

[xBis7]

I ran the same benchmark tests with and without the hint and I don't see any issues with the change. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test