Skip to content

Isolate scheduled DagRun creation failures with savepoints - #71287

Open
fat-catTW wants to merge 2 commits into
apache:mainfrom
fat-catTW:fix-dagrun-creation-savepoint
Open

Isolate scheduled DagRun creation failures with savepoints#71287
fat-catTW wants to merge 2 commits into
apache:mainfrom
fat-catTW:fix-dagrun-creation-savepoint

Conversation

@fat-catTW

@fat-catTWfat-catTW commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Why

SchedulerJobRunner._create_dag_runs catches exceptions while creating scheduled DagRuns and then continues with the next Dag. This does not work reliably when the failure is caused by a database error, because the SQLAlchemy session can be left in a failed transaction state.

When that happens, one failing DagRun creation attempt can prevent the scheduler from creating DagRuns for other healthy Dags in the same batch.

Solution

Wrap each scheduled DagRun creation attempt in a nested transaction/savepoint. If a database-level failure happens for one Dag, only that DagRun creation attempt is rolled back and the outer scheduler transaction remains usable.

The in-memory active run counter is updated only after the savepoint succeeds, keeping scheduler state aligned with the database transaction outcome.

A regression test covers a flush-time database failure for one Dag and verifies that another Dag in the same batch still creates its DagRun successfully.

relates: #59120

  • [X] Yes (please specify the tool below)

Generated-by: [Codex] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborgboring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Aug 7, 2026
@fat-catTW
fat-catTWforce-pushed the fix-dagrun-creation-savepoint branch from f5e6d6f to e9deb87CompareAugust 8, 2026 10:24

@steveahnahnsteveahnahn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Reverting the savepoint wrapping reproduces the poisoned-transaction failure and the fix clears it LGTM aside from the one nit

Comment threadairflow-core/src/airflow/utils/sqlalchemy.py
A single database error while creating a scheduled DagRun can leave the scheduler session in a failed transaction state. That prevents the scheduler from reliably continuing with other healthy Dags in the same batch.
@fat-catTW
fat-catTWforce-pushed the fix-dagrun-creation-savepoint branch from e9deb87 to ac70c7dCompareAugust 12, 2026 06:19
@potiukpotiuk added the ready for maintainer review Set after triaging when all criteria pass. label Aug 13, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Schedulerincluding HA (high availability) schedulerready for maintainer reviewSet after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@fat-catTW@steveahnahn@potiuk