Skip to content

Ensure scheduler uses best available serdag for dag run - #49097

Merged
kaxil merged 3 commits into
apache:mainfrom
astronomer:dag-run-scheduling-should-use-correct-serdag
Apr 15, 2025
Merged

Ensure scheduler uses best available serdag for dag run#49097
kaxil merged 3 commits into
apache:mainfrom
astronomer:dag-run-scheduling-should-use-correct-serdag

Conversation

@dstandish

@dstandishdstandish commented Apr 10, 2025

Copy link
Copy Markdown
Contributor

Each dag run is associated with a set of serdags via the bundle version / dag_id combination. When making scheduling decisions we need to use one of these serdags. Using the latest one is the best we can do right now.

Simpler alternative to #48956

Closes: #49007

@boring-cyborgboring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Apr 10, 2025
@dstandish
dstandishforce-pushed the dag-run-scheduling-should-use-correct-serdag branch from b0cf728 to 5cb6289CompareApril 13, 2025 12:24
Comment threadairflow-core/src/airflow/models/dagrun.py
@dstandish
dstandishforce-pushed the dag-run-scheduling-should-use-correct-serdag branch from b60797f to 5c40adeCompareApril 14, 2025 04:13
@dstandish
dstandish marked this pull request as ready for review April 14, 2025 04:13
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
@dstandish
dstandishforce-pushed the dag-run-scheduling-should-use-correct-serdag branch from be6a292 to 61d8be1CompareApril 14, 2025 14:43
@dstandishdstandish added the full tests needed We need to run full set of tests for this PR to merge label Apr 14, 2025
@dstandish
dstandishforce-pushed the dag-run-scheduling-should-use-correct-serdag branch 3 times, most recently from e494aab to 98ccdb7CompareApril 14, 2025 18:49
Each dag run is associated with a set of serdags via the bundle version / dag_id combination. When making scheduling decisions we need to use one of these serdags. Using the latest one is the best we can do right now.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Schedulerincluding HA (high availability) schedulerfull tests neededWe need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Versioned dag run uses the latest version of serialized dag in the Scheduler instead of sticking to their version

4 participants

@dstandish@ashb@ephraimbuddy@kaxil