Skip to content

Add run_on_latest_version support for backfill and clear operations - #52177

Merged
jedcunningham merged 14 commits into
apache:mainfrom
astronomer:properly-clear-ti-specific-version
Jul 22, 2025
Merged

Add run_on_latest_version support for backfill and clear operations#52177
jedcunningham merged 14 commits into
apache:mainfrom
astronomer:properly-clear-ti-specific-version

Conversation

@ephraimbuddy

@ephraimbuddyephraimbuddy commented Jun 24, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.

When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.

For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.

Summary of changes:

  • Use SchedulerDagBag instead of DagBag for execution API
  • Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
  • Add --run-on-latest-version CLI flag for backfill command
  • Update backfill.py to support running tasks with latest DAG version
  • Add UI checkbox for "Run with latest version" in clear dialogs
  • Update SchedulerDagBag to handle latest version parameter
  • Update API endpoints to support run_on_latest_version parameter

closes: #49007, closes: #49047

@boring-cyborgboring-cyborgBot added area:airflow-ctl area:API Airflow's REST/HTTP API area:Scheduler including HA (high availability) scheduler area:translations area:UI Related to UI/UX. For Frontend Developers. translation:default labels Jun 24, 2025
@ephraimbuddy
ephraimbuddyforce-pushed the properly-clear-ti-specific-version branch 5 times, most recently from 1f4b2e3 to bfc2a49CompareJune 27, 2025 15:04
@ephraimbuddyephraimbuddy changed the title Fix clearing dags of versioned bundles and allow running on previous versionAdd run_on_latest_version support for backfill and clear operationsJun 27, 2025
@ephraimbuddy

ephraimbuddy commented Jun 27, 2025

Copy link
Copy Markdown
ContributorAuthor

Screenshot 2025-06-27 at 16 28 35

@ephraimbuddy
ephraimbuddyforce-pushed the properly-clear-ti-specific-version branch from bfc2a49 to 661e02aCompareJune 27, 2025 15:37
@ephraimbuddy
ephraimbuddy marked this pull request as ready for review June 27, 2025 15:38
jason810496
jason810496 reviewed Jun 29, 2025
View reviewed changes

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. There are some questions about the usage of SchedulerDagBag in core-api.

Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py
@bbovenzi

bbovenzi commented Jun 30, 2025

Copy link
Copy Markdown
Contributor

UI for dagrun clear and task instance clear look good. We'll add this to backfills next?

@ephraimbuddy
ephraimbuddyforce-pushed the properly-clear-ti-specific-version branch from b7ab41b to 71ec6c1CompareJuly 14, 2025 18:49
jedcunningham
jedcunningham approved these changes Jul 21, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
@ephraimbuddy
ephraimbuddyforce-pushed the properly-clear-ti-specific-version branch from 71ec6c1 to 52a004eCompareJuly 21, 2025 20:31
ephraimbuddyand others added 14 commits July 22, 2025 10:59
@ephraimbuddy
Add run_on_latest_version support for backfill and clear operations
1109cad
With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.
When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.
Summary of changes:
- Use SchedulerDagBag instead of DagBag for execution API
- Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
- Add --run-on-latest-version CLI flag for backfill command
- Update backfill.py to support running tasks with latest DAG version
- Add UI checkbox for "Run with latest version" in clear dialogs
- Update SchedulerDagBag to handle latest version parameter
- Update API endpoints to support run_on_latest_version parameter
@ephraimbuddy
Only show the run on latest version button when we are on versioned b…
fafb744
…undle
@ephraimbuddy
Fix test
39ebf82
@ephraimbuddy
Fix test by using serialized dag
d561388
@ephraimbuddy
fixup! Fix test by using serialized dag
91d57c2
@ephraimbuddy
Mark the --run-on-latest-version as experimental
8655ca9
@ephraimbuddy
properly save dag to the db in tests
e13778b
@ephraimbuddy
Fix ui as taskinstance.dag_version is no longer nullable
ac6d4cd
@ephraimbuddy@jedcunningham
Apply suggestions from code review
120dbc6
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy@jedcunningham
Update airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
8ce6088
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Update airflow-core/src/airflow/ui/openapi-gen/requests/schema…
db415ce
…s.gen.ts
@ephraimbuddy
Do not show run on latest button if run was on latest
73eec26
@ephraimbuddy
Update doc
5d7dca3
@ephraimbuddy
fixup! Update doc
a6a1496
@ephraimbuddy
ephraimbuddyforce-pushed the properly-clear-ti-specific-version branch from 52a004e to a6a1496CompareJuly 22, 2025 10:01
pierrejeambrun
pierrejeambrun approved these changes Jul 22, 2025
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!

@jedcunninghamjedcunningham added this to the Airflow 3.1.0 milestone Jul 22, 2025
@jedcunningham
jedcunningham merged commit 3dc4cd2 into apache:mainJul 22, 2025
102 checks passed
@jedcunningham
jedcunningham deleted the properly-clear-ti-specific-version branch July 22, 2025 17:12
kaxil added a commit to apache/airflow-client-python that referenced this pull request Oct 22, 2025
@kaxil
Update Python Client to 3.1.0rc1
e62c042
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@kaxilkaxil mentioned this pull request Oct 22, 2025
Merged
kaxil added a commit to apache/airflow-client-python that referenced this pull request Oct 23, 2025
@kaxil
Update Python Client to 3.1.0rc1 (#138)
241405e
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@ogulcanaydoganogulcanaydogan mentioned this pull request Mar 19, 2026
Open
2 tasks
@imrichardwuimrichardwu mentioned this pull request Jul 31, 2026
Merged
1 task
@dstandishdstandish mentioned this pull request Aug 11, 2026
Open
@ephraimbuddyephraimbuddy mentioned this pull request Aug 24, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbashb left review comments
@kaxilkaxilkaxil left review comments
@bugraoz93bugraoz93bugraoz93 approved these changes
@jason810496jason810496jason810496 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@jscheffljschefflAwaiting requested review from jscheffl
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@eladkaleladkalAwaiting requested review from eladkal

Assignees

No one assigned

Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:Schedulerincluding HA (high availability) schedulerarea:translationsarea:UIRelated to UI/UX. For Frontend Developers.translation:default

Projects

None yet

Milestone

Airflow 3.1.0

8 participants

@ephraimbuddy@bbovenzi@kaxil@ashb@bugraoz93@pierrejeambrun@jedcunningham@jason810496
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Add run_on_latest_version support for backfill and clear operations by ephraimbuddy · Pull Request #52177 · apache/airflow · GitHub
Skip to content
Add run_on_latest_version support for backfill and clear operations - #52177
Merged
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Jul 22, 2025
Merged
Add run_on_latest_version support for backfill and clear operations#52177
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Conversation
@ephraimbuddy
@ephraimbuddyephraimbuddy
commented
Jun 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
With this option, users are able to choose the dag version they want

to run their dag/task after clearing or when running backfill. This

only applies to versioned bundles as non-versioned bundles run with

the latest dag version.
When the user choose the run with latest version, the bundle_version

associated with the dagrun is updated to the latest and the associated

serialized dag version updated to the latest. Choosing not to run

with latest version which is the default means that the bundle version

and serialized dag version that the dag ran with initially would be used

in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run

with the latest version, otherwise it will run with the original bundle

the dagrun was created with. Note that it's only useful when rerunning

a dagrun using backfill. The default behaviour is using the initial bundle/version

and this is intentional otherwise running backfill will fail if there was

task rename in the latest version.
Summary of changes:
  • Use SchedulerDagBag instead of DagBag for execution API
  • Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
  • Add --run-on-latest-version CLI flag for backfill command
  • Update backfill.py to support running tasks with latest DAG version
  • Add UI checkbox for "Run with latest version" in clear dialogs
  • Update SchedulerDagBag to handle latest version parameter
  • Update API endpoints to support run_on_latest_version parameter
closes: #49007, closes: #49047
@boring-cyborgboring-cyborgBot
added

area:airflow-ctl

area:API
Airflow's REST/HTTP API
area:Scheduler
including HA (high availability) scheduler
area:translations

area:UI
Related to UI/UX. For Frontend Developers.
translation:default

labels
Jun 24, 2025
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
5 times, most recently
from
1f4b2e3 to
bfc2a49CompareJune 27, 2025 15:04
@ephraimbuddyephraimbuddy
changed the title
Fix clearing dags of versioned bundles and allow running on previous versionAdd run_on_latest_version support for backfill and clear operationsJun 27, 2025
@ephraimbuddy
ephraimbuddy
commented
Jun 27, 2025

Copy link


Copy Markdown

ContributorAuthor
Screenshot 2025-06-27 at 16 28 35
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
bfc2a49 to
661e02aCompareJune 27, 2025 15:37
@ephraimbuddy
ephraimbuddy
marked this pull request as ready for review
June 27, 2025 15:38
jason810496
jason810496
reviewed
Jun 29, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. There are some questions about the usage of SchedulerDagBag in core-api.
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py
@bbovenzi
bbovenzi
commented
Jun 30, 2025

Copy link


Copy Markdown

Contributor
UI for dagrun clear and task instance clear look good. We'll add this to backfills next?
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
b7ab41b to
71ec6c1CompareJuly 14, 2025 18:49
jedcunningham
jedcunningham
approved these changes
Jul 21, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
71ec6c1 to
52a004eCompareJuly 21, 2025 20:31
ephraimbuddyand others
added 14 commits
July 22, 2025 10:59
@ephraimbuddy
Add run_on_latest_version support for backfill and clear operations
1109cad
With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.
When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.
Summary of changes:
- Use SchedulerDagBag instead of DagBag for execution API
- Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
- Add --run-on-latest-version CLI flag for backfill command
- Update backfill.py to support running tasks with latest DAG version
- Add UI checkbox for "Run with latest version" in clear dialogs
- Update SchedulerDagBag to handle latest version parameter
- Update API endpoints to support run_on_latest_version parameter
@ephraimbuddy
Only show the run on latest version button when we are on versioned b…
fafb744
…undle
@ephraimbuddy
Fix test
39ebf82
@ephraimbuddy
Fix test by using serialized dag
d561388
@ephraimbuddy
fixup! Fix test by using serialized dag
91d57c2
@ephraimbuddy
Mark the --run-on-latest-version as experimental
8655ca9
@ephraimbuddy
properly save dag to the db in tests
e13778b
@ephraimbuddy
Fix ui as taskinstance.dag_version is no longer nullable
ac6d4cd
@ephraimbuddy@jedcunningham
Apply suggestions from code review
120dbc6
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy@jedcunningham
Update airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
8ce6088
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Update airflow-core/src/airflow/ui/openapi-gen/requests/schema…
db415ce
…s.gen.ts
@ephraimbuddy
Do not show run on latest button if run was on latest
73eec26
@ephraimbuddy
Update doc
5d7dca3
@ephraimbuddy
fixup! Update doc
a6a1496
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
52a004e to
a6a1496CompareJuly 22, 2025 10:01
pierrejeambrun
pierrejeambrun
approved these changes
Jul 22, 2025
View reviewed changes
@pierrejeambrunpierrejeambrun
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!
@jedcunninghamjedcunningham
added this to the Airflow 3.1.0 milestone
Jul 22, 2025
@jedcunningham
jedcunningham
merged commit 3dc4cd2
into
apache:mainJul 22, 2025

102 checks passed

@jedcunningham
jedcunningham
deleted the

properly-clear-ti-specific-version

branch
July 22, 2025 17:12
kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 22, 2025
@kaxil
Update Python Client to 3.1.0rc1
e62c042
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@kaxilkaxil
mentioned this pull request
Oct 22, 2025
 Merged

kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 23, 2025
@kaxil
Update Python Client to 3.1.0rc1 (#138)
241405e
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@ogulcanaydoganogulcanaydogan
mentioned this pull request
Mar 19, 2026

Open

2 tasks
@imrichardwuimrichardwu
mentioned this pull request
Jul 31, 2026
 Merged

1 task
@dstandishdstandish
mentioned this pull request
Aug 11, 2026

Open

@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 24, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@kaxilkaxilkaxil left review comments
@bugraoz93bugraoz93bugraoz93 approved these changes
@jason810496jason810496jason810496 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@jscheffljschefflAwaiting requested review from jscheffl
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:Schedulerincluding HA (high availability) schedulerarea:translationsarea:UIRelated to UI/UX. For Frontend Developers.translation:default

Projects


None yet


Milestone

Airflow 3.1.0

8 participants

@ephraimbuddy@bbovenzi@kaxil@ashb@bugraoz93@pierrejeambrun@jedcunningham@jason810496

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add run_on_latest_version support for backfill and clear operations by ephraimbuddy · Pull Request #52177 · apache/airflow · GitHub
Skip to content
Add run_on_latest_version support for backfill and clear operations - #52177
Merged
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Jul 22, 2025
Merged
Add run_on_latest_version support for backfill and clear operations#52177
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Conversation
@ephraimbuddy
@ephraimbuddyephraimbuddy
commented
Jun 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
With this option, users are able to choose the dag version they want

to run their dag/task after clearing or when running backfill. This

only applies to versioned bundles as non-versioned bundles run with

the latest dag version.
When the user choose the run with latest version, the bundle_version

associated with the dagrun is updated to the latest and the associated

serialized dag version updated to the latest. Choosing not to run

with latest version which is the default means that the bundle version

and serialized dag version that the dag ran with initially would be used

in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run

with the latest version, otherwise it will run with the original bundle

the dagrun was created with. Note that it's only useful when rerunning

a dagrun using backfill. The default behaviour is using the initial bundle/version

and this is intentional otherwise running backfill will fail if there was

task rename in the latest version.
Summary of changes:
  • Use SchedulerDagBag instead of DagBag for execution API
  • Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
  • Add --run-on-latest-version CLI flag for backfill command
  • Update backfill.py to support running tasks with latest DAG version
  • Add UI checkbox for "Run with latest version" in clear dialogs
  • Update SchedulerDagBag to handle latest version parameter
  • Update API endpoints to support run_on_latest_version parameter
closes: #49007, closes: #49047
@boring-cyborgboring-cyborgBot
added

area:airflow-ctl

area:API
Airflow's REST/HTTP API
area:Scheduler
including HA (high availability) scheduler
area:translations

area:UI
Related to UI/UX. For Frontend Developers.
translation:default

labels
Jun 24, 2025
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
5 times, most recently
from
1f4b2e3 to
bfc2a49CompareJune 27, 2025 15:04
@ephraimbuddyephraimbuddy
changed the title
Fix clearing dags of versioned bundles and allow running on previous versionAdd run_on_latest_version support for backfill and clear operationsJun 27, 2025
@ephraimbuddy
ephraimbuddy
commented
Jun 27, 2025

Copy link


Copy Markdown

ContributorAuthor
Screenshot 2025-06-27 at 16 28 35
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
bfc2a49 to
661e02aCompareJune 27, 2025 15:37
@ephraimbuddy
ephraimbuddy
marked this pull request as ready for review
June 27, 2025 15:38
jason810496
jason810496
reviewed
Jun 29, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. There are some questions about the usage of SchedulerDagBag in core-api.
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py
@bbovenzi
bbovenzi
commented
Jun 30, 2025

Copy link


Copy Markdown

Contributor
UI for dagrun clear and task instance clear look good. We'll add this to backfills next?
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
b7ab41b to
71ec6c1CompareJuly 14, 2025 18:49
jedcunningham
jedcunningham
approved these changes
Jul 21, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
71ec6c1 to
52a004eCompareJuly 21, 2025 20:31
ephraimbuddyand others
added 14 commits
July 22, 2025 10:59
@ephraimbuddy
Add run_on_latest_version support for backfill and clear operations
1109cad
With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.
When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.
Summary of changes:
- Use SchedulerDagBag instead of DagBag for execution API
- Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
- Add --run-on-latest-version CLI flag for backfill command
- Update backfill.py to support running tasks with latest DAG version
- Add UI checkbox for "Run with latest version" in clear dialogs
- Update SchedulerDagBag to handle latest version parameter
- Update API endpoints to support run_on_latest_version parameter
@ephraimbuddy
Only show the run on latest version button when we are on versioned b…
fafb744
…undle
@ephraimbuddy
Fix test
39ebf82
@ephraimbuddy
Fix test by using serialized dag
d561388
@ephraimbuddy
fixup! Fix test by using serialized dag
91d57c2
@ephraimbuddy
Mark the --run-on-latest-version as experimental
8655ca9
@ephraimbuddy
properly save dag to the db in tests
e13778b
@ephraimbuddy
Fix ui as taskinstance.dag_version is no longer nullable
ac6d4cd
@ephraimbuddy@jedcunningham
Apply suggestions from code review
120dbc6
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy@jedcunningham
Update airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
8ce6088
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Update airflow-core/src/airflow/ui/openapi-gen/requests/schema…
db415ce
…s.gen.ts
@ephraimbuddy
Do not show run on latest button if run was on latest
73eec26
@ephraimbuddy
Update doc
5d7dca3
@ephraimbuddy
fixup! Update doc
a6a1496
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
52a004e to
a6a1496CompareJuly 22, 2025 10:01
pierrejeambrun
pierrejeambrun
approved these changes
Jul 22, 2025
View reviewed changes
@pierrejeambrunpierrejeambrun
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!
@jedcunninghamjedcunningham
added this to the Airflow 3.1.0 milestone
Jul 22, 2025
@jedcunningham
jedcunningham
merged commit 3dc4cd2
into
apache:mainJul 22, 2025

102 checks passed

@jedcunningham
jedcunningham
deleted the

properly-clear-ti-specific-version

branch
July 22, 2025 17:12
kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 22, 2025
@kaxil
Update Python Client to 3.1.0rc1
e62c042
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@kaxilkaxil
mentioned this pull request
Oct 22, 2025
 Merged

kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 23, 2025
@kaxil
Update Python Client to 3.1.0rc1 (#138)
241405e
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@ogulcanaydoganogulcanaydogan
mentioned this pull request
Mar 19, 2026

Open

2 tasks
@imrichardwuimrichardwu
mentioned this pull request
Jul 31, 2026
 Merged

1 task
@dstandishdstandish
mentioned this pull request
Aug 11, 2026

Open

@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 24, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@kaxilkaxilkaxil left review comments
@bugraoz93bugraoz93bugraoz93 approved these changes
@jason810496jason810496jason810496 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@jscheffljschefflAwaiting requested review from jscheffl
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:Schedulerincluding HA (high availability) schedulerarea:translationsarea:UIRelated to UI/UX. For Frontend Developers.translation:default

Projects


None yet


Milestone

Airflow 3.1.0

8 participants

@ephraimbuddy@bbovenzi@kaxil@ashb@bugraoz93@pierrejeambrun@jedcunningham@jason810496

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add run_on_latest_version support for backfill and clear operations by ephraimbuddy · Pull Request #52177 · apache/airflow · GitHub
Skip to content
Add run_on_latest_version support for backfill and clear operations - #52177
Merged
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Jul 22, 2025
Merged
Add run_on_latest_version support for backfill and clear operations#52177
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Conversation
@ephraimbuddy
@ephraimbuddyephraimbuddy
commented
Jun 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
With this option, users are able to choose the dag version they want

to run their dag/task after clearing or when running backfill. This

only applies to versioned bundles as non-versioned bundles run with

the latest dag version.
When the user choose the run with latest version, the bundle_version

associated with the dagrun is updated to the latest and the associated

serialized dag version updated to the latest. Choosing not to run

with latest version which is the default means that the bundle version

and serialized dag version that the dag ran with initially would be used

in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run

with the latest version, otherwise it will run with the original bundle

the dagrun was created with. Note that it's only useful when rerunning

a dagrun using backfill. The default behaviour is using the initial bundle/version

and this is intentional otherwise running backfill will fail if there was

task rename in the latest version.
Summary of changes:
  • Use SchedulerDagBag instead of DagBag for execution API
  • Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
  • Add --run-on-latest-version CLI flag for backfill command
  • Update backfill.py to support running tasks with latest DAG version
  • Add UI checkbox for "Run with latest version" in clear dialogs
  • Update SchedulerDagBag to handle latest version parameter
  • Update API endpoints to support run_on_latest_version parameter
closes: #49007, closes: #49047
@boring-cyborgboring-cyborgBot
added

area:airflow-ctl

area:API
Airflow's REST/HTTP API
area:Scheduler
including HA (high availability) scheduler
area:translations

area:UI
Related to UI/UX. For Frontend Developers.
translation:default

labels
Jun 24, 2025
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
5 times, most recently
from
1f4b2e3 to
bfc2a49CompareJune 27, 2025 15:04
@ephraimbuddyephraimbuddy
changed the title
Fix clearing dags of versioned bundles and allow running on previous versionAdd run_on_latest_version support for backfill and clear operationsJun 27, 2025
@ephraimbuddy
ephraimbuddy
commented
Jun 27, 2025

Copy link


Copy Markdown

ContributorAuthor
Screenshot 2025-06-27 at 16 28 35
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
bfc2a49 to
661e02aCompareJune 27, 2025 15:37
@ephraimbuddy
ephraimbuddy
marked this pull request as ready for review
June 27, 2025 15:38
jason810496
jason810496
reviewed
Jun 29, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. There are some questions about the usage of SchedulerDagBag in core-api.
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py
@bbovenzi
bbovenzi
commented
Jun 30, 2025

Copy link


Copy Markdown

Contributor
UI for dagrun clear and task instance clear look good. We'll add this to backfills next?
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
b7ab41b to
71ec6c1CompareJuly 14, 2025 18:49
jedcunningham
jedcunningham
approved these changes
Jul 21, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
71ec6c1 to
52a004eCompareJuly 21, 2025 20:31
ephraimbuddyand others
added 14 commits
July 22, 2025 10:59
@ephraimbuddy
Add run_on_latest_version support for backfill and clear operations
1109cad
With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.
When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.
Summary of changes:
- Use SchedulerDagBag instead of DagBag for execution API
- Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
- Add --run-on-latest-version CLI flag for backfill command
- Update backfill.py to support running tasks with latest DAG version
- Add UI checkbox for "Run with latest version" in clear dialogs
- Update SchedulerDagBag to handle latest version parameter
- Update API endpoints to support run_on_latest_version parameter
@ephraimbuddy
Only show the run on latest version button when we are on versioned b…
fafb744
…undle
@ephraimbuddy
Fix test
39ebf82
@ephraimbuddy
Fix test by using serialized dag
d561388
@ephraimbuddy
fixup! Fix test by using serialized dag
91d57c2
@ephraimbuddy
Mark the --run-on-latest-version as experimental
8655ca9
@ephraimbuddy
properly save dag to the db in tests
e13778b
@ephraimbuddy
Fix ui as taskinstance.dag_version is no longer nullable
ac6d4cd
@ephraimbuddy@jedcunningham
Apply suggestions from code review
120dbc6
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy@jedcunningham
Update airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
8ce6088
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Update airflow-core/src/airflow/ui/openapi-gen/requests/schema…
db415ce
…s.gen.ts
@ephraimbuddy
Do not show run on latest button if run was on latest
73eec26
@ephraimbuddy
Update doc
5d7dca3
@ephraimbuddy
fixup! Update doc
a6a1496
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
52a004e to
a6a1496CompareJuly 22, 2025 10:01
pierrejeambrun
pierrejeambrun
approved these changes
Jul 22, 2025
View reviewed changes
@pierrejeambrunpierrejeambrun
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!
@jedcunninghamjedcunningham
added this to the Airflow 3.1.0 milestone
Jul 22, 2025
@jedcunningham
jedcunningham
merged commit 3dc4cd2
into
apache:mainJul 22, 2025

102 checks passed

@jedcunningham
jedcunningham
deleted the

properly-clear-ti-specific-version

branch
July 22, 2025 17:12
kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 22, 2025
@kaxil
Update Python Client to 3.1.0rc1
e62c042
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@kaxilkaxil
mentioned this pull request
Oct 22, 2025
 Merged

kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 23, 2025
@kaxil
Update Python Client to 3.1.0rc1 (#138)
241405e
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@ogulcanaydoganogulcanaydogan
mentioned this pull request
Mar 19, 2026

Open

2 tasks
@imrichardwuimrichardwu
mentioned this pull request
Jul 31, 2026
 Merged

1 task
@dstandishdstandish
mentioned this pull request
Aug 11, 2026

Open

@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 24, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@kaxilkaxilkaxil left review comments
@bugraoz93bugraoz93bugraoz93 approved these changes
@jason810496jason810496jason810496 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@jscheffljschefflAwaiting requested review from jscheffl
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:Schedulerincluding HA (high availability) schedulerarea:translationsarea:UIRelated to UI/UX. For Frontend Developers.translation:default

Projects


None yet


Milestone

Airflow 3.1.0

8 participants

@ephraimbuddy@bbovenzi@kaxil@ashb@bugraoz93@pierrejeambrun@jedcunningham@jason810496

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Add run_on_latest_version support for backfill and clear operations by ephraimbuddy · Pull Request #52177 · apache/airflow · GitHub
Skip to content
Add run_on_latest_version support for backfill and clear operations - #52177
Merged
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Jul 22, 2025
Merged
Add run_on_latest_version support for backfill and clear operations#52177
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Conversation
@ephraimbuddy
@ephraimbuddyephraimbuddy
commented
Jun 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
With this option, users are able to choose the dag version they want

to run their dag/task after clearing or when running backfill. This

only applies to versioned bundles as non-versioned bundles run with

the latest dag version.
When the user choose the run with latest version, the bundle_version

associated with the dagrun is updated to the latest and the associated

serialized dag version updated to the latest. Choosing not to run

with latest version which is the default means that the bundle version

and serialized dag version that the dag ran with initially would be used

in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run

with the latest version, otherwise it will run with the original bundle

the dagrun was created with. Note that it's only useful when rerunning

a dagrun using backfill. The default behaviour is using the initial bundle/version

and this is intentional otherwise running backfill will fail if there was

task rename in the latest version.
Summary of changes:
  • Use SchedulerDagBag instead of DagBag for execution API
  • Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
  • Add --run-on-latest-version CLI flag for backfill command
  • Update backfill.py to support running tasks with latest DAG version
  • Add UI checkbox for "Run with latest version" in clear dialogs
  • Update SchedulerDagBag to handle latest version parameter
  • Update API endpoints to support run_on_latest_version parameter
closes: #49007, closes: #49047
@boring-cyborgboring-cyborgBot
added

area:airflow-ctl

area:API
Airflow's REST/HTTP API
area:Scheduler
including HA (high availability) scheduler
area:translations

area:UI
Related to UI/UX. For Frontend Developers.
translation:default

labels
Jun 24, 2025
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
5 times, most recently
from
1f4b2e3 to
bfc2a49CompareJune 27, 2025 15:04
@ephraimbuddyephraimbuddy
changed the title
Fix clearing dags of versioned bundles and allow running on previous versionAdd run_on_latest_version support for backfill and clear operationsJun 27, 2025
@ephraimbuddy
ephraimbuddy
commented
Jun 27, 2025

Copy link


Copy Markdown

ContributorAuthor
Screenshot 2025-06-27 at 16 28 35
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
bfc2a49 to
661e02aCompareJune 27, 2025 15:37
@ephraimbuddy
ephraimbuddy
marked this pull request as ready for review
June 27, 2025 15:38
jason810496
jason810496
reviewed
Jun 29, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. There are some questions about the usage of SchedulerDagBag in core-api.
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py
@bbovenzi
bbovenzi
commented
Jun 30, 2025

Copy link


Copy Markdown

Contributor
UI for dagrun clear and task instance clear look good. We'll add this to backfills next?
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
b7ab41b to
71ec6c1CompareJuly 14, 2025 18:49
jedcunningham
jedcunningham
approved these changes
Jul 21, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
71ec6c1 to
52a004eCompareJuly 21, 2025 20:31
ephraimbuddyand others
added 14 commits
July 22, 2025 10:59
@ephraimbuddy
Add run_on_latest_version support for backfill and clear operations
1109cad
With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.
When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.
Summary of changes:
- Use SchedulerDagBag instead of DagBag for execution API
- Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
- Add --run-on-latest-version CLI flag for backfill command
- Update backfill.py to support running tasks with latest DAG version
- Add UI checkbox for "Run with latest version" in clear dialogs
- Update SchedulerDagBag to handle latest version parameter
- Update API endpoints to support run_on_latest_version parameter
@ephraimbuddy
Only show the run on latest version button when we are on versioned b…
fafb744
…undle
@ephraimbuddy
Fix test
39ebf82
@ephraimbuddy
Fix test by using serialized dag
d561388
@ephraimbuddy
fixup! Fix test by using serialized dag
91d57c2
@ephraimbuddy
Mark the --run-on-latest-version as experimental
8655ca9
@ephraimbuddy
properly save dag to the db in tests
e13778b
@ephraimbuddy
Fix ui as taskinstance.dag_version is no longer nullable
ac6d4cd
@ephraimbuddy@jedcunningham
Apply suggestions from code review
120dbc6
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy@jedcunningham
Update airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
8ce6088
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Update airflow-core/src/airflow/ui/openapi-gen/requests/schema…
db415ce
…s.gen.ts
@ephraimbuddy
Do not show run on latest button if run was on latest
73eec26
@ephraimbuddy
Update doc
5d7dca3
@ephraimbuddy
fixup! Update doc
a6a1496
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
52a004e to
a6a1496CompareJuly 22, 2025 10:01
pierrejeambrun
pierrejeambrun
approved these changes
Jul 22, 2025
View reviewed changes
@pierrejeambrunpierrejeambrun
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!
@jedcunninghamjedcunningham
added this to the Airflow 3.1.0 milestone
Jul 22, 2025
@jedcunningham
jedcunningham
merged commit 3dc4cd2
into
apache:mainJul 22, 2025

102 checks passed

@jedcunningham
jedcunningham
deleted the

properly-clear-ti-specific-version

branch
July 22, 2025 17:12
kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 22, 2025
@kaxil
Update Python Client to 3.1.0rc1
e62c042
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@kaxilkaxil
mentioned this pull request
Oct 22, 2025
 Merged

kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 23, 2025
@kaxil
Update Python Client to 3.1.0rc1 (#138)
241405e
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@ogulcanaydoganogulcanaydogan
mentioned this pull request
Mar 19, 2026

Open

2 tasks
@imrichardwuimrichardwu
mentioned this pull request
Jul 31, 2026
 Merged

1 task
@dstandishdstandish
mentioned this pull request
Aug 11, 2026

Open

@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 24, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@kaxilkaxilkaxil left review comments
@bugraoz93bugraoz93bugraoz93 approved these changes
@jason810496jason810496jason810496 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@jscheffljschefflAwaiting requested review from jscheffl
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:Schedulerincluding HA (high availability) schedulerarea:translationsarea:UIRelated to UI/UX. For Frontend Developers.translation:default

Projects


None yet


Milestone

Airflow 3.1.0

8 participants

@ephraimbuddy@bbovenzi@kaxil@ashb@bugraoz93@pierrejeambrun@jedcunningham@jason810496

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add run_on_latest_version support for backfill and clear operations by ephraimbuddy · Pull Request #52177 · apache/airflow · GitHub
Skip to content
Add run_on_latest_version support for backfill and clear operations - #52177
Merged
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Jul 22, 2025
Merged
Add run_on_latest_version support for backfill and clear operations#52177
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Conversation
@ephraimbuddy
@ephraimbuddyephraimbuddy
commented
Jun 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
With this option, users are able to choose the dag version they want

to run their dag/task after clearing or when running backfill. This

only applies to versioned bundles as non-versioned bundles run with

the latest dag version.
When the user choose the run with latest version, the bundle_version

associated with the dagrun is updated to the latest and the associated

serialized dag version updated to the latest. Choosing not to run

with latest version which is the default means that the bundle version

and serialized dag version that the dag ran with initially would be used

in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run

with the latest version, otherwise it will run with the original bundle

the dagrun was created with. Note that it's only useful when rerunning

a dagrun using backfill. The default behaviour is using the initial bundle/version

and this is intentional otherwise running backfill will fail if there was

task rename in the latest version.
Summary of changes:
  • Use SchedulerDagBag instead of DagBag for execution API
  • Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
  • Add --run-on-latest-version CLI flag for backfill command
  • Update backfill.py to support running tasks with latest DAG version
  • Add UI checkbox for "Run with latest version" in clear dialogs
  • Update SchedulerDagBag to handle latest version parameter
  • Update API endpoints to support run_on_latest_version parameter
closes: #49007, closes: #49047
@boring-cyborgboring-cyborgBot
added

area:airflow-ctl

area:API
Airflow's REST/HTTP API
area:Scheduler
including HA (high availability) scheduler
area:translations

area:UI
Related to UI/UX. For Frontend Developers.
translation:default

labels
Jun 24, 2025
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
5 times, most recently
from
1f4b2e3 to
bfc2a49CompareJune 27, 2025 15:04
@ephraimbuddyephraimbuddy
changed the title
Fix clearing dags of versioned bundles and allow running on previous versionAdd run_on_latest_version support for backfill and clear operationsJun 27, 2025
@ephraimbuddy
ephraimbuddy
commented
Jun 27, 2025

Copy link


Copy Markdown

ContributorAuthor
Screenshot 2025-06-27 at 16 28 35
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
bfc2a49 to
661e02aCompareJune 27, 2025 15:37
@ephraimbuddy
ephraimbuddy
marked this pull request as ready for review
June 27, 2025 15:38
jason810496
jason810496
reviewed
Jun 29, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. There are some questions about the usage of SchedulerDagBag in core-api.
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py
@bbovenzi
bbovenzi
commented
Jun 30, 2025

Copy link


Copy Markdown

Contributor
UI for dagrun clear and task instance clear look good. We'll add this to backfills next?
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
b7ab41b to
71ec6c1CompareJuly 14, 2025 18:49
jedcunningham
jedcunningham
approved these changes
Jul 21, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
71ec6c1 to
52a004eCompareJuly 21, 2025 20:31
ephraimbuddyand others
added 14 commits
July 22, 2025 10:59
@ephraimbuddy
Add run_on_latest_version support for backfill and clear operations
1109cad
With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.
When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.
Summary of changes:
- Use SchedulerDagBag instead of DagBag for execution API
- Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
- Add --run-on-latest-version CLI flag for backfill command
- Update backfill.py to support running tasks with latest DAG version
- Add UI checkbox for "Run with latest version" in clear dialogs
- Update SchedulerDagBag to handle latest version parameter
- Update API endpoints to support run_on_latest_version parameter
@ephraimbuddy
Only show the run on latest version button when we are on versioned b…
fafb744
…undle
@ephraimbuddy
Fix test
39ebf82
@ephraimbuddy
Fix test by using serialized dag
d561388
@ephraimbuddy
fixup! Fix test by using serialized dag
91d57c2
@ephraimbuddy
Mark the --run-on-latest-version as experimental
8655ca9
@ephraimbuddy
properly save dag to the db in tests
e13778b
@ephraimbuddy
Fix ui as taskinstance.dag_version is no longer nullable
ac6d4cd
@ephraimbuddy@jedcunningham
Apply suggestions from code review
120dbc6
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy@jedcunningham
Update airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
8ce6088
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Update airflow-core/src/airflow/ui/openapi-gen/requests/schema…
db415ce
…s.gen.ts
@ephraimbuddy
Do not show run on latest button if run was on latest
73eec26
@ephraimbuddy
Update doc
5d7dca3
@ephraimbuddy
fixup! Update doc
a6a1496
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
52a004e to
a6a1496CompareJuly 22, 2025 10:01
pierrejeambrun
pierrejeambrun
approved these changes
Jul 22, 2025
View reviewed changes
@pierrejeambrunpierrejeambrun
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!
@jedcunninghamjedcunningham
added this to the Airflow 3.1.0 milestone
Jul 22, 2025
@jedcunningham
jedcunningham
merged commit 3dc4cd2
into
apache:mainJul 22, 2025

102 checks passed

@jedcunningham
jedcunningham
deleted the

properly-clear-ti-specific-version

branch
July 22, 2025 17:12
kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 22, 2025
@kaxil
Update Python Client to 3.1.0rc1
e62c042
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@kaxilkaxil
mentioned this pull request
Oct 22, 2025
 Merged

kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 23, 2025
@kaxil
Update Python Client to 3.1.0rc1 (#138)
241405e
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@ogulcanaydoganogulcanaydogan
mentioned this pull request
Mar 19, 2026

Open

2 tasks
@imrichardwuimrichardwu
mentioned this pull request
Jul 31, 2026
 Merged

1 task
@dstandishdstandish
mentioned this pull request
Aug 11, 2026

Open

@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 24, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@kaxilkaxilkaxil left review comments
@bugraoz93bugraoz93bugraoz93 approved these changes
@jason810496jason810496jason810496 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@jscheffljschefflAwaiting requested review from jscheffl
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:Schedulerincluding HA (high availability) schedulerarea:translationsarea:UIRelated to UI/UX. For Frontend Developers.translation:default

Projects


None yet


Milestone

Airflow 3.1.0

8 participants

@ephraimbuddy@bbovenzi@kaxil@ashb@bugraoz93@pierrejeambrun@jedcunningham@jason810496

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add run_on_latest_version support for backfill and clear operations by ephraimbuddy · Pull Request #52177 · apache/airflow · GitHub
Skip to content
Add run_on_latest_version support for backfill and clear operations - #52177
Merged
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Jul 22, 2025
Merged
Add run_on_latest_version support for backfill and clear operations#52177
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Conversation
@ephraimbuddy
@ephraimbuddyephraimbuddy
commented
Jun 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
With this option, users are able to choose the dag version they want

to run their dag/task after clearing or when running backfill. This

only applies to versioned bundles as non-versioned bundles run with

the latest dag version.
When the user choose the run with latest version, the bundle_version

associated with the dagrun is updated to the latest and the associated

serialized dag version updated to the latest. Choosing not to run

with latest version which is the default means that the bundle version

and serialized dag version that the dag ran with initially would be used

in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run

with the latest version, otherwise it will run with the original bundle

the dagrun was created with. Note that it's only useful when rerunning

a dagrun using backfill. The default behaviour is using the initial bundle/version

and this is intentional otherwise running backfill will fail if there was

task rename in the latest version.
Summary of changes:
  • Use SchedulerDagBag instead of DagBag for execution API
  • Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
  • Add --run-on-latest-version CLI flag for backfill command
  • Update backfill.py to support running tasks with latest DAG version
  • Add UI checkbox for "Run with latest version" in clear dialogs
  • Update SchedulerDagBag to handle latest version parameter
  • Update API endpoints to support run_on_latest_version parameter
closes: #49007, closes: #49047
@boring-cyborgboring-cyborgBot
added

area:airflow-ctl

area:API
Airflow's REST/HTTP API
area:Scheduler
including HA (high availability) scheduler
area:translations

area:UI
Related to UI/UX. For Frontend Developers.
translation:default

labels
Jun 24, 2025
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
5 times, most recently
from
1f4b2e3 to
bfc2a49CompareJune 27, 2025 15:04
@ephraimbuddyephraimbuddy
changed the title
Fix clearing dags of versioned bundles and allow running on previous versionAdd run_on_latest_version support for backfill and clear operationsJun 27, 2025
@ephraimbuddy
ephraimbuddy
commented
Jun 27, 2025

Copy link


Copy Markdown

ContributorAuthor
Screenshot 2025-06-27 at 16 28 35
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
bfc2a49 to
661e02aCompareJune 27, 2025 15:37
@ephraimbuddy
ephraimbuddy
marked this pull request as ready for review
June 27, 2025 15:38
jason810496
jason810496
reviewed
Jun 29, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. There are some questions about the usage of SchedulerDagBag in core-api.
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py
@bbovenzi
bbovenzi
commented
Jun 30, 2025

Copy link


Copy Markdown

Contributor
UI for dagrun clear and task instance clear look good. We'll add this to backfills next?
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
b7ab41b to
71ec6c1CompareJuly 14, 2025 18:49
jedcunningham
jedcunningham
approved these changes
Jul 21, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
71ec6c1 to
52a004eCompareJuly 21, 2025 20:31
ephraimbuddyand others
added 14 commits
July 22, 2025 10:59
@ephraimbuddy
Add run_on_latest_version support for backfill and clear operations
1109cad
With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.
When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.
Summary of changes:
- Use SchedulerDagBag instead of DagBag for execution API
- Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
- Add --run-on-latest-version CLI flag for backfill command
- Update backfill.py to support running tasks with latest DAG version
- Add UI checkbox for "Run with latest version" in clear dialogs
- Update SchedulerDagBag to handle latest version parameter
- Update API endpoints to support run_on_latest_version parameter
@ephraimbuddy
Only show the run on latest version button when we are on versioned b…
fafb744
…undle
@ephraimbuddy
Fix test
39ebf82
@ephraimbuddy
Fix test by using serialized dag
d561388
@ephraimbuddy
fixup! Fix test by using serialized dag
91d57c2
@ephraimbuddy
Mark the --run-on-latest-version as experimental
8655ca9
@ephraimbuddy
properly save dag to the db in tests
e13778b
@ephraimbuddy
Fix ui as taskinstance.dag_version is no longer nullable
ac6d4cd
@ephraimbuddy@jedcunningham
Apply suggestions from code review
120dbc6
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy@jedcunningham
Update airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
8ce6088
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Update airflow-core/src/airflow/ui/openapi-gen/requests/schema…
db415ce
…s.gen.ts
@ephraimbuddy
Do not show run on latest button if run was on latest
73eec26
@ephraimbuddy
Update doc
5d7dca3
@ephraimbuddy
fixup! Update doc
a6a1496
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
52a004e to
a6a1496CompareJuly 22, 2025 10:01
pierrejeambrun
pierrejeambrun
approved these changes
Jul 22, 2025
View reviewed changes
@pierrejeambrunpierrejeambrun
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!
@jedcunninghamjedcunningham
added this to the Airflow 3.1.0 milestone
Jul 22, 2025
@jedcunningham
jedcunningham
merged commit 3dc4cd2
into
apache:mainJul 22, 2025

102 checks passed

@jedcunningham
jedcunningham
deleted the

properly-clear-ti-specific-version

branch
July 22, 2025 17:12
kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 22, 2025
@kaxil
Update Python Client to 3.1.0rc1
e62c042
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@kaxilkaxil
mentioned this pull request
Oct 22, 2025
 Merged

kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 23, 2025
@kaxil
Update Python Client to 3.1.0rc1 (#138)
241405e
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@ogulcanaydoganogulcanaydogan
mentioned this pull request
Mar 19, 2026

Open

2 tasks
@imrichardwuimrichardwu
mentioned this pull request
Jul 31, 2026
 Merged

1 task
@dstandishdstandish
mentioned this pull request
Aug 11, 2026

Open

@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 24, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@kaxilkaxilkaxil left review comments
@bugraoz93bugraoz93bugraoz93 approved these changes
@jason810496jason810496jason810496 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@jscheffljschefflAwaiting requested review from jscheffl
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:Schedulerincluding HA (high availability) schedulerarea:translationsarea:UIRelated to UI/UX. For Frontend Developers.translation:default

Projects


None yet


Milestone

Airflow 3.1.0

8 participants

@ephraimbuddy@bbovenzi@kaxil@ashb@bugraoz93@pierrejeambrun@jedcunningham@jason810496

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Add run_on_latest_version support for backfill and clear operations by ephraimbuddy · Pull Request #52177 · apache/airflow · GitHub
Skip to content
Add run_on_latest_version support for backfill and clear operations - #52177
Merged
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Jul 22, 2025
Merged
Add run_on_latest_version support for backfill and clear operations#52177
jedcunningham merged 14 commits into
apache:mainfrom 
astronomer:properly-clear-ti-specific-version
Conversation
@ephraimbuddy
@ephraimbuddyephraimbuddy
commented
Jun 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
With this option, users are able to choose the dag version they want

to run their dag/task after clearing or when running backfill. This

only applies to versioned bundles as non-versioned bundles run with

the latest dag version.
When the user choose the run with latest version, the bundle_version

associated with the dagrun is updated to the latest and the associated

serialized dag version updated to the latest. Choosing not to run

with latest version which is the default means that the bundle version

and serialized dag version that the dag ran with initially would be used

in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run

with the latest version, otherwise it will run with the original bundle

the dagrun was created with. Note that it's only useful when rerunning

a dagrun using backfill. The default behaviour is using the initial bundle/version

and this is intentional otherwise running backfill will fail if there was

task rename in the latest version.
Summary of changes:
  • Use SchedulerDagBag instead of DagBag for execution API
  • Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
  • Add --run-on-latest-version CLI flag for backfill command
  • Update backfill.py to support running tasks with latest DAG version
  • Add UI checkbox for "Run with latest version" in clear dialogs
  • Update SchedulerDagBag to handle latest version parameter
  • Update API endpoints to support run_on_latest_version parameter
closes: #49007, closes: #49047
@boring-cyborgboring-cyborgBot
added

area:airflow-ctl

area:API
Airflow's REST/HTTP API
area:Scheduler
including HA (high availability) scheduler
area:translations

area:UI
Related to UI/UX. For Frontend Developers.
translation:default

labels
Jun 24, 2025
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
5 times, most recently
from
1f4b2e3 to
bfc2a49CompareJune 27, 2025 15:04
@ephraimbuddyephraimbuddy
changed the title
Fix clearing dags of versioned bundles and allow running on previous versionAdd run_on_latest_version support for backfill and clear operationsJun 27, 2025
@ephraimbuddy
ephraimbuddy
commented
Jun 27, 2025

Copy link


Copy Markdown

ContributorAuthor
Screenshot 2025-06-27 at 16 28 35
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
bfc2a49 to
661e02aCompareJune 27, 2025 15:37
@ephraimbuddy
ephraimbuddy
marked this pull request as ready for review
June 27, 2025 15:38
jason810496
jason810496
reviewed
Jun 29, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. There are some questions about the usage of SchedulerDagBag in core-api.
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/app.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Comment threadairflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py
@bbovenzi
bbovenzi
commented
Jun 30, 2025

Copy link


Copy Markdown

Contributor
UI for dagrun clear and task instance clear look good. We'll add this to backfills next?
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
b7ab41b to
71ec6c1CompareJuly 14, 2025 18:49
jedcunningham
jedcunningham
approved these changes
Jul 21, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
Comment threadairflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
71ec6c1 to
52a004eCompareJuly 21, 2025 20:31
ephraimbuddyand others
added 14 commits
July 22, 2025 10:59
@ephraimbuddy
Add run_on_latest_version support for backfill and clear operations
1109cad
With this option, users are able to choose the dag version they want
to run their dag/task after clearing or when running backfill. This
only applies to versioned bundles as non-versioned bundles run with
the latest dag version.
When the user choose the run with latest version, the bundle_version
associated with the dagrun is updated to the latest and the associated
serialized dag version updated to the latest. Choosing not to run
with latest version which is the default means that the bundle version
and serialized dag version that the dag ran with initially would be used
in running it again.
For backfill, there's now --run-on-latest-version flag that makes it run
with the latest version, otherwise it will run with the original bundle
the dagrun was created with. Note that it's only useful when rerunning
a dagrun using backfill. The default behaviour is using the initial bundle/version
and this is intentional otherwise running backfill will fail if there was
task rename in the latest version.
Summary of changes:
- Use SchedulerDagBag instead of DagBag for execution API
- Add run_on_latest_version field to DAGRunClearBody and ClearTaskInstancesBody models
- Add --run-on-latest-version CLI flag for backfill command
- Update backfill.py to support running tasks with latest DAG version
- Add UI checkbox for "Run with latest version" in clear dialogs
- Update SchedulerDagBag to handle latest version parameter
- Update API endpoints to support run_on_latest_version parameter
@ephraimbuddy
Only show the run on latest version button when we are on versioned b…
fafb744
…undle
@ephraimbuddy
Fix test
39ebf82
@ephraimbuddy
Fix test by using serialized dag
d561388
@ephraimbuddy
fixup! Fix test by using serialized dag
91d57c2
@ephraimbuddy
Mark the --run-on-latest-version as experimental
8655ca9
@ephraimbuddy
properly save dag to the db in tests
e13778b
@ephraimbuddy
Fix ui as taskinstance.dag_version is no longer nullable
ac6d4cd
@ephraimbuddy@jedcunningham
Apply suggestions from code review
120dbc6
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy@jedcunningham
Update airflow-core/src/airflow/ui/openapi-gen/requests/schemas.gen.ts
8ce6088
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@ephraimbuddy
fixup! Update airflow-core/src/airflow/ui/openapi-gen/requests/schema…
db415ce
…s.gen.ts
@ephraimbuddy
Do not show run on latest button if run was on latest
73eec26
@ephraimbuddy
Update doc
5d7dca3
@ephraimbuddy
fixup! Update doc
a6a1496
@ephraimbuddy
ephraimbuddyforce-pushed
the
properly-clear-ti-specific-version
branch
from
52a004e to
a6a1496CompareJuly 22, 2025 10:01
pierrejeambrun
pierrejeambrun
approved these changes
Jul 22, 2025
View reviewed changes
@pierrejeambrunpierrejeambrun
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!
@jedcunninghamjedcunningham
added this to the Airflow 3.1.0 milestone
Jul 22, 2025
@jedcunningham
jedcunningham
merged commit 3dc4cd2
into
apache:mainJul 22, 2025

102 checks passed

@jedcunningham
jedcunningham
deleted the

properly-clear-ti-specific-version

branch
July 22, 2025 17:12
kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 22, 2025
@kaxil
Update Python Client to 3.1.0rc1
e62c042
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@kaxilkaxil
mentioned this pull request
Oct 22, 2025
 Merged

kaxil added a commit
to apache/airflow-client-python
that referenced
this pull request
Oct 23, 2025
@kaxil
Update Python Client to 3.1.0rc1 (#138)
241405e
(from https://github.com/apache/airflow/tree/python-client/3.1.0rc1)
## New Features:
- Add `map_index` filter to TaskInstance API queries ([#55614](apache/airflow#55614))
- Add `has_import_errors` filter to Core API GET /dags endpoint ([#54563](apache/airflow#54563))
- Add `dag_version` filter to get_dag_runs endpoint ([#54882](apache/airflow#54882))
- Implement pattern search for event log endpoint ([#55114](apache/airflow#55114))
- Add asset-based filtering support to DAG API endpoint ([#54263](apache/airflow#54263))
- Add Greater Than and Less Than range filters to DagRuns and Task Instance list ([#54302](apache/airflow#54302))
- Add `try_number` as filter to task instances ([#54695](apache/airflow#54695))
- Add filters to Browse XComs endpoint ([#54049](apache/airflow#54049))
- Add Filtering by DAG Bundle Name and Version to API routes ([#54004](apache/airflow#54004))
- Add search filter for DAG runs by triggering user name ([#53652](apache/airflow#53652))
- Enable multi sorting (AIP-84) ([#53408](apache/airflow#53408))
- Add `run_on_latest_version` support for backfill and clear operations ([#52177](apache/airflow#52177))
- Add `run_id_pattern` search for Dag Run API ([#52437](apache/airflow#52437))
- Add tracking of triggering user to Dag runs ([#51738](apache/airflow#51738))
- Expose DAG parsing duration in the API ([#54752](apache/airflow#54752))
## New API Endpoints:
- Add Human-in-the-Loop (HITL) endpoints for approval workflows ([#52868](apache/airflow#52868), [#53373](apache/airflow#53373), [#53376](apache/airflow#53376), [#53885](apache/airflow#53885), [#53923](apache/airflow#53923), [#54308](apache/airflow#54308), [#54310](apache/airflow#54310), [#54723](apache/airflow#54723), [#54773](apache/airflow#54773), [#55019](apache/airflow#55019), [#55463](apache/airflow#55463), [#55525](apache/airflow#55525), [#55535](apache/airflow#55535), [#55603](apache/airflow#55603), [#55776](apache/airflow#55776))
- Add endpoint to watch dag run until finish ([#51920](apache/airflow#51920))
- Add TI bulk actions endpoint ([#50443](apache/airflow#50443))
- Add Keycloak Refresh Token Endpoint ([#51657](apache/airflow#51657))
## Deprecations:
- Mark `DagDetailsResponse.concurrency` as deprecated ([#55150](apache/airflow#55150))
## Bug Fixes:
- Fix dag import error modal pagination ([#55719](apache/airflow#55719))
@ogulcanaydoganogulcanaydogan
mentioned this pull request
Mar 19, 2026

Open

2 tasks
@imrichardwuimrichardwu
mentioned this pull request
Jul 31, 2026
 Merged

1 task
@dstandishdstandish
mentioned this pull request
Aug 11, 2026

Open

@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 24, 2026
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@kaxilkaxilkaxil left review comments
@bugraoz93bugraoz93bugraoz93 approved these changes
@jason810496jason810496jason810496 left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@jscheffljschefflAwaiting requested review from jscheffl
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels

area:airflow-ctlarea:APIAirflow's REST/HTTP APIarea:Schedulerincluding HA (high availability) schedulerarea:translationsarea:UIRelated to UI/UX. For Frontend Developers.translation:default

Projects


None yet


Milestone

Airflow 3.1.0

8 participants

@ephraimbuddy@bbovenzi@kaxil@ashb@bugraoz93@pierrejeambrun@jedcunningham@jason810496