Respect run on latest version when clearing a running Dag run

[ephraimbuddy]

run_on_latest_version should have the same meaning for running and queued Dag runs, and when callers preserve a finished run's state. Otherwise cleared running tasks can retry against stale code while the Dag run still points at an older serialized Dag or bundle.

Keeping the run, cleared task instances, integrity checks, and bundle selection aligned preserves the user's explicit rerun choice without rewriting unrelated task-version history.

---

Was generative AI tooling used to co-author this PR?

• Yes — Codex (5.6 Sol)

Generated-by: Codex (5.6 Sol) following the guidelines

[pierrejeambrun]

LGTM, but would love another pair of eyes.

Just one nit

[dstandish]

Hi Ephraim.

Want to flag a design concern

The field created_dag_version_id has existed as a column since AIP-65 DAG versioning (#42913/#43735). #49097 added the ORM relationship (created_dag_version) on top of it, with a docstring restating what the name already communicated: "the dag version column that was in effect at dag run creation time."

Documented or not, the name itself was always the contract — a write-once historical fact, not a live pointer. Per a conversation with @jedcunningham (the original author of AIP-65), that was intentional: bundle-versioned (pinned) runs were always meant to execute the version pinned at creation, with no path to move off it. "Run with the latest code" was supposed to be achieved by not using a versioned bundle — unpinned bundles already always resolve to the latest DagVersion (DBDagBag._version_from_dag_run).

Starting with #54984 and extended through #59764 and #65835/#66901, created_dag_version_id started being mutated after creation to support run_on_latest_version for pinned runs — a capability the original design didn't intend to exist for that case. This PR extends the same mutation to running/queued runs, going further in that direction.

That mutation has already produced concrete, reachable bugs elsewhere in the codebase, because other code correctly assumed the field's original (immutable) contract:

#71454 — DagRun.dag_versions silently drops versions still in use by task instances that weren't part of a given clear.
#71455 — only_new clear can report zero new tasks when there genuinely are some, for the same reason.
I've filed #71453 to track the underlying contract problem across all of this.

Given the original design intent, I think this is worth pausing on: should run_on_latest_version apply to bundle-versioned/pinned runs at all, rather than being extended further here? If the answer is yes and this is a deliberate, considered departure from the original pinning guarantee, it should probably written down somewhere (and the field's contract/docs updated to match), rather than continuing to build on a field whose name and documentation still say something the code no longer does.

[ephraimbuddy]

Hi Ephraim.

Want to flag a design concern

The field created_dag_version_id has existed as a column since AIP-65 DAG versioning (#42913/#43735). #49097 added the ORM relationship (created_dag_version) on top of it, with a docstring restating what the name already communicated: "the dag version column that was in effect at dag run creation time."

Documented or not, the name itself was always the contract — a write-once historical fact, not a live pointer. Per a conversation with @jedcunningham (the original author of AIP-65), that was intentional: bundle-versioned (pinned) runs were always meant to execute the version pinned at creation, with no path to move off it. "Run with the latest code" was supposed to be achieved by not using a versioned bundle — unpinned bundles already always resolve to the latest DagVersion (DBDagBag._version_from_dag_run).

Starting with #54984 and extended through #59764 and #65835/#66901, created_dag_version_id started being mutated after creation to support run_on_latest_version for pinned runs — a capability the original design didn't intend to exist for that case. This PR extends the same mutation to running/queued runs, going further in that direction.

That mutation has already produced concrete, reachable bugs elsewhere in the codebase, because other code correctly assumed the field's original (immutable) contract:

#71454 — DagRun.dag_versions silently drops versions still in use by task instances that weren't part of a given clear. #71455 — only_new clear can report zero new tasks when there genuinely are some, for the same reason. I've filed #71453 to track the underlying contract problem across all of this.

Given the original design intent, I think this is worth pausing on: should run_on_latest_version apply to bundle-versioned/pinned runs at all, rather than being extended further here? If the answer is yes and this is a deliberate, considered departure from the original pinning guarantee, it should probably written down somewhere (and the field's contract/docs updated to match), rather than continuing to build on a field whose name and documentation still say something the code no longer does.

Thanks for raising this. I agree that the name and documentation of created_dag_version_id no longer match its behavior, and that should be resolved.

I don’t think that should block this PR, though. #71425 does not introduce the mutation for queued/running Dag runs: main already updates created_dag_version_id and bundle_version in that path following #65835, which was backported by #66901. This PR makes the existing behavior consistent by also moving a running TI that becomes RESTARTING, calling verify_integrity, and applying the update when dag_run_state=False.

The #71455 repro also does not reproduce on the finished-run path as written. That path calls verify_integrity, which creates the TI for task C during the first clear, so the later only_new result is correctly empty. There is a reachable queued/running variant because main currently moves the run pointer without verify_integrity; #71425 fixes that inconsistency.

I agree #71454 is valid. DagRun.dag_versions should report versions represented by the run’s TI/TIH rows rather than assume that a versioned run only contains its current pointer. Likewise, _get_new_task_ids should determine missing tasks from the run’s actual TI rows.

On the broader design question, run_on_latest_version is now a released user-facing capability across the clear UI/API, backfills, configuration, and the Dag-level parameter. The API still marks it experimental, but removing it from versioned bundles would nevertheless be a separate user-visible design decision. Disabling versioning is not equivalent: users can reasonably want runs pinned by default while deliberately rerunning a failed task against fixed code.

From the consumers I checked, the execution paths use created_dag_version_id as the version the run currently resolves to. I have not found a consumer that requires immutable creation-time provenance. Adding another column would also have unclear upgrade semantics because _update_dagrun_to_latest_version can rewrite all current TI version IDs without creating TI history, meaning the original value may already be unrecoverable.

My preference is therefore to land #71425 to restore the existing invariants, fix #71454 and _get_new_task_ids from the TI/TIH source of truth, and use #71453 to settle the documentation and possible physical rename separately.

---

Drafted-by: Codex (5.6 Sol); reviewed by @ephraimbuddy before posting