Skip to content

Make dag_version_id in TI non-nullable - #50825

Merged
ephraimbuddy merged 39 commits into
apache:mainfrom
astronomer:make-ti-dag-version-id-non-nullable2
Jul 8, 2025
Merged

Make dag_version_id in TI non-nullable#50825
ephraimbuddy merged 39 commits into
apache:mainfrom
astronomer:make-ti-dag-version-id-non-nullable2

Conversation

@ephraimbuddy

@ephraimbuddyephraimbuddy commented May 20, 2025

Copy link
Copy Markdown
Contributor

Enforces serialization of DAGs before creating DagRuns and TaskInstances.

The changes include:

  • Making dag_version_id a required parameter in TaskInstance constructor
  • Migrating the dag_version_id column in TaskInstance to be non-nullable
  • Adding a check in _create_orm_dagrun to raise AirflowException if dag is not serialized
  • Updating tests to ensure DAGs are properly serialized before creating DagRuns

This prevents running runs with missing serialized DAG versions.

@ephraimbuddy
ephraimbuddyforce-pushed the make-ti-dag-version-id-non-nullable2 branch from 20c6e75 to 08bfa28CompareMay 20, 2025 17:14
@ephraimbuddy
ephraimbuddyforce-pushed the make-ti-dag-version-id-non-nullable2 branch from 9d3fc74 to 63eb72bCompareMay 21, 2025 15:28
@ephraimbuddy
ephraimbuddyforce-pushed the make-ti-dag-version-id-non-nullable2 branch 2 times, most recently from aad0a5f to 80439a1CompareMay 21, 2025 17:35
uranusjr
uranusjr reviewed May 22, 2025
View reviewed changes
Comment threadairflow-core/docs/migrations-ref.rst Outdated
uranusjr
uranusjr reviewed May 22, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/cli/commands/task_command.py
@ephraimbuddy
ephraimbuddyforce-pushed the make-ti-dag-version-id-non-nullable2 branch 2 times, most recently from 5409839 to 616e0a1CompareMay 24, 2025 22:18
@ephraimbuddy
ephraimbuddyforce-pushed the make-ti-dag-version-id-non-nullable2 branch from 616e0a1 to 070f7a3CompareMay 25, 2025 10:55
@ephraimbuddy
ephraimbuddyforce-pushed the make-ti-dag-version-id-non-nullable2 branch from 564ac88 to f3c6444CompareJuly 8, 2025 15:28
@ephraimbuddy
ephraimbuddy merged commit 7ea7fcd into apache:mainJul 8, 2025
193 of 194 checks passed
@ephraimbuddy
ephraimbuddy deleted the make-ti-dag-version-id-non-nullable2 branch July 8, 2025 17:55
HsiuChuanHsu pushed a commit to HsiuChuanHsu/airflow that referenced this pull request Jul 10, 2025
@ephraimbuddy@HsiuChuanHsu
Make dag_version_id in TI non-nullable (apache#50825)
2086f2d
* Make dag_version_id in TI non-nullable
Enforces proper serialization of DAGs before creating DagRuns and TaskInstances.
The changes include:
- Making dag_version_id a required parameter in TaskInstance constructor
- Migrating the dag_version_id column in TaskInstance to be non-nullable
- Adding a check in _create_orm_dagrun to raise AirflowException if dag is not serialized
- Updating tests to ensure DAGs are properly serialized before creating DagRuns
This prevents running runs with missing serialized DAG versions.
* fixup! Make dag_version_id in TI non-nullable
* Fix tests for some providers
* fixup! Fix tests for some providers
* fixup! fixup! Fix tests for some providers
* more test fixes
* fixup! more test fixes
* fixup! fixup! more test fixes
* Update more tests
* fixup! Update more tests
* fixup! fixup! Update more tests
* fixup! fixup! fixup! Update more tests
* fixup! fixup! fixup! fixup! Update more tests
* fixup! fixup! fixup! fixup! fixup! Update more tests
* Add compat for MockedTaskInstance
* fixup! Add compat for MockedTaskInstance
* fixup! fixup! Add compat for MockedTaskInstance
* fix system test
* fixup! fix system test
* A fix for kube integration tests
* Update the expected TIs from test multiple versions number file
* fix incorrect imports
* fix time delta async test
* Remove non working kube test fix
* Fix task sdk test
* Add cadwyn migration and fix some tests
* fix some provider tests
* Add data migration before migrating dag_version_id
* Update test, TaskInstanceResponse and fix assert dag_version in task_command.py
* Resolve conflicts
* Fix test
* Fix more tests
* fixup! fixup! Fix more tests
* Fix bad rebase
* update fab www-hash
* Fix conflicts
* Fix MappedTaskInstancedata
* add dag_version_id to a new test
* Fix fab www hash
stephen-bracken pushed a commit to stephen-bracken/airflow that referenced this pull request Jul 15, 2025
@ephraimbuddyBracken, Stephen (London)
Make dag_version_id in TI non-nullable (apache#50825)
961bf5b
* Make dag_version_id in TI non-nullable
Enforces proper serialization of DAGs before creating DagRuns and TaskInstances.
The changes include:
- Making dag_version_id a required parameter in TaskInstance constructor
- Migrating the dag_version_id column in TaskInstance to be non-nullable
- Adding a check in _create_orm_dagrun to raise AirflowException if dag is not serialized
- Updating tests to ensure DAGs are properly serialized before creating DagRuns
This prevents running runs with missing serialized DAG versions.
* fixup! Make dag_version_id in TI non-nullable
* Fix tests for some providers
* fixup! Fix tests for some providers
* fixup! fixup! Fix tests for some providers
* more test fixes
* fixup! more test fixes
* fixup! fixup! more test fixes
* Update more tests
* fixup! Update more tests
* fixup! fixup! Update more tests
* fixup! fixup! fixup! Update more tests
* fixup! fixup! fixup! fixup! Update more tests
* fixup! fixup! fixup! fixup! fixup! Update more tests
* Add compat for MockedTaskInstance
* fixup! Add compat for MockedTaskInstance
* fixup! fixup! Add compat for MockedTaskInstance
* fix system test
* fixup! fix system test
* A fix for kube integration tests
* Update the expected TIs from test multiple versions number file
* fix incorrect imports
* fix time delta async test
* Remove non working kube test fix
* Fix task sdk test
* Add cadwyn migration and fix some tests
* fix some provider tests
* Add data migration before migrating dag_version_id
* Update test, TaskInstanceResponse and fix assert dag_version in task_command.py
* Resolve conflicts
* Fix test
* Fix more tests
* fixup! fixup! Fix more tests
* Fix bad rebase
* update fab www-hash
* Fix conflicts
* Fix MappedTaskInstancedata
* add dag_version_id to a new test
* Fix fab www hash
@kaxilkaxil mentioned this pull request Jul 23, 2025
Closed
kaxil
kaxil reviewed Jul 23, 2025
View reviewed changes
Comment threadairflow-core/tests/unit/cli/commands/test_task_command.py
Comment on lines +280 to +282
dag = DagBag().get_dag("test_mapped_classic")
dag.sync_to_db()
SerializedDagModel.write_dag(dag, bundle_name="testing")

@kaxilkaxilJul 23, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has introduced a bug. This was the clue where we had to change the test

@kaxilkaxil mentioned this pull request Jul 24, 2025
Merged
@amoghrajeshamoghrajesh mentioned this pull request Aug 11, 2025
Closed
ephraimbuddy added a commit to astronomer/airflow that referenced this pull request Aug 11, 2025
@ephraimbuddy
Partially revertapache#50825on database level
2cdd207
This is alternative to apache#53820. Here we make the TI.dag_version_id
nullable on the database level. it's still enforced in code
@dheerajturagadheerajturaga mentioned this pull request Feb 1, 2026
Closed
1 task
@multimericmultimeric mentioned this pull request Mar 23, 2026
Closed
2 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@uranusjruranusjruranusjr left review comments
@kaxilkaxilkaxil left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun left review comments
@dstandishdstandishdstandish approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@ashbashbAwaiting requested review from ashbashb is a code owner
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
@hussein-awalahussein-awalaAwaiting requested review from hussein-awalahussein-awala is a code owner
@eladkaleladkalAwaiting requested review from eladkal
@o-nikolaso-nikolasAwaiting requested review from o-nikolaso-nikolas is a code owner
@mobuchowskimobuchowskiAwaiting requested review from mobuchowskimobuchowski is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner
@mik-lajmik-lajAwaiting requested review from mik-laj
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bbovenzibbovenziAwaiting requested review from bbovenzibbovenzi is a code owner
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamiltonryanahamilton is a code owner
@shubhamraj-gitshubhamraj-gitAwaiting requested review from shubhamraj-gitshubhamraj-git is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@rawwarrawwarAwaiting requested review from rawwarrawwar is a code owner
@vincbeckvincbeckAwaiting requested review from vincbeck

Assignees

@ephraimbuddyephraimbuddy

Labels

area:APIAirflow's REST/HTTP APIarea:CLIarea:db-migrationsPRs with DB migrationarea:Schedulerincluding HA (high availability) schedulerarea:serializationarea:task-sdkarea:Triggererfull tests neededWe need to run full set of tests for this PR to mergekind:documentation

Projects

None yet

Milestone

Airflow 3.1.0

Development

Successfully merging this pull request may close these issues.

7 participants

@ephraimbuddy@pierrejeambrun@uranusjr@kaxil@dstandish@jedcunningham@phanikumv