Explicitly rollback on task state update exception by uranusjr · Pull Request #71076 · apache/airflow

uranusjr · 2026-08-04T09:32:05Z

One small gap discovered when analyzing #70951.

This ensures everything is reset when something fails, including asset registration db calls previously made.

Lee-W

should we add unit test to it?

This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr · 2026-08-04T09:57:31Z

Test added

github-actions · 2026-08-04T11:18:19Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions · 2026-08-04T11:18:47Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 64002a4 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

uranusjr · 2026-08-04T11:19:36Z

I’ll simply add this to #71065.

…71065) * [v3-3-test] Release TI lock before asset listener callbacks (#70951) * Release TI lock before asset listener callbacks Asset registration on the task-success path (ti_update_state) ran the listener hooks synchronously inside the transaction holding a row lock on the task_instance table. A slow listener, multiplied across a large fan-out of asset events, could hold that lock for minutes, causing statement timeouts. The listener hooks are now deferred until the end of the endpoint instead of executed inline during asset event creation. Registration writes to the database still happen under the caller's transaction, so durability is unchanged; this only moves the best-effort listener hooks off the lock. * Optimize asset alias assoc insert * Fix exhausted iterator reuse bug * Test asset reg callback cases (cherry picked from commit 79db995) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * [v3-3-test] Explicitly rollback on task state update exception (#71076) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

uranusjr · 2026-08-04T09:32:05Z

One small gap discovered when analyzing #70951.

This ensures everything is reset when something fails, including asset registration db calls previously made.

Lee-W

should we add unit test to it?

This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr · 2026-08-04T09:57:31Z

Test added

github-actions · 2026-08-04T11:18:19Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions · 2026-08-04T11:18:47Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 64002a4 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

uranusjr · 2026-08-04T11:19:36Z

I’ll simply add this to #71065.

…71065) * [v3-3-test] Release TI lock before asset listener callbacks (#70951) * Release TI lock before asset listener callbacks Asset registration on the task-success path (ti_update_state) ran the listener hooks synchronously inside the transaction holding a row lock on the task_instance table. A slow listener, multiplied across a large fan-out of asset events, could hold that lock for minutes, causing statement timeouts. The listener hooks are now deferred until the end of the endpoint instead of executed inline during asset event creation. Registration writes to the database still happen under the caller's transaction, so durability is unchanged; this only moves the best-effort listener hooks off the lock. * Optimize asset alias assoc insert * Fix exhausted iterator reuse bug * Test asset reg callback cases (cherry picked from commit 79db995) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * [v3-3-test] Explicitly rollback on task state update exception (#71076) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

uranusjr · 2026-08-04T09:32:05Z

One small gap discovered when analyzing #70951.

This ensures everything is reset when something fails, including asset registration db calls previously made.

Lee-W

should we add unit test to it?

This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr · 2026-08-04T09:57:31Z

Test added

github-actions · 2026-08-04T11:18:19Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions · 2026-08-04T11:18:47Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 64002a4 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

uranusjr · 2026-08-04T11:19:36Z

I’ll simply add this to #71065.

…71065) * [v3-3-test] Release TI lock before asset listener callbacks (#70951) * Release TI lock before asset listener callbacks Asset registration on the task-success path (ti_update_state) ran the listener hooks synchronously inside the transaction holding a row lock on the task_instance table. A slow listener, multiplied across a large fan-out of asset events, could hold that lock for minutes, causing statement timeouts. The listener hooks are now deferred until the end of the endpoint instead of executed inline during asset event creation. Registration writes to the database still happen under the caller's transaction, so durability is unchanged; this only moves the best-effort listener hooks off the lock. * Optimize asset alias assoc insert * Fix exhausted iterator reuse bug * Test asset reg callback cases (cherry picked from commit 79db995) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * [v3-3-test] Explicitly rollback on task state update exception (#71076) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

uranusjr · 2026-08-04T09:32:05Z

One small gap discovered when analyzing #70951.

This ensures everything is reset when something fails, including asset registration db calls previously made.

Lee-W

should we add unit test to it?

This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr · 2026-08-04T09:57:31Z

Test added

github-actions · 2026-08-04T11:18:19Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions · 2026-08-04T11:18:47Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 64002a4 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

uranusjr · 2026-08-04T11:19:36Z

I’ll simply add this to #71065.

…71065) * [v3-3-test] Release TI lock before asset listener callbacks (#70951) * Release TI lock before asset listener callbacks Asset registration on the task-success path (ti_update_state) ran the listener hooks synchronously inside the transaction holding a row lock on the task_instance table. A slow listener, multiplied across a large fan-out of asset events, could hold that lock for minutes, causing statement timeouts. The listener hooks are now deferred until the end of the endpoint instead of executed inline during asset event creation. Registration writes to the database still happen under the caller's transaction, so durability is unchanged; this only moves the best-effort listener hooks off the lock. * Optimize asset alias assoc insert * Fix exhausted iterator reuse bug * Test asset reg callback cases (cherry picked from commit 79db995) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * [v3-3-test] Explicitly rollback on task state update exception (#71076) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

uranusjr · 2026-08-04T09:32:05Z

One small gap discovered when analyzing #70951.

This ensures everything is reset when something fails, including asset registration db calls previously made.

Lee-W

should we add unit test to it?

This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr · 2026-08-04T09:57:31Z

Test added

github-actions · 2026-08-04T11:18:19Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions · 2026-08-04T11:18:47Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 64002a4 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

uranusjr · 2026-08-04T11:19:36Z

I’ll simply add this to #71065.

…71065) * [v3-3-test] Release TI lock before asset listener callbacks (#70951) * Release TI lock before asset listener callbacks Asset registration on the task-success path (ti_update_state) ran the listener hooks synchronously inside the transaction holding a row lock on the task_instance table. A slow listener, multiplied across a large fan-out of asset events, could hold that lock for minutes, causing statement timeouts. The listener hooks are now deferred until the end of the endpoint instead of executed inline during asset event creation. Registration writes to the database still happen under the caller's transaction, so durability is unchanged; this only moves the best-effort listener hooks off the lock. * Optimize asset alias assoc insert * Fix exhausted iterator reuse bug * Test asset reg callback cases (cherry picked from commit 79db995) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * [v3-3-test] Explicitly rollback on task state update exception (#71076) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

uranusjr · 2026-08-04T09:32:05Z

One small gap discovered when analyzing #70951.

This ensures everything is reset when something fails, including asset registration db calls previously made.

Lee-W

should we add unit test to it?

This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr · 2026-08-04T09:57:31Z

Test added

github-actions · 2026-08-04T11:18:19Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions · 2026-08-04T11:18:47Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 64002a4 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

uranusjr · 2026-08-04T11:19:36Z

I’ll simply add this to #71065.

…71065) * [v3-3-test] Release TI lock before asset listener callbacks (#70951) * Release TI lock before asset listener callbacks Asset registration on the task-success path (ti_update_state) ran the listener hooks synchronously inside the transaction holding a row lock on the task_instance table. A slow listener, multiplied across a large fan-out of asset events, could hold that lock for minutes, causing statement timeouts. The listener hooks are now deferred until the end of the endpoint instead of executed inline during asset event creation. Registration writes to the database still happen under the caller's transaction, so durability is unchanged; this only moves the best-effort listener hooks off the lock. * Optimize asset alias assoc insert * Fix exhausted iterator reuse bug * Test asset reg callback cases (cherry picked from commit 79db995) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * [v3-3-test] Explicitly rollback on task state update exception (#71076) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

uranusjr · 2026-08-04T09:32:05Z

One small gap discovered when analyzing #70951.

This ensures everything is reset when something fails, including asset registration db calls previously made.

Lee-W

should we add unit test to it?

This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr · 2026-08-04T09:57:31Z

Test added

github-actions · 2026-08-04T11:18:19Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions · 2026-08-04T11:18:47Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 64002a4 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

uranusjr · 2026-08-04T11:19:36Z

I’ll simply add this to #71065.

…71065) * [v3-3-test] Release TI lock before asset listener callbacks (#70951) * Release TI lock before asset listener callbacks Asset registration on the task-success path (ti_update_state) ran the listener hooks synchronously inside the transaction holding a row lock on the task_instance table. A slow listener, multiplied across a large fan-out of asset events, could hold that lock for minutes, causing statement timeouts. The listener hooks are now deferred until the end of the endpoint instead of executed inline during asset event creation. Registration writes to the database still happen under the caller's transaction, so durability is unchanged; this only moves the best-effort listener hooks off the lock. * Optimize asset alias assoc insert * Fix exhausted iterator reuse bug * Test asset reg callback cases (cherry picked from commit 79db995) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * [v3-3-test] Explicitly rollback on task state update exception (#71076) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

uranusjr · 2026-08-04T09:32:05Z

One small gap discovered when analyzing #70951.

This ensures everything is reset when something fails, including asset registration db calls previously made.

Lee-W

should we add unit test to it?

This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr · 2026-08-04T09:57:31Z

Test added

github-actions · 2026-08-04T11:18:19Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions · 2026-08-04T11:18:47Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 64002a4 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

uranusjr · 2026-08-04T11:19:36Z

I’ll simply add this to #71065.

…71065) * [v3-3-test] Release TI lock before asset listener callbacks (#70951) * Release TI lock before asset listener callbacks Asset registration on the task-success path (ti_update_state) ran the listener hooks synchronously inside the transaction holding a row lock on the task_instance table. A slow listener, multiplied across a large fan-out of asset events, could hold that lock for minutes, causing statement timeouts. The listener hooks are now deferred until the end of the endpoint instead of executed inline during asset event creation. Registration writes to the database still happen under the caller's transaction, so durability is unchanged; this only moves the best-effort listener hooks off the lock. * Optimize asset alias assoc insert * Fix exhausted iterator reuse bug * Test asset reg callback cases (cherry picked from commit 79db995) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * [v3-3-test] Explicitly rollback on task state update exception (#71076) --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

uranusjr requested review from amoghrajesh, ashb and kaxil as code owners August 4, 2026 09:32

boring-cyborgBot added the area:task-sdk label Aug 4, 2026

uranusjr mentioned this pull request Aug 4, 2026
[v3-3-test] Release TI lock before asset listener callbacks (#70951) #71065
Merged

Lee-W approved these changes Aug 4, 2026
View reviewed changes

Lee-W added the backport-to-v3-3-test Backport to v3-3-test label Aug 4, 2026

Explicitly rollback on task state update exception
2482b71
This ensures everything is reset when something fails, including asset registration db calls previously made.

uranusjr force-pushed the task-state-tighten branch from c12e418 to 2482b71Compare August 4, 2026 09:57

Lee-W approved these changes Aug 4, 2026
View reviewed changes

uranusjr merged commit 64002a4 into apache:mainAug 4, 2026
79 checks passed

uranusjr deleted the task-state-tighten branch August 4, 2026 11:17

github-actionsBot added this to the Airflow 3.3.2 milestone Aug 4, 2026

uranusjr added a commit that referenced this pull request Aug 4, 2026
Explicitly rollback on task state update exception (#71076)
2efe792

uranusjr added a commit that referenced this pull request Aug 4, 2026
[v3-3-test] Explicitly rollback on task state update exception (#71076)
b5e4f91

vatsrahul1001 modified the milestones: Airflow 3.3.2, Airflow 3.3.1 Aug 4, 2026

dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
Explicitly rollback on task state update exception (apache#71076)
e306556