Speed up Trigger.clean_unused query to prevent Triggerer crashes by AntonioBergonzi · Pull Request #68244 · apache/airflow

AntonioBergonzi · 2026-06-08T16:55:37Z

Description

Changed the query that checks for triggers that have no more callbacks, assets or task instances associated to them from count + aggregate to anti join on task_instance.
The result is the same (we are checking for non existence) but the query is more efficient. Also added skip locked since we run multiple triggerers and the different queries interfere with each other.

I did not add tests since I'm not introducing a new functionality, so the current one should cover the code change. If you think additional tests are needed, please let me know.

Testing

Plans of the queries

OLD QUERY:

EXPLAIN
DELETE
FROM trigger
WHERE trigger.id IN (SELECT trigger.id
FROM trigger
LEFT OUTER JOIN task_instance ON trigger.id = task_instance.trigger_id
WHERE NOT (EXISTS (SELECT 1
FROM asset_watcher
WHERE trigger.id = asset_watcher.trigger_id
AND (EXISTS (SELECT 1
FROM asset
WHERE asset.id = asset_watcher.asset_id)))
)
AND NOT (EXISTS (SELECT 1
FROM callback
WHERE trigger.id = callback.trigger_id))
GROUP BY trigger.id
HAVING count(task_instance.trigger_id) = 0);

yields

-[ RECORD 1 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Delete on trigger (cost=49.31..93458.14 rows=0 width=0)
-[ RECORD 2 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Nested Loop (cost=49.31..93458.14 rows=1 width=34)
-[ RECORD 3 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Subquery Scan on "ANY_subquery" (cost=49.04..93455.59 rows=1 width=32)
-[ RECORD 4 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> GroupAggregate (cost=49.04..93455.58 rows=1 width=4)
-[ RECORD 5 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Group Key: trigger_1.id
-[ RECORD 6 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Filter: (count(task_instance.trigger_id) = 0)
-[ RECORD 7 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Nested Loop Left Join (cost=49.04..192.44 rows=18652448 width=8)
-[ RECORD 8 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Nested Loop Anti Join (cost=48.61..72.06 rows=72 width=4)
-[ RECORD 9 ]---------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Merge Anti Join (cost=48.46..48.83 rows=72 width=4)
-[ RECORD 10 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Merge Cond: (trigger_1.id = asset_watcher.trigger_id)
-[ RECORD 11 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Sort (cost=45.94..46.12 rows=72 width=4)
-[ RECORD 12 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Sort Key: trigger_1.id
-[ RECORD 13 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Seq Scan on trigger trigger_1 (cost=0.00..43.72 rows=72 width=4)
-[ RECORD 14 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Sort (cost=2.52..2.52 rows=1 width=4)
-[ RECORD 15 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Sort Key: asset_watcher.trigger_id
-[ RECORD 16 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Nested Loop (cost=0.29..2.51 rows=1 width=4)
-[ RECORD 17 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Seq Scan on asset_watcher (cost=0.00..0.00 rows=1 width=8)
-[ RECORD 18 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Index Only Scan using asset_pkey on asset (cost=0.29..2.51 rows=1 width=4)
-[ RECORD 19 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Index Cond: (id = asset_watcher.asset_id)
-[ RECORD 20 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Index Only Scan using idx_callback_trigger_id on callback (cost=0.15..0.32 rows=1 width=4)
-[ RECORD 21 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Index Cond: (trigger_id = trigger_1.id)
-[ RECORD 22 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Index Only Scan using ti_trigger_id on task_instance (cost=0.44..1.66 rows=1 width=4)
-[ RECORD 23 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Index Cond: (trigger_id = trigger_1.id)
-[ RECORD 24 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Index Scan using trigger_pkey on trigger (cost=0.27..2.49 rows=1 width=10)
-[ RECORD 25 ]--------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Index Cond: (id = "ANY_subquery".id)

NEW QUERY:

EXPLAIN
DELETE
FROM trigger
WHERE trigger.id IN (SELECT trigger.id
FROM trigger
WHERE NOT (EXISTS (SELECT 1
FROM asset_watcher
WHERE trigger.id = asset_watcher.trigger_id
AND (EXISTS (SELECT 1
FROM asset
WHERE asset.id = asset_watcher.asset_id)))
)
AND NOT (EXISTS (SELECT 1
FROM callback
WHERE trigger.id = callback.trigger_id))
AND NOT (EXISTS (SELECT 1
FROM task_instance
WHERE trigger.id = task_instance.trigger_id)) FOR
UPDATE SKIP LOCKED
);

yields

-[ RECORD 1 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Delete on trigger (cost=328.23..372.97 rows=0 width=0)
-[ RECORD 2 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Hash Semi Join (cost=328.23..372.97 rows=73 width=34)
-[ RECORD 3 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Hash Cond: (trigger.id = "ANY_subquery".id)
-[ RECORD 4 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Seq Scan on trigger (cost=0.00..43.73 rows=73 width=10)
-[ RECORD 5 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Hash (cost=327.32..327.32 rows=73 width=32)
-[ RECORD 6 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Subquery Scan on "ANY_subquery" (cost=0.88..327.32 rows=73 width=32)
-[ RECORD 7 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> LockRows (cost=0.88..326.59 rows=73 width=34)
-[ RECORD 8 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Nested Loop Anti Join (cost=0.88..325.86 rows=73 width=34)
-[ RECORD 9 ]-----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Nested Loop Anti Join (cost=0.44..129.83 rows=73 width=28)
-[ RECORD 10 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Nested Loop Anti Join (cost=0.29..47.33 rows=73 width=22)
-[ RECORD 11 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Join Filter: (trigger_1.id = asset_watcher.trigger_id)
-[ RECORD 12 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Seq Scan on trigger trigger_1 (cost=0.00..43.73 rows=73 width=10)
-[ RECORD 13 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Materialize (cost=0.29..2.51 rows=1 width=16)
-[ RECORD 14 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Nested Loop (cost=0.29..2.51 rows=1 width=16)
-[ RECORD 15 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Seq Scan on asset_watcher (cost=0.00..0.00 rows=1 width=14)
-[ RECORD 16 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Index Scan using asset_pkey on asset (cost=0.29..2.51 rows=1 width=10)
-[ RECORD 17 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Index Cond: (id = asset_watcher.asset_id)
-[ RECORD 18 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Index Scan using idx_callback_trigger_id on callback (cost=0.15..1.12 rows=1 width=10)
-[ RECORD 19 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Index Cond: (trigger_id = trigger_1.id)
-[ RECORD 20 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | -> Index Scan using ti_trigger_id on task_instance (cost=0.44..2.66 rows=1 width=10)
-[ RECORD 21 ]----------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN | Index Cond: (trigger_id = trigger_1.id)

Also we stopped having queries that crashed the triggerer

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Claude Sonnet 1M

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

boring-cyborg · 2026-06-08T16:55:42Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Copilot

Pull request overview

This PR optimizes Trigger.clean_unused() by switching from an OUTER JOIN + GROUP BY/HAVING count-based approach to a NOT EXISTS-style anti-join when identifying unused triggers, and adds row-level locking with SKIP LOCKED to reduce interference when multiple triggerers are running concurrently.

Changes:

Replace JOIN/aggregate “no TaskInstance rows” check with ~cls.task_instance.has() (anti-join / NOT EXISTS semantics).
Add SKIP LOCKED row locking to avoid concurrent triggerers contending for the same trigger rows during cleanup.

hussein-awala

LGTM, I went through this carefully since it rewrites the delete predicate, and I'm satisfied it's a safe, well-motivated optimization.

For SKIP LOCKED: Good call for multi-triggerer, a single clean_unused() pass may now skip rows locked by a concurrent triggerer and leave them for the next run instead of always sweeping everything, which is the right trade-off.

ashb

Code looks okay, please update the PR title to say what it fixes/does, not what change it makes to the code.

(See the guide in our PT template on writing commit messages, that applies to PR title too.)

… + aggregate

… between concurrent triggerer pods

AntonioBergonzi · 2026-07-27T08:53:26Z

please update the PR title to say what it fixes/does, not what change it makes to the code.

Good point! Done 😄

boring-cyborg · 2026-07-29T08:39:03Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

github-actions · 2026-07-29T08:43:23Z

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-3-test

hussein-awala · 2026-07-29T08:55:27Z

Congrats @AntonioBergonzi on your first commit 🎉

…crashes (#68244) (#70668) * Use NOT EXISTS anti-join in Trigger.clean_unused instead of LEFT JOIN + aggregate * add FOR UPDATE SKIP LOCKED to Trigger.clean_unused to avoid deadlocks between concurrent triggerer pods * use with_row_locks helper and scope lock to trigger table (cherry picked from commit 2c1035a) Co-authored-by: AntonioBergonzi <48454032+AntonioBergonzi@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

…che#68244) * Use NOT EXISTS anti-join in Trigger.clean_unused instead of LEFT JOIN + aggregate * add FOR UPDATE SKIP LOCKED to Trigger.clean_unused to avoid deadlocks between concurrent triggerer pods * use with_row_locks helper and scope lock to trigger table

boring-cyborgBot added the area:Triggerer label Jun 8, 2026

BeauDevCode mentioned this pull request Jun 9, 2026
Batch triggerer unused trigger cleanup deletes #68269
Closed

AntonioBergonzi changed the title ~~Use NOT EXISTS anti-join in Trigger.clean_unused instead of aggregate~~Use NOT EXISTS anti-join in Trigger.clean_unused and add index in callback tableJun 12, 2026

AntonioBergonzi force-pushed the ab/improve_slow_delete branch from c0f8d0c to 725b143Compare June 12, 2026 13:56

AntonioBergonzi changed the title ~~Use NOT EXISTS anti-join in Trigger.clean_unused and add index in callback table~~Use NOT EXISTS anti-join in Trigger.clean_unused instead of aggregatingJun 12, 2026

AntonioBergonzi changed the title ~~Use NOT EXISTS anti-join in Trigger.clean_unused instead of aggregating~~Use NOT EXISTS anti-join in Trigger.clean_unused and added skip lockedJun 12, 2026

AntonioBergonzi marked this pull request as ready for review June 12, 2026 14:06

AntonioBergonzi requested review from XD-DENG and ashb as code owners June 12, 2026 14:06

hussein-awala requested review from Copilot and hussein-awala June 12, 2026 15:43

Copilotstarted reviewing on behalf of hussein-awala June 12, 2026 15:44 View session

CopilotAI reviewed Jun 12, 2026
View reviewed changes

Comment threadairflow-core/src/airflow/models/trigger.py Outdated

hussein-awala approved these changes Jun 15, 2026
View reviewed changes

hussein-awala requested review from amoghrajesh, dstandish and jscheffl June 15, 2026 14:42

hussein-awala added the type:improvement Changelog: Improvements label Jun 17, 2026

hussein-awala added this to the Airflow 3.3.0 milestone Jun 17, 2026

hussein-awala added the ready for maintainer review Set after triaging when all criteria pass. label Jun 17, 2026

vatsrahul1001 modified the milestones: Airflow 3.3.0, Airflow 3.3.1 Jun 25, 2026

ashb reviewed Jul 24, 2026
View reviewed changes

vatsrahul1001 added the backport-to-v3-3-test Backport to v3-3-test label Jul 24, 2026

AntonioBergonzi added 3 commits July 24, 2026 21:09

Use NOT EXISTS anti-join in Trigger.clean_unused instead of LEFT JOIN…
02c5397
… + aggregate

add FOR UPDATE SKIP LOCKED to Trigger.clean_unused to avoid deadlocks…
b5dffee
… between concurrent triggerer pods

use with_row_locks helper and scope lock to trigger table
f894790

eladkal force-pushed the ab/improve_slow_delete branch from ab47768 to f894790Compare July 24, 2026 18:09

AntonioBergonzi changed the title ~~Use NOT EXISTS anti-join in Trigger.clean_unused and added skip locked~~Fix Triggerer crashes caused by slow Trigger.clean_unused query Jul 27, 2026

AntonioBergonzi changed the title ~~Fix Triggerer crashes caused by slow Trigger.clean_unused query~~ Speed up Trigger.clean_unused query to prevent Triggerer crashes Jul 27, 2026

hussein-awala requested a review from ashb July 27, 2026 09:32

vatsrahul1001 approved these changes Jul 29, 2026
View reviewed changes

vatsrahul1001 merged commit 2c1035a into apache:mainJul 29, 2026
78 checks passed

shivaam mentioned this pull request Aug 3, 2026
Batch triggerer unused trigger cleanup deletes #70961
Open
1 task

This was referenced Aug 25, 2026
Trigger.clean_unused() on MySQL can cascade-delete the task_instance row of a task that defers during the sweep #72061
Open
Fix deferred task instances being cascade-deleted by Trigger.clean_unused on MySQL #72062
Open

Navigation Menu

Uh oh!

Speed up Trigger.clean_unused query to prevent Triggerer crashes - #68244

Speed up Trigger.clean_unused query to prevent Triggerer crashes #68244
vatsrahul1001 merged 3 commits into
apache:mainfrom
AntonioBergonzi:ab/improve_slow_delete

AntonioBergonzi commented Jun 8, 2026•
edited
Loading

Uh oh!

boring-cyborgBot commented Jun 8, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

Uh oh!

hussein-awala left a comment

Uh oh!

ashb left a comment

Uh oh!

AntonioBergonzi commented Jul 27, 2026

Uh oh!

Uh oh!

boring-cyborgBot commented Jul 29, 2026

Uh oh!

github-actionsBot commented Jul 29, 2026

Uh oh!

hussein-awala commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

AntonioBergonzi commented Jun 8, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborgBot commented Jun 8, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

hussein-awala left a comment

Choose a reason for hiding this comment

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

AntonioBergonzi commented Jul 27, 2026

Uh oh!

Uh oh!

boring-cyborgBot commented Jul 29, 2026

Uh oh!

github-actionsBot commented Jul 29, 2026

Backport successfully created: v3-3-test

Uh oh!

hussein-awala commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AntonioBergonzi commented Jun 8, 2026•
edited
Loading