Skip to content

Add CLI support for bulk pause and resume of DAGs - #38265

Merged
vincbeck merged 4 commits into
apache:mainfrom
shahar1:pause-multiple-dags
Mar 20, 2024
Merged

Add CLI support for bulk pause and resume of DAGs#38265
vincbeck merged 4 commits into
apache:mainfrom
shahar1:pause-multiple-dags

Conversation

@shahar1

@shahar1shahar1 commented Mar 18, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

closes: #36956

This PR adds the ability for airflow dags (un)pause commands to treat dag_id as a regex (similar to backfill), so we could apply it to multiple DAGs.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@shahar1shahar1 mentioned this pull request Mar 18, 2024
Open
@shahar1
shahar1force-pushed the pause-multiple-dags branch 4 times, most recently from b9761e6 to e5aa9acCompareMarch 18, 2024 16:48
@shahar1

shahar1 commented Mar 18, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Following @eladkal's comment in #36994, I added a confirmation prompt to avoid accidental (un)pausing. I also formatted the output with AirflowConsole().print_as to make it easier to read.

@shahar1
shahar1force-pushed the pause-multiple-dags branch from e5aa9ac to 0d6a43fCompareMarch 18, 2024 17:44
@shahar1
shahar1 marked this pull request as draft March 18, 2024 20:20
@shahar1
shahar1force-pushed the pause-multiple-dags branch from 0d6a43f to db3a942CompareMarch 18, 2024 21:29
@shahar1
shahar1 marked this pull request as ready for review March 18, 2024 21:32
dirrao
dirrao reviewed Mar 19, 2024
View reviewed changes
Comment threadairflow/cli/commands/dag_command.py Outdated
@shahar1
shahar1force-pushed the pause-multiple-dags branch 3 times, most recently from 509bf86 to afa2bb6CompareMarch 19, 2024 06:27
vincbeck
vincbeck reviewed Mar 19, 2024
View reviewed changes
Comment threadairflow/cli/cli_config.py Outdated
@shahar1
shahar1force-pushed the pause-multiple-dags branch from afa2bb6 to dc3bee0CompareMarch 19, 2024 17:30
@shahar1
shahar1 requested a review from vincbeckMarch 19, 2024 17:32
vincbeck
vincbeck reviewed Mar 19, 2024
View reviewed changes
Comment threadairflow/cli/cli_config.py Outdated
@shahar1
shahar1 requested a review from vincbeckMarch 20, 2024 09:27
@vincbeck
vincbeck merged commit fecc1ed into apache:mainMar 20, 2024
@eladkaleladkal added this to the Airflow 2.9.0 milestone Mar 22, 2024
@eladkaleladkal added the type:improvement Changelog: Improvements label Mar 22, 2024
ephraimbuddy
ephraimbuddy reviewed Mar 27, 2024
View reviewed changes

@ephraimbuddyephraimbuddy left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like unpausing a single dag is now broken by this PR. I have not tested bulk pause/unpause but for one dag, it seems broken

Ok. It now errors when the dag is paused and you are pausing again. I think it should just exit without error if DAG is already paused

@shahar1

shahar1 commented Mar 28, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

It seems like unpausing a single dag is now broken by this PR. I have not tested bulk pause/unpause but for one dag, it seems broken

Ok. It now errors when the dag is paused and you are pausing again. I think it should just exit without error if DAG is already paused

Apologies for the confusion, maybe raising an error for pausing an already paused dag (and vice versa) is indeed too aggressive - I'll try to come up with a fix later on.

@ephraimbuddy

ephraimbuddy commented Mar 28, 2024

Copy link
Copy Markdown
Contributor

It seems like unpausing a single dag is now broken by this PR. I have not tested bulk pause/unpause but for one dag, it seems broken
Ok. It now errors when the dag is paused and you are pausing again. I think it should just exit without error if DAG is already paused

Apologies for the confusion, maybe raising an error for pausing an already paused dag (and vice versa) is indeed too aggressive - I'll try to come up with a fix later on.

Added the change here: #38585. Take a look when you have time

@shahar1
shahar1 deleted the pause-multiple-dags branch June 12, 2024 13:13
@boushphongboushphong mentioned this pull request Jun 20, 2024
Merged
ephraimbuddy pushed a commit that referenced this pull request Jul 1, 2024
@boushphong
Fix dag (un)pausing won't work on environment where dag files are mis…
e3d62c3
…sing (#40345)
closes: #38834#38265 added bulk pause and resume of DAGs. However, this PR seems to reuse the cli util method that collects DAGs from the default dag folder but not from the metadata DB. Hence, this would cause the unpause command to fail on environments where the dag folder is missing.
utkarsharma2 pushed a commit that referenced this pull request Jul 2, 2024
@boushphong@utkarsharma2
Fix dag (un)pausing won't work on environment where dag files are mis…
1630faa
…sing (#40345)
closes: #38834#38265 added bulk pause and resume of DAGs. However, this PR seems to reuse the cli util method that collects DAGs from the default dag folder but not from the metadata DB. Hence, this would cause the unpause command to fail on environments where the dag folder is missing.
(cherry picked from commit e3d62c3)
ephraimbuddy pushed a commit that referenced this pull request Jul 2, 2024
@boushphong@ephraimbuddy
Fix dag (un)pausing won't work on environment where dag files are mis…
23ca6fa
…sing (#40345)
closes: #38834#38265 added bulk pause and resume of DAGs. However, this PR seems to reuse the cli util method that collects DAGs from the default dag folder but not from the metadata DB. Hence, this would cause the unpause command to fail on environments where the dag folder is missing.
(cherry picked from commit e3d62c3)
@boushphongboushphong mentioned this pull request Jul 21, 2024
Closed
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
@boushphong@romsharon98
Fix dag (un)pausing won't work on environment where dag files are mis…
a80a9ef
…sing (apache#40345)
closes: apache#38834apache#38265 added bulk pause and resume of DAGs. However, this PR seems to reuse the cli util method that collects DAGs from the default dag folder but not from the metadata DB. Hence, this would cause the unpause command to fail on environments where the dag folder is missing.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ephraimbuddyephraimbuddyephraimbuddy left review comments
@dirraodirraodirrao left review comments
@vincbeckvincbeckvincbeck approved these changes

Assignees

No one assigned

Labels

area:CLItype:improvementChangelog: Improvements

Projects

None yet

Milestone

Airflow 2.9.0

Development

Successfully merging this pull request may close these issues.

Add CLI support for bulk pause and resume of DAGs

5 participants

@shahar1@ephraimbuddy@dirrao@vincbeck@eladkal