Skip to content

Add dry run for backfill - #45062

Merged
phanikumv merged 26 commits into
apache:mainfrom
astronomer:backfill_dry_run
Jan 12, 2025
Merged

Add dry run for backfill#45062
phanikumv merged 26 commits into
apache:mainfrom
astronomer:backfill_dry_run

Conversation

@prabhusneha

@prabhusnehaprabhusneha commented Dec 18, 2024

Copy link
Copy Markdown
Contributor

Closes#44395

Response:
image

@boring-cyborgboring-cyborgBot added the area:UI Related to UI/UX. For Frontend Developers. label Dec 18, 2024
@prabhusneha
prabhusneha marked this pull request as ready for review December 19, 2024 13:55
pierrejeambrun
pierrejeambrun reviewed Dec 19, 2024
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed with daniel, maybe a separate endpoint makes more sense to avoid mixed returned type BackfillResponse | BackfillDryRunResponse on the same endpoint. That's hard to handle for clients.

bbovenzi
bbovenzi reviewed Dec 19, 2024
View reviewed changes
Comment threadairflow/api_fastapi/core_api/datamodels/backfills.py
pierrejeambrun
pierrejeambrun reviewed Dec 20, 2024
View reviewed changes
Comment threadairflow/api_fastapi/core_api/routes/public/backfills.py Outdated
@prabhusneha

prabhusneha commented Dec 23, 2024

Copy link
Copy Markdown
ContributorAuthor

As discussed with daniel, maybe a separate endpoint makes more sense to avoid mixed returned type BackfillResponse | BackfillDryRunResponse on the same endpoint. That's hard to handle for clients.

Created a separate endpoint for dry run.

pierrejeambrun
pierrejeambrun reviewed Dec 25, 2024
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looking nice.

A few improvement suggestions.

Comment threadairflow/api_fastapi/core_api/routes/public/backfills.py Outdated
Comment threadairflow/api_fastapi/core_api/routes/public/backfills.py Outdated
Comment threadairflow/api_fastapi/core_api/routes/public/backfills.py Outdated
Comment threadairflow/api_fastapi/core_api/routes/public/backfills.py Outdated
Comment threadairflow/api_fastapi/core_api/datamodels/backfills.py Outdated
pierrejeambrun
pierrejeambrun reviewed Dec 25, 2024
View reviewed changes
Comment threadairflow/api_fastapi/core_api/datamodels/backfills.py Outdated
@jscheffl

jscheffl commented Jan 1, 2025

Copy link
Copy Markdown
Contributor

Note: As PR #45312 has been merged, the code formatting rules have changed for new UI. Please rebase and re-run pre-commit checks to ensure that formatting in folder airflow/ui is adjusted.

pierrejeambrun
pierrejeambrun reviewed Jan 6, 2025
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looking good.

A few suggestions.

I would wait for @dstandish approval before merging that, just to be sure that the backfill logic is correct. (It looks good to me)

Comment threadairflow/models/backfill.py Outdated
Comment threadairflow/models/backfill.py Outdated
Comment threadairflow/models/backfill.py Outdated
Comment threadairflow/models/backfill.py Outdated
pierrejeambrun
pierrejeambrun reviewed Jan 6, 2025
View reviewed changes
Comment threadairflow/models/backfill.py Outdated
Comment threadairflow/models/backfill.py Outdated
Comment threadairflow/api_fastapi/core_api/datamodels/backfills.py Outdated
Comment threadairflow/api_fastapi/core_api/datamodels/backfills.py Outdated
pierrejeambrun
pierrejeambrun reviewed Jan 6, 2025
View reviewed changes
Comment threadairflow/models/backfill.py Outdated
dstandish
dstandish reviewed Jan 6, 2025
View reviewed changes
Comment threadairflow/api_fastapi/core_api/routes/public/backfills.py Outdated
dstandish
dstandish reviewed Jan 6, 2025
View reviewed changes
Comment threadairflow/models/backfill.py Outdated
dstandish
dstandish approved these changes Jan 10, 2025
View reviewed changes

@dstandishdstandish left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only thing i changed is to not lock the dag runs for dry run since it's not necessary.

pierrejeambrun
pierrejeambrun approved these changes Jan 10, 2025
View reviewed changes

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks!

Comment threadairflow/models/backfill.py Outdated
Comment threadairflow/api_fastapi/core_api/routes/public/backfills.py
@phanikumv
phanikumv merged commit 65d6676 into apache:mainJan 12, 2025
@phanikumv
phanikumv deleted the backfill_dry_run branch January 12, 2025 16:48
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jan 13, 2025
Sneha PrabhuSneha Prabhu
@dstandish@karenbraganz
Add dry run for backfill (apache#45062)
8449a4b
* Add dry run for backfill
---------
Co-authored-by: Sneha Prabhu <snehaprabhu@Snehas-MacBook-Pro.local>
Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
Sneha PrabhuSneha Prabhu
@dstandish@HariGS-DB
Add dry run for backfill (apache#45062)
eafc9c7
* Add dry run for backfill
---------
Co-authored-by: Sneha Prabhu <snehaprabhu@Snehas-MacBook-Pro.local>
Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
Sneha PrabhuSneha Prabhu
@dstandish@dauinh
Add dry run for backfill (apache#45062)
50a3e15
* Add dry run for backfill
---------
Co-authored-by: Sneha Prabhu <snehaprabhu@Snehas-MacBook-Pro.local>
Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
Sneha PrabhuSneha Prabhu
@dstandish@got686-yandex
Add dry run for backfill (apache#45062)
e84d6ce
* Add dry run for backfill
---------
Co-authored-by: Sneha Prabhu <snehaprabhu@Snehas-MacBook-Pro.local>
Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
@uranusjruranusjr mentioned this pull request Feb 23, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@uranusjruranusjruranusjr left review comments
@bbovenzibbovenzibbovenzi left review comments
@pierrejeambrunpierrejeambrunpierrejeambrun approved these changes
@dstandishdstandishdstandish approved these changes
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamilton
@jscheffljschefflAwaiting requested review from jscheffl
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@ashbashbAwaiting requested review from ashb

Assignees

No one assigned

Labels

area:UIRelated to UI/UX. For Frontend Developers.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add robust dry run capability for backfill

7 participants

@prabhusneha@jscheffl@uranusjr@bbovenzi@pierrejeambrun@dstandish@phanikumv