Uh oh!
There was an error while loading. Please reload this page.
Refactor bulk_save_to_db - #42245
Merged
Merged
Conversation
uranusjrforce-pushed
the
refactor-dag-load
branch
4 times, most recently
from
September 18, 2024 09:57
7ddfec5 to
5d83fe0CompareThis function collects information from DAG objects, and creates/updates database rows against them. However, it handles A LOT of information, reading a lot of objects, touching a lot of models. The function is not very readable. A new module has been introduced in airflow.dag_processing.collection to encapsulate the logic previously in bulk_save_to_db. Some loops are broken down into multiple loops, so each loop does not do too much (which leads to a lot of long-living variables that reduce readability). Not much is changed aside from that, just mostly splitting things into separate steps to make things clearer.
The dataset manager already does this.
uranusjrforce-pushed
the
refactor-dag-load
branch
from
September 18, 2024 23:45
7d9f778 to
e2ca6ceCompareThis saves a few calls when no dataset/alias references were found in any DAGs.
uranusjrforce-pushed
the
refactor-dag-load
branch
from
September 18, 2024 23:52
e2ca6ce to
d1862e6Compareuranusjrforce-pushed
the
refactor-dag-load
branch
from
September 19, 2024 00:11
d7ceda8 to
7d1e961Compareuranusjr
commented
Sep 19, 2024
MemberAuthor
Note of self: Submit another PR to refactor |
uranusjr
commented
Sep 19, 2024
MemberAuthor
Also maybe we need a new method in DatasetManager for creating aliases. (Or even more for other things?) |
uranusjr
commented
Sep 19, 2024
MemberAuthor
Here we gooooooo |
uranusjr
marked this pull request as ready for review
September 19, 2024 01:41
uranusjr
requested review from
XD-DENG, ashb, ephraimbuddy, jedcunningham and kaxil
as code ownersSeptember 19, 2024 01:41
Lee-W
reviewed
Sep 19, 2024
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
Lee-W
approved these changes
Sep 20, 2024
joaopamaral pushed a commit
to joaopamaral/airflow
that referenced
this pull request
Oct 21, 2024
Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This function collects information from DAG objects, and creates/updates database rows against them. However, it handles A LOT of information, reading a lot of objects, touching a lot of models. The function is not very readable.
A new module has been introduced in airflow.dag_processing.collection to encapsulate the logic previously in bulk_save_to_db.
Some loops are broken down into multiple loops, so each loop does not do too much (which leads to a lot of long-living variables that reduce readability). Not much is changed aside from that, just mostly splitting things into separate steps to make things clearer.