Skip to content

AIP-72: Support DAG parsing context in Task SDK - #45694

Merged
kaxil merged 1 commit into
apache:mainfrom
astronomer:magic-loop-sdk
Jan 15, 2025
Merged

AIP-72: Support DAG parsing context in Task SDK#45694
kaxil merged 1 commit into
apache:mainfrom
astronomer:magic-loop-sdk

Conversation

@kaxil

@kaxilkaxil commented Jan 15, 2025

Copy link
Copy Markdown
Member

closes#45693

Adds support for the Magic Loop: https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html#optimizing-dag-parsing-delays-during-execution with Task SDK


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but we likely need a compat import shim for airflow/utils/dag_parsing_context.py -- do we have a place to capture those anywhere.

Also cc @Lee-W@uranusjr Another deprecated import to track in Ruff please.

@kaxilkaxil added the airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes label Jan 15, 2025
@kaxilkaxil mentioned this pull request Jan 15, 2025
2 tasks
@kaxil

Copy link
Copy Markdown
MemberAuthor

Looks good, but we likely need a compat import shim for airflow/utils/dag_parsing_context.py -- do we have a place to capture those anywhere.

Also cc @Lee-W@uranusjr Another deprecated import to track in Ruff please.

Good point, Yeah, #41641

Added newsfragment too

@kaxil
kaxil merged commit 2841ca9 into apache:mainJan 15, 2025
@kaxil
kaxil deleted the magic-loop-sdk branch January 15, 2025 22:23
@Lee-W

Copy link
Copy Markdown
Member

ruff PR created astral-sh/ruff#15525

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

airflow3.0:breakingCandidates for Airflow 3.0 that contain breaking changesarea:CLIarea:dev-toolsarea:providersarea:Schedulerincluding HA (high availability) schedulerarea:task-sdkkind:documentationprovider:celery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support the "magic loop" with Task SDK

3 participants

@kaxil@Lee-W@ashb