Skip to content

AIP-72: Task SDK support for on_task_instance_* listeners, make OpenLineage compatible - #45294

Merged
mobuchowski merged 1 commit into
apache:mainfrom
mobuchowski:tasksdk-call-listeners
Feb 11, 2025
Merged

AIP-72: Task SDK support for on_task_instance_* listeners, make OpenLineage compatible#45294
mobuchowski merged 1 commit into
apache:mainfrom
mobuchowski:tasksdk-call-listeners

Conversation

@mobuchowski

@mobuchowskimobuchowski commented Dec 30, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

With AIP-72, there is no access to the database session from the worker process, and the runtime objects have some differences to the db models. This PR contains three commits that deal with that situation:

  • First commit adjusts on_task_instance_* listeners interface to AIP-72: drops session argument and makes task_instance argument an instance of RuntimeTaskInstance class, not database model
  • Second commit adds basic support for calling listeners in Task SDK, additionally adding some context fields that allow OL and other listeners not use DB.
  • Third commit adjusts OpenLineage integration to work with new interface and Task SDK.

Some followup work:

  • wrap listener calls into Activity to make logging better visible from UI, and distinct from task logs
  • add separate interface for API to capture manual/API task changes. Impossible to reuse current interface since object would be a different type (not RuntimeTaskInstance)
  • add support for more types of listeners, more in task Ensure Listeners work with Task SDK #45491

closes#45423

@boring-cyborgboring-cyborgBot added area:Executors-core LocalExecutor & SequentialExecutor area:providers area:Scheduler including HA (high availability) scheduler area:task-sdk provider:openlineage AIP-53 labels Dec 30, 2024
@mobuchowski
mobuchowskiforce-pushed the tasksdk-call-listeners branch 4 times, most recently from f07a78e to 81b886bCompareJanuary 2, 2025 08:20
@potiuk

potiuk commented Jan 2, 2025

Copy link
Copy Markdown
Member

@mobuchowski - can you please rebase that one -> we found and issue with @jscheffl with the new caching scheme - fixed in #45347 that would run "main" version of the tests. I am asking in all affected PRs to rebase.

@potiuk
potiukforce-pushed the tasksdk-call-listeners branch from 81b886b to 15fd95aCompareJanuary 2, 2025 12:26
@potiuk

potiuk commented Jan 2, 2025

Copy link
Copy Markdown
Member

Actually - I rebased it now.

@mobuchowski

mobuchowski commented Jan 2, 2025

Copy link
Copy Markdown
ContributorAuthor

I will rebase very soon as I'm working on some of the test failures anyway 🙂

@mobuchowski
mobuchowskiforce-pushed the tasksdk-call-listeners branch 7 times, most recently from 35a94e2 to 86297afCompareJanuary 6, 2025 16:35
@mobuchowski
mobuchowski marked this pull request as ready for review January 6, 2025 16:52
@mobuchowski
mobuchowskiforce-pushed the tasksdk-call-listeners branch from 86297af to 6d88389CompareJanuary 6, 2025 17:04
@mobuchowskimobuchowski changed the title [draft] AIP-72: call on_task_instance_* listenersAIP-72: call on_task_instance_* listenersJan 6, 2025
@mobuchowski
mobuchowskiforce-pushed the tasksdk-call-listeners branch 4 times, most recently from c2e2165 to 305fff0CompareJanuary 24, 2025 12:21
@mobuchowski
mobuchowskiforce-pushed the tasksdk-call-listeners branch 8 times, most recently from 3dc5ae4 to 39172c3CompareFebruary 4, 2025 09:26
@vikramkoka

vikramkoka commented Feb 7, 2025

Copy link
Copy Markdown
Contributor

@mobuchowski
Hope all is well. Just checking in how this is coming along?

You had also mentioned that you would be raising another PR once #45732 was merged. Since that is already merged, is that follow-up PR already in progress.

Thank you!

@mobuchowski
mobuchowskiforce-pushed the tasksdk-call-listeners branch from 39172c3 to 0705a85CompareFebruary 7, 2025 19:12
@mobuchowski
change listener API, add basic support for task instance listeners in…
61da140
… TaskSDK, make OpenLineage provider support Airflow 3's listener interface
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski
mobuchowskiforce-pushed the tasksdk-call-listeners branch from 0705a85 to 61da140CompareFebruary 7, 2025 19:39
@mobuchowski

mobuchowski commented Feb 7, 2025

Copy link
Copy Markdown
ContributorAuthor

Hey @vikramkoka - rebased this PR and fixed conflicts, it should be good to merge.

Haven't started on followup to #45732 but will try to find some time next week. I don't think we need to hold up this PR for that, especially since rebasing 2000 lines PR on fast moving target takes significant amount of time.

uranusjr
uranusjr reviewed Feb 11, 2025
View reviewed changes
Comment threadtask_sdk/src/airflow/sdk/execution_time/supervisor.py
uranusjr
uranusjr approved these changes Feb 11, 2025
View reviewed changes

@uranusjruranusjr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this works. Should have a follow-up PR changing OL identifier for DagRun and TaskInstance to use run_id.

@mobuchowski
mobuchowski merged commit 0047a68 into apache:mainFeb 11, 2025
@amoghrajeshamoghrajesh mentioned this pull request Feb 11, 2025
Merged
potiuk added a commit to potiuk/airflow that referenced this pull request Feb 12, 2025
@potiuk
Always clean airflow installation in provider's compatibility tests
17023bd
So far cleaning airflow installation only happened in canary runs
and it caused some PRs not failing when they should - for exmaple
the apache#45294 was green when it should fail because uuid6 package was
not removed before installing old version of Airlfow.
Cleaning airflow installation is fast with uv so we should be
ok with running it always for compatibility tests.
@potiukpotiuk mentioned this pull request Feb 12, 2025
Merged
potiuk added a commit to potiuk/airflow that referenced this pull request Feb 12, 2025
@potiuk
Always clean airflow installation in provider's compatibility tests
a89bc85
So far cleaning airflow installation only happened in canary runs
and it caused some PRs not failing when they should - for exmaple
the apache#45294 was green when it should fail because uuid6 package was
not removed before installing old version of Airlfow.
Cleaning airflow installation is fast with uv so we should be
ok with running it always for compatibility tests.
potiuk added a commit that referenced this pull request Feb 12, 2025
@potiuk
Always clean airflow installation in provider's compatibility tests (#…
024e23d
…46693)
So far cleaning airflow installation only happened in canary runs
and it caused some PRs not failing when they should - for exmaple
the #45294 was green when it should fail because uuid6 package was
not removed before installing old version of Airlfow.
Cleaning airflow installation is fast with uv so we should be
ok with running it always for compatibility tests.
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
@mobuchowski@ambika-garg
change listener API, add basic support for task instance listeners in…
70154aa
… TaskSDK, make OpenLineage provider support Airflow 3's listener interface (apache#45294)
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
@potiuk@ambika-garg
Always clean airflow installation in provider's compatibility tests (a…
c0e00bc
…pache#46693)
So far cleaning airflow installation only happened in canary runs
and it caused some PRs not failing when they should - for exmaple
the apache#45294 was green when it should fail because uuid6 package was
not removed before installing old version of Airlfow.
Cleaning airflow installation is fast with uv so we should be
ok with running it always for compatibility tests.
@amoghrajeshamoghrajesh mentioned this pull request Feb 21, 2025
Merged
@eladkaleladkal mentioned this pull request Feb 21, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxilkaxilkaxil left review comments
@uranusjruranusjruranusjr approved these changes
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@pierrejeambrunpierrejeambrunAwaiting requested review from pierrejeambrun
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@ashbashbAwaiting requested review from ashbashb is a code owner
@o-nikolaso-nikolasAwaiting requested review from o-nikolas
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
+1 more reviewer
@kacpermudakacpermudakacpermuda approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:Executors-coreLocalExecutor & SequentialExecutorarea:providersarea:Schedulerincluding HA (high availability) schedulerarea:task-sdkprovider:openlineageAIP-53

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Ensure Task Instance Listeners work with Task SDK

7 participants

@mobuchowski@potiuk@vikramkoka@ashb@uranusjr@kaxil@kacpermuda