Ability to add custom facet in OpenLineage events by anandhimurali · Pull Request #38982 · apache/airflow

anandhimurali · 2024-04-12T23:33:19Z

Motivation
Currently, users need to write custom extractor inorder to add custom facets in the lineage events. Instead, providing a way to inject custom facets by accepting a config.

Modifications
The get_custom_facets function in the OpenLineage provider is enhanced to fetch the Airflow config AIRFLOW__OPENLINEAGE__CUSTOM_RUN_FACETS, execute those functions and append their return values to the run facet of the lineage event for all the operators.

Result
Enables users to add custom facets to the lineage events by defining functions instead of custom extractor.

Configs documentation for AIRFLOW__OPENLINEAGE__CUSTOM_RUN_FACETS

User Docs

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

boring-cyborg · 2024-04-12T23:33:23Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

XD-DENG · 2024-04-18T18:08:23Z

Hi @eladkal , @anandhimurali is my colleague from the same team and I'm really excited about her 1st PR to Airflow!

Do you have an idea who among the committers may be the best to help review this openlineage-related change?

Any suggestion/input would be greatly appreciated. Thanks a lot!

eladkal · 2024-04-18T19:52:25Z

Ping @mobuchowski for review :)

mobuchowski · 2024-04-25T10:03:13Z

Hey @anandhimurali @XD-DENG I think ability to add custom facets to arbitrary operators is a great idea.

I have two points I think it's worth addressing:

Whether we pass all the necessary info. I think given task_instance we can call get_template_context and get all the data, so this might not be an issue.
Whether this acts "per DAG/Task" or it's a mechanism that works for all the tasks equally - this might be defined inside the function so it still might not be an issue other than complicating the function a bit more.

Overall I like the solution, but might be good to specify the goals regarding 1) and 2) - because for sure, later, someone will come and ask "how to make this custom facet work only on some particular tasks" and "how do I pass some custom info to my function".

kacpermuda

I like the idea 🚀 Apart from what @mobuchowski mentioned above, I added some small comments about the code itself.

kacpermuda · 2024-05-14T14:23:14Z

Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄

anandhimurali · 2024-05-14T16:00:34Z

Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄

Hi @kacpermuda, Thanks for the interest and I'm excited to hear that. Sorry I've been busy for the last few weeks. I'll look into the comments this week.

anandhimurali · 2024-05-16T05:58:28Z

I have two points I think it's worth addressing:
Whether we pass all the necessary info. I think given task_instance we can call get_template_context and get all the data, so this might not be an issue.
Whether this acts "per DAG/Task" or it's a mechanism that works for all the tasks equally - this might be defined inside the function so it still might not be an issue other than complicating the function a bit more.
Overall I like the solution, but might be good to specify the goals regarding 1) and 2) - because for sure, later, someone will come and ask "how to make this custom facet work only on some particular tasks" and "how do I pass some custom info to my function".

Hi @mobuchowski,

Yes, I think task_instance is sufficient to fetch necessary info about the run context.
The custom_facet_functions defined works equally for all tasks. Yes, user can choose to return None for scenarios where they won't to add the custom facet.

Let me know if this needs to be documented anywhere. Thanks.

kacpermuda

As you already mentioned, i think this should be documented for the users. We should include the function signature with type hints, so that users know how to properly implement it + some examples from our side.

Probably here and here

mobuchowski · 2024-07-01T14:08:04Z

@anandhimurali any interest in getting this PR into shape and approved? I think this is good, really useful feature.

anandhimurali · 2024-07-01T19:41:04Z

@anandhimurali any interest in getting this PR into shape and approved? I think this is good, really useful feature.

@mobuchowski Yes, definitely. I just have to add some documentation and will get this done this week. Thanks and sorry for the delay.

ahidalgob · 2024-07-03T10:23:44Z

I think it would be good to rename this functionality to custom_run_facets, we might need to inject other facets too, specifically Job.facets.

This is a great feature:)

…_FACET_FUNCTIONS

Co-authored-by: Kacper Muda <mudakacper@gmail.com>

kacpermuda

LGTM

JDarDagran

one small thing, otherwise great job! 🚀

boring-cyborg · 2024-07-22T11:17:22Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

mobuchowski · 2024-07-22T11:17:41Z

Thanks @anandhimurali for implementing this feature 🚀

* Ability to add custom facet in OpenLineage events * Update airflow/providers/openlineage/provider.yaml Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com> * Update airflow/providers/openlineage/provider.yaml Co-authored-by: Kacper Muda <mudakacper@gmail.com> * Adding None type hint for the custom facet function * Fix a test after rebase * Removed the legacy OPENLINEAGE_ configs format for OPENLINEAGE_CUSTOM_FACET_FUNCTIONS * Duplicate facet key check * Update airflow/providers/openlineage/utils/utils.py Co-authored-by: Kacper Muda <mudakacper@gmail.com> * Update airflow/providers/openlineage/utils/utils.py Co-authored-by: Kacper Muda <mudakacper@gmail.com> * Fixes after rebase * Adding user docs for custom_facet_functions * Rename custom_facet_functions as custom_run_facets * Increment version for custom_run_facets feature * Enrich example with access to operator and return value as None. * Add try-except for custom facet function execution * Fix the typing for the custom facet fucntion return type * Documentation: funcs are executed only for START events * Fix the typing for the custom facet function return type * Fixes after pre-commit hook checks * Adding start_date to test DAGs for 2.7 compatibility tests * Removing a out of scope __init__ file added by pre-commit check --------- Co-authored-by: Anandhi <anandhi_murali@apple.com> Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com> Co-authored-by: Kacper Muda <mudakacper@gmail.com>

boring-cyborgBot added area:providers provider:openlineage AIP-53 labels Apr 12, 2024

anandhimurali force-pushed the ol-custom-facet branch from 446c159 to 435db7dCompare April 12, 2024 23:34

anandhimurali marked this pull request as ready for review April 13, 2024 00:47

anandhimurali requested a review from mobuchowski as a code owner April 13, 2024 00:47

eladkal reviewed Apr 14, 2024
View reviewed changes

Comment threadairflow/providers/openlineage/provider.yaml Outdated

kacpermuda reviewed Apr 25, 2024
View reviewed changes

Comment threadairflow/providers/openlineage/conf.py Outdated
Comment threadairflow/providers/openlineage/utils/utils.py Outdated
Comment threadairflow/providers/openlineage/provider.yaml Outdated
Comment threadairflow/providers/openlineage/utils/utils.py Outdated

anandhimurali force-pushed the ol-custom-facet branch from d0265d1 to b4414cbCompare May 16, 2024 04:09

anandhimurali force-pushed the ol-custom-facet branch from 6e96d73 to 03ed172Compare May 16, 2024 06:43

kacpermuda mentioned this pull request May 21, 2024
[openlineage] fix: Prevent error when extractor can't be imported #39736
Merged

kacpermuda reviewed May 21, 2024
View reviewed changes

Comment threadairflow/providers/openlineage/utils/utils.py Outdated
Comment threadairflow/providers/openlineage/utils/utils.py Outdated

eladkal mentioned this pull request May 26, 2024
Status of testing Providers that were prepared on May 26, 2024 #39842
Closed

anandhimurali force-pushed the ol-custom-facet branch from 421e0fe to e851940Compare June 21, 2024 04:46

eladkal force-pushed the ol-custom-facet branch from e851940 to 1eea113Compare June 30, 2024 05:01

anandhimurali force-pushed the ol-custom-facet branch 6 times, most recently from 4f193aa to 0d63f59Compare July 3, 2024 05:02

Anandhiand others added 16 commits July 18, 2024 10:47

Fix a test after rebase
5597404

Removed the legacy OPENLINEAGE_ configs format for OPENLINEAGE_CUSTOM…
457584e
…_FACET_FUNCTIONS

Duplicate facet key check
d246f3a

Update airflow/providers/openlineage/utils/utils.py
5465664
Co-authored-by: Kacper Muda <mudakacper@gmail.com>

Update airflow/providers/openlineage/utils/utils.py
16fe262
Co-authored-by: Kacper Muda <mudakacper@gmail.com>

Fixes after rebase
1d9b1f5

Adding user docs for custom_facet_functions
9847bd6

Rename custom_facet_functions as custom_run_facets
f268e9d

Increment version for custom_run_facets feature
bfffb9e

Enrich example with access to operator and return value as None.
531f996

Add try-except for custom facet function execution
9b6f0c8

Fix the typing for the custom facet fucntion return type
88f8c07

Documentation: funcs are executed only for START events
ead05fd

Fix the typing for the custom facet function return type
6505f4f

Fixes after pre-commit hook checks
497bdba

Adding start_date to test DAGs for 2.7 compatibility tests
8c404b9

anandhimurali force-pushed the ol-custom-facet branch from 81a8282 to 8c404b9Compare July 18, 2024 17:48

kacpermuda reviewed Jul 19, 2024
View reviewed changes

Comment threadtests/system/providers/google/cloud/datapipelines/resources/__init__.py

kacpermuda approved these changes Jul 19, 2024
View reviewed changes

Anandhiand others added 2 commits July 19, 2024 08:50

Removing a out of scope __init__ file added by pre-commit check
18526d7

Merge branch 'main' into ol-custom-facet
75c10d1

JDarDagran approved these changes Jul 22, 2024
View reviewed changes

Comment threaddocs/apache-airflow-providers-openlineage/guides/developer.rst
Comment threadtests/providers/openlineage/utils/custom_facet_fixture.py

mobuchowski approved these changes Jul 22, 2024
View reviewed changes

mobuchowski merged commit e30f810 into apache:mainJul 22, 2024

kacpermuda mentioned this pull request Jul 23, 2024
openlineage: extend custom_run_facets to on_failed and on_complete #40953
Merged

eladkal mentioned this pull request Aug 3, 2024
Status of testing Providers that were prepared on August 03, 2024 #41237
Closed

mobuchowski mentioned this pull request Oct 8, 2024
Add a way for users to specify custom facet builder method that's not integration specific OpenLineage/OpenLineage#3144
Open

Uh oh!

Conversation

anandhimurali commented Apr 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boring-cyborgBot commented Apr 12, 2024

Uh oh!

Uh oh!

XD-DENG commented Apr 18, 2024

Uh oh!

eladkal commented Apr 18, 2024

Uh oh!

mobuchowski commented Apr 25, 2024

Uh oh!

kacpermuda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kacpermuda commented May 14, 2024

Uh oh!

anandhimurali commented May 14, 2024

Uh oh!

anandhimurali commented May 16, 2024

Uh oh!

kacpermuda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mobuchowski commented Jul 1, 2024

Uh oh!

anandhimurali commented Jul 1, 2024

Uh oh!

ahidalgob commented Jul 3, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kacpermuda left a comment

Choose a reason for hiding this comment

Uh oh!

JDarDagran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

boring-cyborgBot commented Jul 22, 2024

Uh oh!

mobuchowski commented Jul 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

anandhimurali commented Apr 12, 2024•
edited
Loading

ahidalgob commented Jul 3, 2024•
edited
Loading