Skip to content

AIP-62: Add HookLineageCollector - #40335

Merged
mobuchowski merged 2 commits into
apache:mainfrom
JDarDagran:aip-62/hook-lineage-collector
Jul 15, 2024
Merged

AIP-62: Add HookLineageCollector#40335
mobuchowski merged 2 commits into
apache:mainfrom
JDarDagran:aip-62/hook-lineage-collector

Conversation

@JDarDagran

@JDarDagranJDarDagran commented Jun 19, 2024

Copy link
Copy Markdown
Contributor

Add HookLineageCollector that during task execution should register and hold lineage sent from hooks.
Add HookLineageReader that defines whether HookLineageCollector should be enabled to process lineage sent from hooks.
Add Dataset factories to make sure Datasets registered with HookLineageCollector is AIP-60 compliant.

Closes: #38766

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch from 7382021 to 1293a07CompareJune 19, 2024 21:48
uranusjr
uranusjr reviewed Jun 20, 2024
View reviewed changes
Comment threadairflow/datasets/__init__.py Outdated
mobuchowski
mobuchowski requested changes Jul 4, 2024
View reviewed changes

@mobuchowskimobuchowski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, however, I'd add some more "practical tests" that show how mechanism works.

Would also love to see better documentation, though I think it can just link to something that would be populated later in provider docs in followup PR.

Comment threadairflow/providers_manager.py
Comment threadairflow/lineage/hook.py Outdated
Comment threadairflow/lineage/hook.py Outdated
@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch from 1293a07 to 4e81003CompareJuly 7, 2024 22:44
@JDarDagran
JDarDagran requested a review from potiuk as a code ownerJuly 7, 2024 22:44
@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch 2 times, most recently from ce0994c to 75ec801CompareJuly 7, 2024 23:44
@mobuchowski
mobuchowski requested a review from uranusjrJuly 8, 2024 16:35
@mobuchowski

mobuchowski commented Jul 8, 2024

Copy link
Copy Markdown
Contributor

@uranusjr want to take another look?

@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch 2 times, most recently from 585ec21 to f6af11aCompareJuly 9, 2024 15:16
@kacpermudakacpermuda mentioned this pull request Jul 9, 2024
Merged
@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch 2 times, most recently from b651b1a to 6c9e124CompareJuly 10, 2024 14:08
@mobuchowskimobuchowski added the AIP-62 Tasks tracking implementation of AIP-62 Getting Lineage from Hook Instrumentation label Jul 11, 2024
jedcunningham
jedcunningham reviewed Jul 12, 2024
View reviewed changes
Comment threaddocs/apache-airflow/administration-and-deployment/lineage.rst Outdated
Comment threadairflow/lineage/hook.py Outdated
Comment threaddocs/apache-airflow/administration-and-deployment/lineage.rst Outdated
@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch 3 times, most recently from 0380e68 to 5f6c253CompareJuly 15, 2024 12:19
@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch 2 times, most recently from 1f54e98 to 53f6a97CompareJuly 15, 2024 13:55
mobuchowski
mobuchowski reviewed Jul 15, 2024
View reviewed changes
Comment threadairflow/lineage/hook.py Outdated
Comment threadairflow/lineage/hook.py Outdated
@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch from 53f6a97 to adecb55CompareJuly 15, 2024 14:13
@JDarDagran
Add HookLineageCollector that during task execution
8d455d6
should register and hold lineage sent from hooks.
Add HookLineageReader that defines whether HookLineageCollector
should be enabled to process lineage sent from hooks.
Add Dataset factories to make sure Datasets registered with
HookLineageCollector is AIP-60 compliant.
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch from adecb55 to 89fed8eCompareJuly 15, 2024 21:28
jedcunningham
jedcunningham reviewed Jul 15, 2024
View reviewed changes
Comment threaddocs/apache-airflow/administration-and-deployment/lineage.rst Outdated
Comment threaddocs/apache-airflow/administration-and-deployment/lineage.rst Outdated
@JDarDagran
Remove default create_dataset method.
351d2bc
Add section in experimental lineage docs.
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
@JDarDagran
JDarDagranforce-pushed the aip-62/hook-lineage-collector branch from 89fed8e to 351d2bcCompareJuly 15, 2024 22:26
@mobuchowski
mobuchowski merged commit cd68840 into apache:mainJul 15, 2024
@ephraimbuddyephraimbuddy added the type:new-feature Changelog: New Features label Jul 22, 2024
@ephraimbuddyephraimbuddy added this to the Airflow 2.10.0 milestone Jul 23, 2024
@ephraimbuddyephraimbuddy added changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) and removed type:new-feature Changelog: New Features labels Jul 24, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
@JDarDagran@romsharon98
AIP-62: Add HookLineageCollector (apache#40335)
6cc446a
* Add HookLineageCollector that during task execution
should register and hold lineage sent from hooks.
Add HookLineageReader that defines whether HookLineageCollector
should be enabled to process lineage sent from hooks.
Add Dataset factories to make sure Datasets registered with
HookLineageCollector is AIP-60 compliant.
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
* Remove default `create_dataset` method.
Add section in experimental lineage docs.
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
---------
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
This was referenced Jul 28, 2024
Closed
Closed
@kacpermudakacpermuda mentioned this pull request Oct 31, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mobuchowskimobuchowskimobuchowski approved these changes
@jedcunninghamjedcunninghamjedcunningham left review comments
@potiukpotiukAwaiting requested review from potiuk
@uranusjruranusjrAwaiting requested review from uranusjr

Assignees

No one assigned

Labels

AIP-62Tasks tracking implementation of AIP-62 Getting Lineage from Hook Instrumentationarea:lineagechangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)

Projects

None yet

Milestone

Airflow 2.10.0

Development

Successfully merging this pull request may close these issues.

Implement HookLineageCollector for collection of Hook-generated datasets

5 participants

@JDarDagran@mobuchowski@uranusjr@jedcunningham@ephraimbuddy