Skip to content

AIP-62: add hook lineage collection support for S3Hook - #40819

Merged
mobuchowski merged 1 commit into
mainfrom
aip-62/s3hook
Jul 22, 2024
Merged

AIP-62: add hook lineage collection support for S3Hook #40819
mobuchowski merged 1 commit into
mainfrom
aip-62/s3hook

Conversation

@mobuchowski

@mobuchowskimobuchowski commented Jul 16, 2024

Copy link
Copy Markdown
Contributor

This PR implements #40819.

This adds support for automatic lineage collection from S3Hook (non-async methods) and adds support for S3 dataset.

@mobuchowskimobuchowski added the AIP-62 Tasks tracking implementation of AIP-62 Getting Lineage from Hook Instrumentation label Jul 16, 2024
@mobuchowski
mobuchowskiforce-pushed the aip-62/s3hook branch 4 times, most recently from 6e2cfb5 to d34b054CompareJuly 16, 2024 22:22
@mobuchowski
mobuchowskiforce-pushed the aip-62/s3hook branch 2 times, most recently from 3418daa to b6508e4CompareJuly 16, 2024 23:45
@mobuchowski
mobuchowskiforce-pushed the aip-62/s3hook branch 2 times, most recently from dbca2ae to c27ffcdCompareJuly 17, 2024 09:43
Comment threadairflow/lineage/hook.py
Comment threadgenerated/provider_dependencies.json

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with one nit about compat dependency.

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski
mobuchowski merged commit 0212f67 into mainJul 22, 2024
@ephraimbuddyephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jul 22, 2024
@ephraimbuddyephraimbuddy added this to the Airflow 2.10.0 milestone Jul 23, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski
mobuchowski deleted the aip-62/s3hook branch August 2, 2024 13:40
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIP-62Tasks tracking implementation of AIP-62 Getting Lineage from Hook Instrumentationarea:dev-toolsarea:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)provider:amazonAWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@mobuchowski@potiuk@vincbeck@ephraimbuddy