Skip to content

openlineage: isolate metadata extraction by executing OL methods in separate, forked process - #40078

Merged
potiuk merged 1 commit into
mainfrom
openlineage-process-execution
Jun 14, 2024
Merged

openlineage: isolate metadata extraction by executing OL methods in separate, forked process#40078
potiuk merged 1 commit into
mainfrom
openlineage-process-execution

Conversation

@mobuchowski

@mobuchowskimobuchowski commented Jun 5, 2024

Copy link
Copy Markdown
Contributor

This PR builds on #39890 (already merged).

After this change, OpenLineage will execute metadata extraction in separate, forked process.
It's a technique modeled to what interaction between LocalTaskJobRunner and StandardTaskRunner looks like - a process, in this case process of StandardTaskRunner watches over OpenLineage listener process during metadata extraction.

This adds a layer of isolation between task execution and OpenLineage, adding a level of assurance that OpenLineage execution does not interfere with task execution in a way other than taking time.
Additionally, this allows us to add configurable timeout for OL execute methods.

The reason for that is, beyond configurability, that sometimes metadata extraction code can hang - for example, when dealing with Snowflake connection issue snowflakedb/snowflake-connector-python#1898 - and we want to give as much guarantees that OL will not cause task to fail.

@boring-cyborgboring-cyborgBot added area:providers area:Scheduler including HA (high availability) scheduler provider:google Google (including GCP) related issues provider:openlineage AIP-53 provider:snowflake Issues related to Snowflake provider labels Jun 5, 2024
@mobuchowski
mobuchowskiforce-pushed the openlineage-process-execution branch 4 times, most recently from 2c96acb to 44ba855CompareJune 7, 2024 11:51
Comment threadairflow/providers/openlineage/conf.py Outdated
@mobuchowski
mobuchowskiforce-pushed the openlineage-process-execution branch 3 times, most recently from 69865d6 to 7c13f5dCompareJune 11, 2024 13:11
Comment threadairflow/providers/openlineage/conf.py
Comment threadairflow/providers/openlineage/plugins/listener.py Outdated
Comment threadairflow/providers/openlineage/plugins/listener.py Outdated
Comment threadairflow/providers/openlineage/plugins/listener.py Outdated
Comment threadairflow/providers/openlineage/sqlparser.py
Comment threadairflow/providers/openlineage/utils/sql.py
Comment threadairflow/providers/snowflake/hooks/snowflake.py Outdated

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few nits about splitting the PR

@potiuk

Copy link
Copy Markdown
Member

Ah I missed those are two commits/PRs already :)

@mobuchowski
mobuchowskiforce-pushed the openlineage-process-execution branch 2 times, most recently from d863e62 to 55e5792CompareJune 12, 2024 14:54
@mobuchowski
mobuchowskiforce-pushed the openlineage-process-execution branch from 55e5792 to 8cbb8bcCompareJune 13, 2024 12:46
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski
mobuchowskiforce-pushed the openlineage-process-execution branch from 8cbb8bc to 187d87eCompareJune 14, 2024 12:39
@potiuk
potiuk merged commit 1a8d12f into mainJun 14, 2024
@eladkal
eladkal deleted the openlineage-process-execution branch June 29, 2024 17:32
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
…ss (apache#40078)
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providersarea:Schedulerincluding HA (high availability) schedulerprovider:googleGoogle (including GCP) related issuesprovider:openlineageAIP-53provider:snowflakeIssues related to Snowflake provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@mobuchowski@potiuk