Skip to content

openlineage, redshift: do not call DB for schemas below Airflow 2.10 - #40197

Merged
mobuchowski merged 1 commit into
mainfrom
redshift-remove-calls
Jun 14, 2024
Merged

openlineage, redshift: do not call DB for schemas below Airflow 2.10#40197
mobuchowski merged 1 commit into
mainfrom
redshift-remove-calls

Conversation

@mobuchowski

@mobuchowskimobuchowski commented Jun 12, 2024

Copy link
Copy Markdown
Contributor

Fix for #39969.
Similar to #39113 - do not run external queries for Redshift. The difference in this PR is that with 2.10 and above - if we get #40078 merged we can run this and accept that in very small amount of cases Redshift extraction will time out - but won't affect task.

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@JDarDagran

Copy link
Copy Markdown
Contributor

Do you think we could add similar condition to revert/change on #39113?

@mobuchowski

Copy link
Copy Markdown
ContributorAuthor

@JDarDagran yup, however I'd not mix those two up in same PR here 🙂 And would do this followup actually when #40078 gets merged.

@pankajastropankajastro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment threadairflow/providers/amazon/aws/hooks/redshift_sql.py
Comment threadairflow/providers/openlineage/utils/utils.py
@mobuchowski
mobuchowski merged commit 6f40984 into mainJun 14, 2024
@RNHTTR
RNHTTR deleted the redshift-remove-calls branch June 20, 2024 22:21
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
…pache#40197)
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providersprovider:amazonAWS/Amazon - related issuesprovider:openlineageAIP-53

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@mobuchowski@JDarDagran@potiuk@ephraimbuddy@pankajastro@kacpermuda