Skip to content

Add Deferrable switch to SnowflakeSqlApiOperator - #31596

Merged
potiuk merged 22 commits into
apache:mainfrom
utkarsharma2:DeferrableSnowflakeSqlApiOperator
Jul 4, 2023
Merged

Add Deferrable switch to SnowflakeSqlApiOperator#31596
potiuk merged 22 commits into
apache:mainfrom
utkarsharma2:DeferrableSnowflakeSqlApiOperator

Conversation

@utkarsharma2

@utkarsharma2utkarsharma2 commented May 29, 2023

Copy link
Copy Markdown
Contributor

This PR donates the SnowflakeSqlApiOperator deferrable developed in astronomer-providers repo to Apache airflow.

@utkarsharma2
utkarsharma2 marked this pull request as draft May 29, 2023 13:24
@utkarsharma2
utkarsharma2 marked this pull request as ready for review May 30, 2023 04:30
@Lee-WLee-W mentioned this pull request Jun 6, 2023
Merged
uranusjr
uranusjr reviewed Jun 13, 2023
View reviewed changes
Comment threadairflow/providers/snowflake/hooks/snowflake_sql_api.py Outdated
utkarsharma2and others added 2 commits June 13, 2023 16:00
phanikumv
phanikumv reviewed Jun 15, 2023
View reviewed changes

@phanikumvphanikumv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

jedcunningham
jedcunningham reviewed Jun 15, 2023
View reviewed changes
Comment threadairflow/providers/snowflake/triggers/snowflake_trigger.py Outdated
Comment threadairflow/providers/snowflake/triggers/snowflake_trigger.py Outdated
Comment threadairflow/providers/snowflake/triggers/snowflake_trigger.py Outdated
Comment threadairflow/providers/snowflake/triggers/snowflake_trigger.py Outdated
Comment threadairflow/providers/snowflake/triggers/snowflake_trigger.py Outdated
uranusjr
uranusjr reviewed Jun 16, 2023
View reviewed changes
Comment threadairflow/providers/snowflake/hooks/snowflake_sql_api.py Outdated
Comment threadairflow/providers/snowflake/hooks/snowflake_sql_api.py
Comment on lines +274 to +275
resp = await response.json()
return self._process_response(status_code, resp)

@uranusjruranusjrJun 16, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should make the response body be fetched lazily, what could take a while (and might actually be the part worth deferring—not sure)

@utkarsharma2utkarsharma2Jun 19, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@uranusjr I'm not sure I understand you completely, but based on what I got. The function get_sql_api_query_status_async is called indirectly when we are polling for the status of the query within triggered. So deferring it would not be ideal, right?

utkarsharma2and others added 2 commits June 16, 2023 14:12
@utkarsharma2@jedcunningham
Update airflow/providers/snowflake/triggers/snowflake_trigger.py
1149913
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@utkarsharma2@jedcunningham
Update airflow/providers/snowflake/triggers/snowflake_trigger.py
d1f7cf2
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
uranusjr
uranusjr reviewed Jun 16, 2023
View reviewed changes
Comment threadairflow/providers/snowflake/triggers/snowflake_trigger.py Outdated
utkarsharma2and others added 4 commits June 19, 2023 07:29
@utkarsharma2@uranusjr
Update airflow/providers/snowflake/triggers/snowflake_trigger.py
4b5037e
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
@utkarsharma2@jedcunningham
Update airflow/providers/snowflake/triggers/snowflake_trigger.py
e8fc37a
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@utkarsharma2
Sync param list
64c72e6
@utkarsharma2
Optimise status polling
f3b13fc
@utkarsharma2
utkarsharma2force-pushed the DeferrableSnowflakeSqlApiOperator branch from 07d3fae to b39b57bCompareJune 28, 2023 11:06
@potiuk

potiuk commented Jul 3, 2023

Copy link
Copy Markdown
Member

I think you should rebase/solve conflicts on that one jsut to be sure.

utkarsharma2
utkarsharma2 commented Jul 4, 2023
View reviewed changes
Comment threadairflow/providers/snowflake/hooks/snowflake_sql_api.py Outdated
@potiuk
potiuk merged commit 891c2e4 into apache:mainJul 4, 2023
@eladkaleladkal mentioned this pull request Jul 6, 2023
Closed
69 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@phanikumvphanikumvphanikumv approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@uranusjruranusjrAwaiting requested review from uranusjr
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham

Assignees

No one assigned

Labels

area:providerskind:documentationprovider:snowflakeIssues related to Snowflake provider

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@utkarsharma2@potiuk@uranusjr@jedcunningham@phanikumv