Skip to content

Add sql_hook_params parameter to S3ToSqlOperator - #33427

Merged
potiuk merged 1 commit into
apache:mainfrom
alexbegg:S3ToSql-hook_params
Aug 19, 2023
Merged

Add sql_hook_params parameter to S3ToSqlOperator#33427
potiuk merged 1 commit into
apache:mainfrom
alexbegg:S3ToSql-hook_params

Conversation

@alexbegg

@alexbeggalexbegg commented Aug 16, 2023

Copy link
Copy Markdown
Contributor

Adding sql_hook_params parameter to SqlToS3Operator. This will allow you to pass extra config params to the underlying SQL hook.

This uses the same "sql_hook_params" parameter name as already used in SqlToSlackOperator.

(This is related to #33425 which adds sql_hook_params to the opposite transfer, SqlToS3Operator)


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@alexbegg

alexbegg commented Aug 16, 2023

Copy link
Copy Markdown
ContributorAuthor

@uranusjr This PR retains how S3ToSqlOperator accesses BaseHook, but adds a way to pass hook_params parameter to BaseHook.get_hook, which then calls Connection.get_hook (which itself already accepts hook_params).

Comment threadairflow/providers/amazon/aws/transfers/s3_to_sql.py Outdated
@alexbegg
alexbegg marked this pull request as draft August 16, 2023 07:36
@vincbeckvincbeck changed the title Add sql_hook_params parameter to S3ToSqlOperatorAdd sql_hook_params parameter to S3ToSqlOperatorAug 16, 2023
Adding `sql_hook_params` parameter to `SqlToS3Operator`. This will allow you to pass extra config params to the underlying SQL hook.
This uses the same "sql_hook_params" parameter name as already used in `SqlToSlackOperator`.
@alexbegg
alexbegg marked this pull request as ready for review August 18, 2023 22:48
@alexbegg

Copy link
Copy Markdown
ContributorAuthor

@uranusjr I corrected the test to access the hook using the changed method of getting it from Connection.get_hook (which supports hook_params) instead of BaseHook.get_hook (which doesn't), the test passes now.

@alexbegg

alexbegg commented Aug 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Please remove area:core-operators label from this PR as it no longer changes the core operators

@potiuk
potiuk merged commit 1407e27 into apache:mainAug 19, 2023
@alexbegg
alexbegg deleted the S3ToSql-hook_params branch August 19, 2023 07:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providersprovider:amazonAWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@alexbegg@potiuk@uranusjr