Skip to content

Add sql_hook_params parameter to SqlToS3Operator - #33425

Merged
vincbeck merged 1 commit into
apache:mainfrom
alexbegg:SqlToS3-hook_params
Aug 16, 2023
Merged

Add sql_hook_params parameter to SqlToS3Operator#33425
vincbeck merged 1 commit into
apache:mainfrom
alexbegg:SqlToS3-hook_params

Conversation

@alexbegg

@alexbeggalexbegg commented Aug 16, 2023

Copy link
Copy Markdown
Contributor

Adding sql_hook_params parameter to SqlToS3Operator. This will allow you to pass extra config params to the underlying SQL hook.

This uses the same parameter name, "sql_hook_params", as already used in SqlToSlackOperator.

(This is related to #33427 which adds sql_hook_params to the opposite transfer, S3ToSqlOperator, however that one needed a different approach by using BaseHook.get_hook instead of Connection.get_hook)


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborgboring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Aug 16, 2023
@alexbeggalexbegg changed the title Adding sql_hook_params to S3ToSql and SqlToS3 operatorsAdd sql_hook_params to S3ToSql and SqlToS3 operatorsAug 16, 2023
@alexbegg
alexbeggforce-pushed the SqlToS3-hook_params branch 2 times, most recently from e904703 to 9903043CompareAugust 16, 2023 00:34
@alexbegg
alexbegg marked this pull request as draft August 16, 2023 00:40
@alexbegg
alexbegg marked this pull request as ready for review August 16, 2023 01:16
@alexbegg

alexbegg commented Aug 16, 2023

Copy link
Copy Markdown
ContributorAuthor

FYI, I added a test to tests/providers/amazon/aws/transfers/test_s3_to_sql.py that is passing, but the other 2 tests in that file are failing. The code for those tests has not changed in months, and I have not changed the setup method at all. I don't know what is causing this.

@uranusjr

Copy link
Copy Markdown
Member

Because you changed how the BaseHook is accessed and it breaks the mock.

@alexbeggalexbegg changed the title Add sql_hook_params to S3ToSql and SqlToS3 operatorsAdd sql_hook_params to SqlToS3OperatorAug 16, 2023
@alexbeggalexbegg changed the title Add sql_hook_params to SqlToS3OperatorAdd sql_hook_params parameter to SqlToS3OperatorAug 16, 2023
Adding `sql_hook_params` parameter to `SqlToS3Operator`. This will allow you to pass extra config params to the underlying SQL hook.
This uses the same "sql_hook_params" parameter name as already used in `SqlToSlackOperator`.
@alexbegg

Copy link
Copy Markdown
ContributorAuthor

I have now split the S3ToSqlOperator changes into its own PR: #33427, it needed a different approach since it uses BaseHook.get_hook instead of Connection.get_hook. Since they take different approaches, I think they should be reviewed separately.

@vincbeckvincbeck changed the title Add sql_hook_params parameter to SqlToS3OperatorAdd sql_hook_params parameter to SqlToS3OperatorAug 16, 2023
@vincbeck
vincbeck merged commit 45d5f64 into apache:mainAug 16, 2023
@alexbegg
alexbegg deleted the SqlToS3-hook_params branch August 16, 2023 18:10
ferruzzi pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 17, 2023
Adding `sql_hook_params` parameter to `SqlToS3Operator`. This will allow you to pass extra config params to the underlying SQL hook.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providersprovider:amazonAWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@alexbegg@uranusjr@vincbeck