Uh oh!
There was an error while loading. Please reload this page.
Change parameters conn_id into mongo_conn_id - #28946
Conversation
It looks like all manuals speak about mongo_conn_id, but the code uses conn_id. Hence is ignoring the mongo_conn_id that is set and falls back to the default connection. Hope this helps.
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
| hook_name = "MongoDB" | ||
| def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> None: | ||
| def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> None: |
There was a problem hiding this comment.
This add breaking changes in case if user provide arguments as keyword:
MongoHook(conn_id="awesome-conn-id")There was a problem hiding this comment.
@Wesseldr Thanks for the contribution and your first PR! Definitely the docstring for the hook doesn't match what's actually required for use.
+1 on what @Taragolis mentioned.
We try to be very cognizant when thinking about backwards compat for users and try not to introduce breaking changes if avoidable. We really don't want users to upgrade a provider version and suddenly DAGs begin to break. In this situation, if the parameter name is changing, a DeprecationWarning should be raised that conn_id is deprecated and mongo_conn_id should be used. You can find a number of examples in other providers' hooks where a parameter is deprecated for reference on how to do this.
For the deprecation period, the hook should accept both conn_id and mongo_conn_id to maintain backwards compat and ultimately resolve to the self.mongo_conn_id value. Also the MongoSensor should ideally be updated to call this hook with keyword args.
The unit tests for the provider need to be updated accordingly too.
We're more than happy to help out if you are stuck. You can find us on Airflow Slack or we can communicate in this PR directly too.
| hook_name = "MongoDB" | ||
| def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> None: | ||
| def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> None: |
There was a problem hiding this comment.
@Wesseldr Thanks for the contribution and your first PR! Definitely the docstring for the hook doesn't match what's actually required for use.
+1 on what @Taragolis mentioned.
We try to be very cognizant when thinking about backwards compat for users and try not to introduce breaking changes if avoidable. We really don't want users to upgrade a provider version and suddenly DAGs begin to break. In this situation, if the parameter name is changing, a DeprecationWarning should be raised that conn_id is deprecated and mongo_conn_id should be used. You can find a number of examples in other providers' hooks where a parameter is deprecated for reference on how to do this.
For the deprecation period, the hook should accept both conn_id and mongo_conn_id to maintain backwards compat and ultimately resolve to the self.mongo_conn_id value. Also the MongoSensor should ideally be updated to call this hook with keyword args.
The unit tests for the provider need to be updated accordingly too.
We're more than happy to help out if you are stuck. You can find us on Airflow Slack or we can communicate in this PR directly too.
@josh-fell@Taragolis Totally agree with avoiding code breaking changes.. There was no wisdom from my side here.. I just made the assumption the manual was written from the design and since that named it There for I posted a pull request for a name change as I started to believe that was the norm.. Then either the manual is updated to represent Let me know what you pick, obviously dropping my PR is no problem if the current parameters are consistent with other providers and how the styleguide recommends these :-) Thank you for so quickly picking up on this and providing feedback&thoughts. Resources The example I followed that used the same Parameter |
Taragolis
commented
Jan 18, 2023
Unfortunetly this is not Airflow documentation, and we can't change it easily. I also agree with the fact that there is some inconsistency between Hook and Operators/Sensors. But there is no problem to resolve it, see some simple snippet: importwarningsfromairflow.hooks.baseimportBaseHookfromairflow.utils.typesimportNOTSETclassMongoHook(BaseHook):
conn_id="mongo_conn_id"default_conn_name="mongo_default"conn_type="mongo"hook_name="MongoDB"def__init__(self, mongo_conn_id: str=NOTSET, *args, **kwargs) ->None:
super().__init__()
conn_id=kwargs.pop("conn_id", NOTSET)
ifconn_idisnotNOTSET:
warnings.warn(
"Parameter `conn_id` is deprecated and will be removed ""in a future releases. Please use `mongo_conn_id` instead.",
DeprecationWarning,
stacklevel=2,
)
ifmongo_conn_idisNOTSET:
mongo_conn_id=conn_idelse:
raiseValueError("You cannot provide both `mongo_conn_id` and `conn_id`.")
elifmongo_conn_idisNOTSET:
mongo_conn_id=self.default_conn_nameself.mongo_conn_id=mongo_conn_id
...
@propertydefconn_id(self):
warnings.warn(
"Property `conn_id` is deprecated and will be removed ""in a future releases. Please use `mongo_conn_id` instead.",
DeprecationWarning,
stacklevel=2,
)
returnself.mongo_conn_idhook1=MongoHook("conn_id_as_attr")
asserthook1.conn_id=="conn_id_as_attr"asserthook1.mongo_conn_id=="conn_id_as_attr"hook2=MongoHook(conn_id="conn_id_as_legacy_kwarg")
asserthook2.mongo_conn_id=="conn_id_as_legacy_kwarg"hook3=MongoHook(mongo_conn_id="conn_id_as_new_kwarg")
asserthook3.mongo_conn_id=="conn_id_as_new_kwarg"try:
MongoHook(mongo_conn_id="conn_id_as_new_kwarg", conn_id="conn_id_as_legacy_kwarg")
exceptValueError:
passelse:
raiseValueError("Unsupported")
hook_with_default_conn_id=MongoHook()
asserthook_with_default_conn_id.mongo_conn_id==MongoHook.default_conn_name |
potiuk
commented
Jan 18, 2023
Yep. Changin it in back-compatible way as @Taragolis explained is a good idea. |
Taragolis
commented
Feb 18, 2023
@Wesseldr, do you have a plan to continue work on this PR? |
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
It looks like all manuals speak about mongo_conn_id, but the code uses conn_id. Hence is ignoring the mongo_conn_id that is set and falls back to the default connection. Hope this helps.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.