Remove redshift async test job

[pankajastro]

Address #28850 (comment)

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

I suggest to sync with AWS before doing further work.
We are not yet sure if we should add this lib
#30032 (comment)

[eladkal]

cc @shubham22@kaxil

[pankajastro]

Hi @eladkal, I agree with your concern. I was testing the workaround suggested by @potiuk here #28850 (comment) to run the test

[shubham22]

I see that this is already merged, but can we please wait until 3/22 before making more deferrable contributions? Recent changes from @potiuk in #30144 do alleviate many concerns, but there is always a chance that users won't be able to access new Boto functionality. We are in the process of sharing the raised concerns internally at AWS. However, I do understand that there is no better alternative than using aiobotocore and tradeoff is not bad after the CI job separation. cc: @syedahsn@o-nikolas

[potiuk]

One more thing here. Technically with the job implemented as it is right now wiht my #30144 we have another possibility. We could reverse the approach.

Currently we remove aiobotocore in the separate job and upgrade to latest boto and run all the relevant provider tests.

But we could do it the opposite: we could use latest boto/botocore in "main", remove aibotocore from the "devel" set of dependencies and install aiobotocore (together with downgrading boto/botocore) in the separate job.

This has slightly different characteristics:

• by default the constraints we publish will not have aiobotocore and willhave botocore/boto not compatible with the latest aiobotocore - this means that if someone would like to use deferrable operators from AWS would have to deliberately install aiobotocore and downgrade boto/botocore with it.
• the separate job would test automatically if the "aiobotocore" compatible boto still works with all the unit-tested provider operators - this way any PR that would be based on newer features would fail at PR time.
• then we could make a deliberate decision that this is ok (and add conditional skips in the unit tests) if that happens

So basically this is this trade-off:

1. (current) deferrable operators work out-of-the-box with the "official constraints" (but without latest boto/botocore)
2. (possible) - deferrable operators will not work with official constraints and they require deliberately installing aiobotocore (and downgrading boto/botocore) - but latest botocore is used in those constraints

Those are the two trade-offs, and we can still change the decision (easily) which way go. With #30144 this is as easy as changing few lines in the CI scripts.

[syedahsn]

I think shipping aiobotocore by default is the better option. It will be a smoother user experience if users don't have to change their workflows to use deferrables if their workflows use features not covered by aiobotocore. It's not an ideal situation in either case, but this option is less likely to cause issues for users wanting to use deferrable operators.

[kaxil]

I am with @o-nikolas on this one -- a case where everything works out of the box is better than if a user has to make choices depending.

Also, we want to increase and drive the adoption of Deferrable operators, and decreasing road-blocks for user's adopting it would be my preference

[shubham22]

Agree with the thoughts expressed by @o-nikolas@syedahsn and @kaxil. Priority should be given to providing a better out-of-the-box user experience. If we don't make aiobotocore the default option, it will increase the workload for users, and as Niko mentioned, it may require refactoring of DAGs once the user enables async. On the other hand, we don't have any evidence yet that restricting the botocore version will cause significant customer problems.

Thank you @potiuk for spending time on this and providing us with options. We're certainly converging to a better approach than what we started with.

[potiuk]

Yeah. I did not want to bias it, but that was also my personal preference (but I wanted to objectively show you the options). It seems much easier to explain as well, and I have a feeling that deferring new features by few months is not a huge issue as the adoption is usually anyhow delayed and deferrable operators not being available by default would be a big bummer.

With the #30161 now merged, the aiobotocore will also be included in the PROD image by default, so we are on track to increase the adoption of deferrable operators :).

[pankajastro]

This is awesome, look like we have solution now. Thank you so much @potiuk for stepping-up bringing it in better shape it was long since pending 🚀

[potiuk]

I treat @o-nikolas approval on #29923 as general OK from the AWS team to follow this approach.

[pankajastro]

Address #28850 (comment)

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

I suggest to sync with AWS before doing further work.
We are not yet sure if we should add this lib
#30032 (comment)

[eladkal]

cc @shubham22@kaxil

[pankajastro]

Hi @eladkal, I agree with your concern. I was testing the workaround suggested by @potiuk here #28850 (comment) to run the test

[shubham22]

I see that this is already merged, but can we please wait until 3/22 before making more deferrable contributions? Recent changes from @potiuk in #30144 do alleviate many concerns, but there is always a chance that users won't be able to access new Boto functionality. We are in the process of sharing the raised concerns internally at AWS. However, I do understand that there is no better alternative than using aiobotocore and tradeoff is not bad after the CI job separation. cc: @syedahsn@o-nikolas

[potiuk]

One more thing here. Technically with the job implemented as it is right now wiht my #30144 we have another possibility. We could reverse the approach.

Currently we remove aiobotocore in the separate job and upgrade to latest boto and run all the relevant provider tests.

But we could do it the opposite: we could use latest boto/botocore in "main", remove aibotocore from the "devel" set of dependencies and install aiobotocore (together with downgrading boto/botocore) in the separate job.

This has slightly different characteristics:

• by default the constraints we publish will not have aiobotocore and willhave botocore/boto not compatible with the latest aiobotocore - this means that if someone would like to use deferrable operators from AWS would have to deliberately install aiobotocore and downgrade boto/botocore with it.
• the separate job would test automatically if the "aiobotocore" compatible boto still works with all the unit-tested provider operators - this way any PR that would be based on newer features would fail at PR time.
• then we could make a deliberate decision that this is ok (and add conditional skips in the unit tests) if that happens

So basically this is this trade-off:

1. (current) deferrable operators work out-of-the-box with the "official constraints" (but without latest boto/botocore)
2. (possible) - deferrable operators will not work with official constraints and they require deliberately installing aiobotocore (and downgrading boto/botocore) - but latest botocore is used in those constraints

Those are the two trade-offs, and we can still change the decision (easily) which way go. With #30144 this is as easy as changing few lines in the CI scripts.

[syedahsn]

I think shipping aiobotocore by default is the better option. It will be a smoother user experience if users don't have to change their workflows to use deferrables if their workflows use features not covered by aiobotocore. It's not an ideal situation in either case, but this option is less likely to cause issues for users wanting to use deferrable operators.

[kaxil]

I am with @o-nikolas on this one -- a case where everything works out of the box is better than if a user has to make choices depending.

Also, we want to increase and drive the adoption of Deferrable operators, and decreasing road-blocks for user's adopting it would be my preference

[shubham22]

Agree with the thoughts expressed by @o-nikolas@syedahsn and @kaxil. Priority should be given to providing a better out-of-the-box user experience. If we don't make aiobotocore the default option, it will increase the workload for users, and as Niko mentioned, it may require refactoring of DAGs once the user enables async. On the other hand, we don't have any evidence yet that restricting the botocore version will cause significant customer problems.

Thank you @potiuk for spending time on this and providing us with options. We're certainly converging to a better approach than what we started with.

[potiuk]

Yeah. I did not want to bias it, but that was also my personal preference (but I wanted to objectively show you the options). It seems much easier to explain as well, and I have a feeling that deferring new features by few months is not a huge issue as the adoption is usually anyhow delayed and deferrable operators not being available by default would be a big bummer.

With the #30161 now merged, the aiobotocore will also be included in the PROD image by default, so we are on track to increase the adoption of deferrable operators :).

[pankajastro]

This is awesome, look like we have solution now. Thank you so much @potiuk for stepping-up bringing it in better shape it was long since pending 🚀

[potiuk]

I treat @o-nikolas approval on #29923 as general OK from the AWS team to follow this approach.

[pankajastro]

Address #28850 (comment)

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

I suggest to sync with AWS before doing further work.
We are not yet sure if we should add this lib
#30032 (comment)

[eladkal]

cc @shubham22@kaxil

[pankajastro]

Hi @eladkal, I agree with your concern. I was testing the workaround suggested by @potiuk here #28850 (comment) to run the test

[shubham22]

I see that this is already merged, but can we please wait until 3/22 before making more deferrable contributions? Recent changes from @potiuk in #30144 do alleviate many concerns, but there is always a chance that users won't be able to access new Boto functionality. We are in the process of sharing the raised concerns internally at AWS. However, I do understand that there is no better alternative than using aiobotocore and tradeoff is not bad after the CI job separation. cc: @syedahsn@o-nikolas

[potiuk]

One more thing here. Technically with the job implemented as it is right now wiht my #30144 we have another possibility. We could reverse the approach.

Currently we remove aiobotocore in the separate job and upgrade to latest boto and run all the relevant provider tests.

But we could do it the opposite: we could use latest boto/botocore in "main", remove aibotocore from the "devel" set of dependencies and install aiobotocore (together with downgrading boto/botocore) in the separate job.

This has slightly different characteristics:

• by default the constraints we publish will not have aiobotocore and willhave botocore/boto not compatible with the latest aiobotocore - this means that if someone would like to use deferrable operators from AWS would have to deliberately install aiobotocore and downgrade boto/botocore with it.
• the separate job would test automatically if the "aiobotocore" compatible boto still works with all the unit-tested provider operators - this way any PR that would be based on newer features would fail at PR time.
• then we could make a deliberate decision that this is ok (and add conditional skips in the unit tests) if that happens

So basically this is this trade-off:

1. (current) deferrable operators work out-of-the-box with the "official constraints" (but without latest boto/botocore)
2. (possible) - deferrable operators will not work with official constraints and they require deliberately installing aiobotocore (and downgrading boto/botocore) - but latest botocore is used in those constraints

Those are the two trade-offs, and we can still change the decision (easily) which way go. With #30144 this is as easy as changing few lines in the CI scripts.

[syedahsn]

I think shipping aiobotocore by default is the better option. It will be a smoother user experience if users don't have to change their workflows to use deferrables if their workflows use features not covered by aiobotocore. It's not an ideal situation in either case, but this option is less likely to cause issues for users wanting to use deferrable operators.

[kaxil]

I am with @o-nikolas on this one -- a case where everything works out of the box is better than if a user has to make choices depending.

Also, we want to increase and drive the adoption of Deferrable operators, and decreasing road-blocks for user's adopting it would be my preference

[shubham22]

Agree with the thoughts expressed by @o-nikolas@syedahsn and @kaxil. Priority should be given to providing a better out-of-the-box user experience. If we don't make aiobotocore the default option, it will increase the workload for users, and as Niko mentioned, it may require refactoring of DAGs once the user enables async. On the other hand, we don't have any evidence yet that restricting the botocore version will cause significant customer problems.

Thank you @potiuk for spending time on this and providing us with options. We're certainly converging to a better approach than what we started with.

[potiuk]

Yeah. I did not want to bias it, but that was also my personal preference (but I wanted to objectively show you the options). It seems much easier to explain as well, and I have a feeling that deferring new features by few months is not a huge issue as the adoption is usually anyhow delayed and deferrable operators not being available by default would be a big bummer.

With the #30161 now merged, the aiobotocore will also be included in the PROD image by default, so we are on track to increase the adoption of deferrable operators :).

[pankajastro]

This is awesome, look like we have solution now. Thank you so much @potiuk for stepping-up bringing it in better shape it was long since pending 🚀

[potiuk]

I treat @o-nikolas approval on #29923 as general OK from the AWS team to follow this approach.

[pankajastro]

Address #28850 (comment)

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

I suggest to sync with AWS before doing further work.
We are not yet sure if we should add this lib
#30032 (comment)

[eladkal]

cc @shubham22@kaxil

[pankajastro]

Hi @eladkal, I agree with your concern. I was testing the workaround suggested by @potiuk here #28850 (comment) to run the test

[shubham22]

I see that this is already merged, but can we please wait until 3/22 before making more deferrable contributions? Recent changes from @potiuk in #30144 do alleviate many concerns, but there is always a chance that users won't be able to access new Boto functionality. We are in the process of sharing the raised concerns internally at AWS. However, I do understand that there is no better alternative than using aiobotocore and tradeoff is not bad after the CI job separation. cc: @syedahsn@o-nikolas

[potiuk]

One more thing here. Technically with the job implemented as it is right now wiht my #30144 we have another possibility. We could reverse the approach.

Currently we remove aiobotocore in the separate job and upgrade to latest boto and run all the relevant provider tests.

But we could do it the opposite: we could use latest boto/botocore in "main", remove aibotocore from the "devel" set of dependencies and install aiobotocore (together with downgrading boto/botocore) in the separate job.

This has slightly different characteristics:

• by default the constraints we publish will not have aiobotocore and willhave botocore/boto not compatible with the latest aiobotocore - this means that if someone would like to use deferrable operators from AWS would have to deliberately install aiobotocore and downgrade boto/botocore with it.
• the separate job would test automatically if the "aiobotocore" compatible boto still works with all the unit-tested provider operators - this way any PR that would be based on newer features would fail at PR time.
• then we could make a deliberate decision that this is ok (and add conditional skips in the unit tests) if that happens

So basically this is this trade-off:

1. (current) deferrable operators work out-of-the-box with the "official constraints" (but without latest boto/botocore)
2. (possible) - deferrable operators will not work with official constraints and they require deliberately installing aiobotocore (and downgrading boto/botocore) - but latest botocore is used in those constraints

Those are the two trade-offs, and we can still change the decision (easily) which way go. With #30144 this is as easy as changing few lines in the CI scripts.

[syedahsn]

I think shipping aiobotocore by default is the better option. It will be a smoother user experience if users don't have to change their workflows to use deferrables if their workflows use features not covered by aiobotocore. It's not an ideal situation in either case, but this option is less likely to cause issues for users wanting to use deferrable operators.

[kaxil]

I am with @o-nikolas on this one -- a case where everything works out of the box is better than if a user has to make choices depending.

Also, we want to increase and drive the adoption of Deferrable operators, and decreasing road-blocks for user's adopting it would be my preference

[shubham22]

Agree with the thoughts expressed by @o-nikolas@syedahsn and @kaxil. Priority should be given to providing a better out-of-the-box user experience. If we don't make aiobotocore the default option, it will increase the workload for users, and as Niko mentioned, it may require refactoring of DAGs once the user enables async. On the other hand, we don't have any evidence yet that restricting the botocore version will cause significant customer problems.

Thank you @potiuk for spending time on this and providing us with options. We're certainly converging to a better approach than what we started with.

[potiuk]

Yeah. I did not want to bias it, but that was also my personal preference (but I wanted to objectively show you the options). It seems much easier to explain as well, and I have a feeling that deferring new features by few months is not a huge issue as the adoption is usually anyhow delayed and deferrable operators not being available by default would be a big bummer.

With the #30161 now merged, the aiobotocore will also be included in the PROD image by default, so we are on track to increase the adoption of deferrable operators :).

[pankajastro]

This is awesome, look like we have solution now. Thank you so much @potiuk for stepping-up bringing it in better shape it was long since pending 🚀

[potiuk]

I treat @o-nikolas approval on #29923 as general OK from the AWS team to follow this approach.

[pankajastro]

Address #28850 (comment)

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

I suggest to sync with AWS before doing further work.
We are not yet sure if we should add this lib
#30032 (comment)

[eladkal]

cc @shubham22@kaxil

[pankajastro]

Hi @eladkal, I agree with your concern. I was testing the workaround suggested by @potiuk here #28850 (comment) to run the test

[shubham22]

I see that this is already merged, but can we please wait until 3/22 before making more deferrable contributions? Recent changes from @potiuk in #30144 do alleviate many concerns, but there is always a chance that users won't be able to access new Boto functionality. We are in the process of sharing the raised concerns internally at AWS. However, I do understand that there is no better alternative than using aiobotocore and tradeoff is not bad after the CI job separation. cc: @syedahsn@o-nikolas

[potiuk]

One more thing here. Technically with the job implemented as it is right now wiht my #30144 we have another possibility. We could reverse the approach.

Currently we remove aiobotocore in the separate job and upgrade to latest boto and run all the relevant provider tests.

But we could do it the opposite: we could use latest boto/botocore in "main", remove aibotocore from the "devel" set of dependencies and install aiobotocore (together with downgrading boto/botocore) in the separate job.

This has slightly different characteristics:

• by default the constraints we publish will not have aiobotocore and willhave botocore/boto not compatible with the latest aiobotocore - this means that if someone would like to use deferrable operators from AWS would have to deliberately install aiobotocore and downgrade boto/botocore with it.
• the separate job would test automatically if the "aiobotocore" compatible boto still works with all the unit-tested provider operators - this way any PR that would be based on newer features would fail at PR time.
• then we could make a deliberate decision that this is ok (and add conditional skips in the unit tests) if that happens

So basically this is this trade-off:

1. (current) deferrable operators work out-of-the-box with the "official constraints" (but without latest boto/botocore)
2. (possible) - deferrable operators will not work with official constraints and they require deliberately installing aiobotocore (and downgrading boto/botocore) - but latest botocore is used in those constraints

Those are the two trade-offs, and we can still change the decision (easily) which way go. With #30144 this is as easy as changing few lines in the CI scripts.

[syedahsn]

I think shipping aiobotocore by default is the better option. It will be a smoother user experience if users don't have to change their workflows to use deferrables if their workflows use features not covered by aiobotocore. It's not an ideal situation in either case, but this option is less likely to cause issues for users wanting to use deferrable operators.

[kaxil]

I am with @o-nikolas on this one -- a case where everything works out of the box is better than if a user has to make choices depending.

Also, we want to increase and drive the adoption of Deferrable operators, and decreasing road-blocks for user's adopting it would be my preference

[shubham22]

Agree with the thoughts expressed by @o-nikolas@syedahsn and @kaxil. Priority should be given to providing a better out-of-the-box user experience. If we don't make aiobotocore the default option, it will increase the workload for users, and as Niko mentioned, it may require refactoring of DAGs once the user enables async. On the other hand, we don't have any evidence yet that restricting the botocore version will cause significant customer problems.

Thank you @potiuk for spending time on this and providing us with options. We're certainly converging to a better approach than what we started with.

[potiuk]

Yeah. I did not want to bias it, but that was also my personal preference (but I wanted to objectively show you the options). It seems much easier to explain as well, and I have a feeling that deferring new features by few months is not a huge issue as the adoption is usually anyhow delayed and deferrable operators not being available by default would be a big bummer.

With the #30161 now merged, the aiobotocore will also be included in the PROD image by default, so we are on track to increase the adoption of deferrable operators :).

[pankajastro]

This is awesome, look like we have solution now. Thank you so much @potiuk for stepping-up bringing it in better shape it was long since pending 🚀

[potiuk]

I treat @o-nikolas approval on #29923 as general OK from the AWS team to follow this approach.

[pankajastro]

Address #28850 (comment)

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

I suggest to sync with AWS before doing further work.
We are not yet sure if we should add this lib
#30032 (comment)

[eladkal]

cc @shubham22@kaxil

[pankajastro]

Hi @eladkal, I agree with your concern. I was testing the workaround suggested by @potiuk here #28850 (comment) to run the test

[shubham22]

I see that this is already merged, but can we please wait until 3/22 before making more deferrable contributions? Recent changes from @potiuk in #30144 do alleviate many concerns, but there is always a chance that users won't be able to access new Boto functionality. We are in the process of sharing the raised concerns internally at AWS. However, I do understand that there is no better alternative than using aiobotocore and tradeoff is not bad after the CI job separation. cc: @syedahsn@o-nikolas

[potiuk]

One more thing here. Technically with the job implemented as it is right now wiht my #30144 we have another possibility. We could reverse the approach.

Currently we remove aiobotocore in the separate job and upgrade to latest boto and run all the relevant provider tests.

But we could do it the opposite: we could use latest boto/botocore in "main", remove aibotocore from the "devel" set of dependencies and install aiobotocore (together with downgrading boto/botocore) in the separate job.

This has slightly different characteristics:

• by default the constraints we publish will not have aiobotocore and willhave botocore/boto not compatible with the latest aiobotocore - this means that if someone would like to use deferrable operators from AWS would have to deliberately install aiobotocore and downgrade boto/botocore with it.
• the separate job would test automatically if the "aiobotocore" compatible boto still works with all the unit-tested provider operators - this way any PR that would be based on newer features would fail at PR time.
• then we could make a deliberate decision that this is ok (and add conditional skips in the unit tests) if that happens

So basically this is this trade-off:

1. (current) deferrable operators work out-of-the-box with the "official constraints" (but without latest boto/botocore)
2. (possible) - deferrable operators will not work with official constraints and they require deliberately installing aiobotocore (and downgrading boto/botocore) - but latest botocore is used in those constraints

Those are the two trade-offs, and we can still change the decision (easily) which way go. With #30144 this is as easy as changing few lines in the CI scripts.

[syedahsn]

I think shipping aiobotocore by default is the better option. It will be a smoother user experience if users don't have to change their workflows to use deferrables if their workflows use features not covered by aiobotocore. It's not an ideal situation in either case, but this option is less likely to cause issues for users wanting to use deferrable operators.

[kaxil]

I am with @o-nikolas on this one -- a case where everything works out of the box is better than if a user has to make choices depending.

Also, we want to increase and drive the adoption of Deferrable operators, and decreasing road-blocks for user's adopting it would be my preference

[shubham22]

Agree with the thoughts expressed by @o-nikolas@syedahsn and @kaxil. Priority should be given to providing a better out-of-the-box user experience. If we don't make aiobotocore the default option, it will increase the workload for users, and as Niko mentioned, it may require refactoring of DAGs once the user enables async. On the other hand, we don't have any evidence yet that restricting the botocore version will cause significant customer problems.

Thank you @potiuk for spending time on this and providing us with options. We're certainly converging to a better approach than what we started with.

[potiuk]

Yeah. I did not want to bias it, but that was also my personal preference (but I wanted to objectively show you the options). It seems much easier to explain as well, and I have a feeling that deferring new features by few months is not a huge issue as the adoption is usually anyhow delayed and deferrable operators not being available by default would be a big bummer.

With the #30161 now merged, the aiobotocore will also be included in the PROD image by default, so we are on track to increase the adoption of deferrable operators :).

[pankajastro]

This is awesome, look like we have solution now. Thank you so much @potiuk for stepping-up bringing it in better shape it was long since pending 🚀

[potiuk]

I treat @o-nikolas approval on #29923 as general OK from the AWS team to follow this approach.

[pankajastro]

Address #28850 (comment)

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

I suggest to sync with AWS before doing further work.
We are not yet sure if we should add this lib
#30032 (comment)

[eladkal]

cc @shubham22@kaxil

[pankajastro]

Hi @eladkal, I agree with your concern. I was testing the workaround suggested by @potiuk here #28850 (comment) to run the test

[shubham22]

I see that this is already merged, but can we please wait until 3/22 before making more deferrable contributions? Recent changes from @potiuk in #30144 do alleviate many concerns, but there is always a chance that users won't be able to access new Boto functionality. We are in the process of sharing the raised concerns internally at AWS. However, I do understand that there is no better alternative than using aiobotocore and tradeoff is not bad after the CI job separation. cc: @syedahsn@o-nikolas

[potiuk]

One more thing here. Technically with the job implemented as it is right now wiht my #30144 we have another possibility. We could reverse the approach.

Currently we remove aiobotocore in the separate job and upgrade to latest boto and run all the relevant provider tests.

But we could do it the opposite: we could use latest boto/botocore in "main", remove aibotocore from the "devel" set of dependencies and install aiobotocore (together with downgrading boto/botocore) in the separate job.

This has slightly different characteristics:

• by default the constraints we publish will not have aiobotocore and willhave botocore/boto not compatible with the latest aiobotocore - this means that if someone would like to use deferrable operators from AWS would have to deliberately install aiobotocore and downgrade boto/botocore with it.
• the separate job would test automatically if the "aiobotocore" compatible boto still works with all the unit-tested provider operators - this way any PR that would be based on newer features would fail at PR time.
• then we could make a deliberate decision that this is ok (and add conditional skips in the unit tests) if that happens

So basically this is this trade-off:

1. (current) deferrable operators work out-of-the-box with the "official constraints" (but without latest boto/botocore)
2. (possible) - deferrable operators will not work with official constraints and they require deliberately installing aiobotocore (and downgrading boto/botocore) - but latest botocore is used in those constraints

Those are the two trade-offs, and we can still change the decision (easily) which way go. With #30144 this is as easy as changing few lines in the CI scripts.

[syedahsn]

I think shipping aiobotocore by default is the better option. It will be a smoother user experience if users don't have to change their workflows to use deferrables if their workflows use features not covered by aiobotocore. It's not an ideal situation in either case, but this option is less likely to cause issues for users wanting to use deferrable operators.

[kaxil]

I am with @o-nikolas on this one -- a case where everything works out of the box is better than if a user has to make choices depending.

Also, we want to increase and drive the adoption of Deferrable operators, and decreasing road-blocks for user's adopting it would be my preference

[shubham22]

Agree with the thoughts expressed by @o-nikolas@syedahsn and @kaxil. Priority should be given to providing a better out-of-the-box user experience. If we don't make aiobotocore the default option, it will increase the workload for users, and as Niko mentioned, it may require refactoring of DAGs once the user enables async. On the other hand, we don't have any evidence yet that restricting the botocore version will cause significant customer problems.

Thank you @potiuk for spending time on this and providing us with options. We're certainly converging to a better approach than what we started with.

[potiuk]

Yeah. I did not want to bias it, but that was also my personal preference (but I wanted to objectively show you the options). It seems much easier to explain as well, and I have a feeling that deferring new features by few months is not a huge issue as the adoption is usually anyhow delayed and deferrable operators not being available by default would be a big bummer.

With the #30161 now merged, the aiobotocore will also be included in the PROD image by default, so we are on track to increase the adoption of deferrable operators :).

[pankajastro]

This is awesome, look like we have solution now. Thank you so much @potiuk for stepping-up bringing it in better shape it was long since pending 🚀

[potiuk]

I treat @o-nikolas approval on #29923 as general OK from the AWS team to follow this approach.

[pankajastro]

Address #28850 (comment)

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

I suggest to sync with AWS before doing further work.
We are not yet sure if we should add this lib
#30032 (comment)

[eladkal]

cc @shubham22@kaxil

[pankajastro]

Hi @eladkal, I agree with your concern. I was testing the workaround suggested by @potiuk here #28850 (comment) to run the test

[shubham22]

I see that this is already merged, but can we please wait until 3/22 before making more deferrable contributions? Recent changes from @potiuk in #30144 do alleviate many concerns, but there is always a chance that users won't be able to access new Boto functionality. We are in the process of sharing the raised concerns internally at AWS. However, I do understand that there is no better alternative than using aiobotocore and tradeoff is not bad after the CI job separation. cc: @syedahsn@o-nikolas

[potiuk]

One more thing here. Technically with the job implemented as it is right now wiht my #30144 we have another possibility. We could reverse the approach.

Currently we remove aiobotocore in the separate job and upgrade to latest boto and run all the relevant provider tests.

But we could do it the opposite: we could use latest boto/botocore in "main", remove aibotocore from the "devel" set of dependencies and install aiobotocore (together with downgrading boto/botocore) in the separate job.

This has slightly different characteristics:

• by default the constraints we publish will not have aiobotocore and willhave botocore/boto not compatible with the latest aiobotocore - this means that if someone would like to use deferrable operators from AWS would have to deliberately install aiobotocore and downgrade boto/botocore with it.
• the separate job would test automatically if the "aiobotocore" compatible boto still works with all the unit-tested provider operators - this way any PR that would be based on newer features would fail at PR time.
• then we could make a deliberate decision that this is ok (and add conditional skips in the unit tests) if that happens

So basically this is this trade-off:

1. (current) deferrable operators work out-of-the-box with the "official constraints" (but without latest boto/botocore)
2. (possible) - deferrable operators will not work with official constraints and they require deliberately installing aiobotocore (and downgrading boto/botocore) - but latest botocore is used in those constraints

Those are the two trade-offs, and we can still change the decision (easily) which way go. With #30144 this is as easy as changing few lines in the CI scripts.

[syedahsn]

I think shipping aiobotocore by default is the better option. It will be a smoother user experience if users don't have to change their workflows to use deferrables if their workflows use features not covered by aiobotocore. It's not an ideal situation in either case, but this option is less likely to cause issues for users wanting to use deferrable operators.

[kaxil]

I am with @o-nikolas on this one -- a case where everything works out of the box is better than if a user has to make choices depending.

Also, we want to increase and drive the adoption of Deferrable operators, and decreasing road-blocks for user's adopting it would be my preference

[shubham22]

Agree with the thoughts expressed by @o-nikolas@syedahsn and @kaxil. Priority should be given to providing a better out-of-the-box user experience. If we don't make aiobotocore the default option, it will increase the workload for users, and as Niko mentioned, it may require refactoring of DAGs once the user enables async. On the other hand, we don't have any evidence yet that restricting the botocore version will cause significant customer problems.

Thank you @potiuk for spending time on this and providing us with options. We're certainly converging to a better approach than what we started with.

[potiuk]

Yeah. I did not want to bias it, but that was also my personal preference (but I wanted to objectively show you the options). It seems much easier to explain as well, and I have a feeling that deferring new features by few months is not a huge issue as the adoption is usually anyhow delayed and deferrable operators not being available by default would be a big bummer.

With the #30161 now merged, the aiobotocore will also be included in the PROD image by default, so we are on track to increase the adoption of deferrable operators :).

[pankajastro]

This is awesome, look like we have solution now. Thank you so much @potiuk for stepping-up bringing it in better shape it was long since pending 🚀

[potiuk]

I treat @o-nikolas approval on #29923 as general OK from the AWS team to follow this approach.