Add support for deferrable operators in AMPP

[syedahsn]

The purpose of this PR is to add the foundations required to convert AMPP operators to deferrable operators.
The aiobotocore library is being used to make all async calls to the boto3 API.

Rather than create separate Async hooks for every service, we create an async_conn property in base_aws.py which behaves similar to the existing conn property. This property allows us to get access to a client that can make async boto3 API calls. This also allows us to make use of the supporting code that exists for the conn property. The get_client_type function is extended to handle returning a ClientCreatorContext object which can be used to get a client that supports asynchronous calls.

Another addition this PR makes is extending get_waiter in base_aws.py to work with custom async waiters. Currently, the get_waiter function allows us to create a custom waiter using a JSON config file. This PR allows passing a client parameter which can be used to return an async waiter generated by the aiobotocore library. This feature will save a lot of code duplication by standardizing the polling portion of Triggers. A README.md file is included that describes how Triggers can be written which make use of these features. Although not all operators and sensor will be able to make use of these features, there are many in the AMPP that can be written in a standardized format following the methods described.

The RedshiftCreateClusterOperator is chosen to demonstrate how an operator would be modified to become a deferrable operator. We use the built-in cluster_available waiter in the Trigger to asynchronously poll the boto3 API to wait for the cluster to become available.
@Taragolis, @uranusjr , @potiuk , @pankajastro I'd love to hear your thoughts on this approach, and see what you think.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

You need to skip your aiobotocore tests conditionally - see other deferrable tests in aws

[vincbeck]

Interesting. I was expecting to check that self.defer have been called, but this one works as well :)

[vincbeck]

There are some static check failures

[syedahsn]

The purpose of this PR is to add the foundations required to convert AMPP operators to deferrable operators.
The aiobotocore library is being used to make all async calls to the boto3 API.

Rather than create separate Async hooks for every service, we create an async_conn property in base_aws.py which behaves similar to the existing conn property. This property allows us to get access to a client that can make async boto3 API calls. This also allows us to make use of the supporting code that exists for the conn property. The get_client_type function is extended to handle returning a ClientCreatorContext object which can be used to get a client that supports asynchronous calls.

Another addition this PR makes is extending get_waiter in base_aws.py to work with custom async waiters. Currently, the get_waiter function allows us to create a custom waiter using a JSON config file. This PR allows passing a client parameter which can be used to return an async waiter generated by the aiobotocore library. This feature will save a lot of code duplication by standardizing the polling portion of Triggers. A README.md file is included that describes how Triggers can be written which make use of these features. Although not all operators and sensor will be able to make use of these features, there are many in the AMPP that can be written in a standardized format following the methods described.

The RedshiftCreateClusterOperator is chosen to demonstrate how an operator would be modified to become a deferrable operator. We use the built-in cluster_available waiter in the Trigger to asynchronously poll the boto3 API to wait for the cluster to become available.
@Taragolis, @uranusjr , @potiuk , @pankajastro I'd love to hear your thoughts on this approach, and see what you think.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

You need to skip your aiobotocore tests conditionally - see other deferrable tests in aws

[vincbeck]

Interesting. I was expecting to check that self.defer have been called, but this one works as well :)

[vincbeck]

There are some static check failures

[syedahsn]

The purpose of this PR is to add the foundations required to convert AMPP operators to deferrable operators.
The aiobotocore library is being used to make all async calls to the boto3 API.

Rather than create separate Async hooks for every service, we create an async_conn property in base_aws.py which behaves similar to the existing conn property. This property allows us to get access to a client that can make async boto3 API calls. This also allows us to make use of the supporting code that exists for the conn property. The get_client_type function is extended to handle returning a ClientCreatorContext object which can be used to get a client that supports asynchronous calls.

Another addition this PR makes is extending get_waiter in base_aws.py to work with custom async waiters. Currently, the get_waiter function allows us to create a custom waiter using a JSON config file. This PR allows passing a client parameter which can be used to return an async waiter generated by the aiobotocore library. This feature will save a lot of code duplication by standardizing the polling portion of Triggers. A README.md file is included that describes how Triggers can be written which make use of these features. Although not all operators and sensor will be able to make use of these features, there are many in the AMPP that can be written in a standardized format following the methods described.

The RedshiftCreateClusterOperator is chosen to demonstrate how an operator would be modified to become a deferrable operator. We use the built-in cluster_available waiter in the Trigger to asynchronously poll the boto3 API to wait for the cluster to become available.
@Taragolis, @uranusjr , @potiuk , @pankajastro I'd love to hear your thoughts on this approach, and see what you think.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

You need to skip your aiobotocore tests conditionally - see other deferrable tests in aws

[vincbeck]

Interesting. I was expecting to check that self.defer have been called, but this one works as well :)

[vincbeck]

There are some static check failures

[syedahsn]

The purpose of this PR is to add the foundations required to convert AMPP operators to deferrable operators.
The aiobotocore library is being used to make all async calls to the boto3 API.

Rather than create separate Async hooks for every service, we create an async_conn property in base_aws.py which behaves similar to the existing conn property. This property allows us to get access to a client that can make async boto3 API calls. This also allows us to make use of the supporting code that exists for the conn property. The get_client_type function is extended to handle returning a ClientCreatorContext object which can be used to get a client that supports asynchronous calls.

Another addition this PR makes is extending get_waiter in base_aws.py to work with custom async waiters. Currently, the get_waiter function allows us to create a custom waiter using a JSON config file. This PR allows passing a client parameter which can be used to return an async waiter generated by the aiobotocore library. This feature will save a lot of code duplication by standardizing the polling portion of Triggers. A README.md file is included that describes how Triggers can be written which make use of these features. Although not all operators and sensor will be able to make use of these features, there are many in the AMPP that can be written in a standardized format following the methods described.

The RedshiftCreateClusterOperator is chosen to demonstrate how an operator would be modified to become a deferrable operator. We use the built-in cluster_available waiter in the Trigger to asynchronously poll the boto3 API to wait for the cluster to become available.
@Taragolis, @uranusjr , @potiuk , @pankajastro I'd love to hear your thoughts on this approach, and see what you think.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

You need to skip your aiobotocore tests conditionally - see other deferrable tests in aws

[vincbeck]

Interesting. I was expecting to check that self.defer have been called, but this one works as well :)

[vincbeck]

There are some static check failures

[syedahsn]

The purpose of this PR is to add the foundations required to convert AMPP operators to deferrable operators.
The aiobotocore library is being used to make all async calls to the boto3 API.

Rather than create separate Async hooks for every service, we create an async_conn property in base_aws.py which behaves similar to the existing conn property. This property allows us to get access to a client that can make async boto3 API calls. This also allows us to make use of the supporting code that exists for the conn property. The get_client_type function is extended to handle returning a ClientCreatorContext object which can be used to get a client that supports asynchronous calls.

Another addition this PR makes is extending get_waiter in base_aws.py to work with custom async waiters. Currently, the get_waiter function allows us to create a custom waiter using a JSON config file. This PR allows passing a client parameter which can be used to return an async waiter generated by the aiobotocore library. This feature will save a lot of code duplication by standardizing the polling portion of Triggers. A README.md file is included that describes how Triggers can be written which make use of these features. Although not all operators and sensor will be able to make use of these features, there are many in the AMPP that can be written in a standardized format following the methods described.

The RedshiftCreateClusterOperator is chosen to demonstrate how an operator would be modified to become a deferrable operator. We use the built-in cluster_available waiter in the Trigger to asynchronously poll the boto3 API to wait for the cluster to become available.
@Taragolis, @uranusjr , @potiuk , @pankajastro I'd love to hear your thoughts on this approach, and see what you think.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

You need to skip your aiobotocore tests conditionally - see other deferrable tests in aws

[vincbeck]

Interesting. I was expecting to check that self.defer have been called, but this one works as well :)

[vincbeck]

There are some static check failures

[syedahsn]

The purpose of this PR is to add the foundations required to convert AMPP operators to deferrable operators.
The aiobotocore library is being used to make all async calls to the boto3 API.

Rather than create separate Async hooks for every service, we create an async_conn property in base_aws.py which behaves similar to the existing conn property. This property allows us to get access to a client that can make async boto3 API calls. This also allows us to make use of the supporting code that exists for the conn property. The get_client_type function is extended to handle returning a ClientCreatorContext object which can be used to get a client that supports asynchronous calls.

Another addition this PR makes is extending get_waiter in base_aws.py to work with custom async waiters. Currently, the get_waiter function allows us to create a custom waiter using a JSON config file. This PR allows passing a client parameter which can be used to return an async waiter generated by the aiobotocore library. This feature will save a lot of code duplication by standardizing the polling portion of Triggers. A README.md file is included that describes how Triggers can be written which make use of these features. Although not all operators and sensor will be able to make use of these features, there are many in the AMPP that can be written in a standardized format following the methods described.

The RedshiftCreateClusterOperator is chosen to demonstrate how an operator would be modified to become a deferrable operator. We use the built-in cluster_available waiter in the Trigger to asynchronously poll the boto3 API to wait for the cluster to become available.
@Taragolis, @uranusjr , @potiuk , @pankajastro I'd love to hear your thoughts on this approach, and see what you think.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

You need to skip your aiobotocore tests conditionally - see other deferrable tests in aws

[vincbeck]

Interesting. I was expecting to check that self.defer have been called, but this one works as well :)

[vincbeck]

There are some static check failures

[syedahsn]

The purpose of this PR is to add the foundations required to convert AMPP operators to deferrable operators.
The aiobotocore library is being used to make all async calls to the boto3 API.

Rather than create separate Async hooks for every service, we create an async_conn property in base_aws.py which behaves similar to the existing conn property. This property allows us to get access to a client that can make async boto3 API calls. This also allows us to make use of the supporting code that exists for the conn property. The get_client_type function is extended to handle returning a ClientCreatorContext object which can be used to get a client that supports asynchronous calls.

Another addition this PR makes is extending get_waiter in base_aws.py to work with custom async waiters. Currently, the get_waiter function allows us to create a custom waiter using a JSON config file. This PR allows passing a client parameter which can be used to return an async waiter generated by the aiobotocore library. This feature will save a lot of code duplication by standardizing the polling portion of Triggers. A README.md file is included that describes how Triggers can be written which make use of these features. Although not all operators and sensor will be able to make use of these features, there are many in the AMPP that can be written in a standardized format following the methods described.

The RedshiftCreateClusterOperator is chosen to demonstrate how an operator would be modified to become a deferrable operator. We use the built-in cluster_available waiter in the Trigger to asynchronously poll the boto3 API to wait for the cluster to become available.
@Taragolis, @uranusjr , @potiuk , @pankajastro I'd love to hear your thoughts on this approach, and see what you think.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

You need to skip your aiobotocore tests conditionally - see other deferrable tests in aws

[vincbeck]

Interesting. I was expecting to check that self.defer have been called, but this one works as well :)

[vincbeck]

There are some static check failures

[syedahsn]

The purpose of this PR is to add the foundations required to convert AMPP operators to deferrable operators.
The aiobotocore library is being used to make all async calls to the boto3 API.

Rather than create separate Async hooks for every service, we create an async_conn property in base_aws.py which behaves similar to the existing conn property. This property allows us to get access to a client that can make async boto3 API calls. This also allows us to make use of the supporting code that exists for the conn property. The get_client_type function is extended to handle returning a ClientCreatorContext object which can be used to get a client that supports asynchronous calls.

Another addition this PR makes is extending get_waiter in base_aws.py to work with custom async waiters. Currently, the get_waiter function allows us to create a custom waiter using a JSON config file. This PR allows passing a client parameter which can be used to return an async waiter generated by the aiobotocore library. This feature will save a lot of code duplication by standardizing the polling portion of Triggers. A README.md file is included that describes how Triggers can be written which make use of these features. Although not all operators and sensor will be able to make use of these features, there are many in the AMPP that can be written in a standardized format following the methods described.

The RedshiftCreateClusterOperator is chosen to demonstrate how an operator would be modified to become a deferrable operator. We use the built-in cluster_available waiter in the Trigger to asynchronously poll the boto3 API to wait for the cluster to become available.
@Taragolis, @uranusjr , @potiuk , @pankajastro I'd love to hear your thoughts on this approach, and see what you think.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

You need to skip your aiobotocore tests conditionally - see other deferrable tests in aws

[vincbeck]

Interesting. I was expecting to check that self.defer have been called, but this one works as well :)

[vincbeck]

There are some static check failures