Dataplex operators

[wojsamjan]

Add support for Google Dataplex. Includes operators, sensors, hooks, example dags, tests and docs.

Authored-by: Wojciech Januszek januszek@google.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

NIT: We prefer to use "%" format for logs, otherwise the string interpolation will be executed indepnendently of the logging level set (yep. I know INFO is default, but it can be changed to ERROR and then it is unnecessary to interpolate it)

[wojsamjan]

Ok, noted

[josh-fell]

Would you mind adding catchup=False? This has been added to all example DAGs to ward off any unexpected DagRuns for users if they copy this DAG for their use and modify start_date or schedule_interval without knowing about the catchup functionality.

[josh-fell]

Could you add parameter/type info in the docstring for this hook? It would be great to see these in the Airflow API documentation which is generated by the docstring.

[josh-fell]

Is this something that users should be able to configure in an Airflow Connection and passed to the hook or is the idea that this key must be configured within the environment itself?

If the latter, should there be a validation here to check that the api_key was provided rather than have the "INVALID API KEY" default value? Or is this default value checked somewhere downstream?

[wojsamjan]

In Airflow Connection I set standard Keyfile Path - pointing to: /files/airflow-breeze-config/keys/<KEY_FILE_NAME>.json On the other hand we have to set an API Key in Credentials on GCP side to connect with discovery API - otherwise we can not perform operations.
I have not seen an option to set this value inside the Airflow Connection - so I used an environment variable. I am open to any suggestions about where it should be stored and any other improvements.

[josh-fell]

Now that 2.2.3 has been released, typing for context can be:

fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:
fromairflow.utils.contextimportContext
...
defexecute(self, context: "Context") ->dict:
...

This can be applied to all of the operators too.

[turbaszek]

TIL 🚀 Thanks @josh-fell

[potiuk]

Yeah. I applied it globally during the Xmas break. Just wonder. Maybe we should add a pre-commit checking if the "old ways" are still used. WDYT @turbaszek@josh-fell ?

[josh-fell]

+1 Definitely worth automating.

[josh-fell]

Same comment here about context typing.

[turbaszek]

Is that necessary?

[wojsamjan]

Yes, API Key is needed to perform operations on dataplex

[turbaszek]

The authentication should be provided via dedicated connection. And as far as I remember GoogleBaseHook already provides all authentication methods supported by Google. If this is something only Dataplex specific we should introduce a new connection type. In this way users will have full control over the credentials. See for example google ads:

airflow/airflow/providers/google/ads/hooks/ads.py

Lines 43 to 68 in 7947b72

	This hook requires two connections:
	- gcp_conn_id - provides service account details (like any other GCP connection)
	- google_ads_conn_id - which contains information from Google Ads config.yaml file
	in the ``extras``. Example of the ``extras``:
	.. code-block:: json
	{
	"google_ads_client": {
	"developer_token": "{{ INSERT_TOKEN }}",
	"path_to_private_key_file": null,
	"delegated_account": "{{ INSERT_DELEGATED_ACCOUNT }}"
	}
	}
	The ``path_to_private_key_file`` is resolved by the hook using credentials from gcp_conn_id.
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on how Google Ads authentication flow works take a look at:
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on the Google Ads API, take a look at the API docs:
	https://developers.google.com/google-ads/api/docs/start

[wojsamjan]

Discussed within the team, for now I am going to remove API_KEY - it was needed for development purposes. Once the Dataplex API will be publicly available it will not be needed any more. I will commit changes and then draft this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[wojsamjan]

@mik-laj@vikramkoka guys could you do a review, would be great. Thank you

[turbaszek]

Looks good to me 👌

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

It needs at least rebase and checking if the errors were accidental.

[wojsamjan]

Add support for Google Dataplex. Includes operators, sensors, hooks, example dags, tests and docs.

Authored-by: Wojciech Januszek januszek@google.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

NIT: We prefer to use "%" format for logs, otherwise the string interpolation will be executed indepnendently of the logging level set (yep. I know INFO is default, but it can be changed to ERROR and then it is unnecessary to interpolate it)

[wojsamjan]

Ok, noted

[josh-fell]

Would you mind adding catchup=False? This has been added to all example DAGs to ward off any unexpected DagRuns for users if they copy this DAG for their use and modify start_date or schedule_interval without knowing about the catchup functionality.

[josh-fell]

Could you add parameter/type info in the docstring for this hook? It would be great to see these in the Airflow API documentation which is generated by the docstring.

[josh-fell]

Is this something that users should be able to configure in an Airflow Connection and passed to the hook or is the idea that this key must be configured within the environment itself?

If the latter, should there be a validation here to check that the api_key was provided rather than have the "INVALID API KEY" default value? Or is this default value checked somewhere downstream?

[wojsamjan]

In Airflow Connection I set standard Keyfile Path - pointing to: /files/airflow-breeze-config/keys/<KEY_FILE_NAME>.json On the other hand we have to set an API Key in Credentials on GCP side to connect with discovery API - otherwise we can not perform operations.
I have not seen an option to set this value inside the Airflow Connection - so I used an environment variable. I am open to any suggestions about where it should be stored and any other improvements.

[josh-fell]

Now that 2.2.3 has been released, typing for context can be:

fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:
fromairflow.utils.contextimportContext
...
defexecute(self, context: "Context") ->dict:
...

This can be applied to all of the operators too.

[turbaszek]

TIL 🚀 Thanks @josh-fell

[potiuk]

Yeah. I applied it globally during the Xmas break. Just wonder. Maybe we should add a pre-commit checking if the "old ways" are still used. WDYT @turbaszek@josh-fell ?

[josh-fell]

+1 Definitely worth automating.

[josh-fell]

Same comment here about context typing.

[turbaszek]

Is that necessary?

[wojsamjan]

Yes, API Key is needed to perform operations on dataplex

[turbaszek]

The authentication should be provided via dedicated connection. And as far as I remember GoogleBaseHook already provides all authentication methods supported by Google. If this is something only Dataplex specific we should introduce a new connection type. In this way users will have full control over the credentials. See for example google ads:

airflow/airflow/providers/google/ads/hooks/ads.py

Lines 43 to 68 in 7947b72

	This hook requires two connections:
	- gcp_conn_id - provides service account details (like any other GCP connection)
	- google_ads_conn_id - which contains information from Google Ads config.yaml file
	in the ``extras``. Example of the ``extras``:
	.. code-block:: json
	{
	"google_ads_client": {
	"developer_token": "{{ INSERT_TOKEN }}",
	"path_to_private_key_file": null,
	"delegated_account": "{{ INSERT_DELEGATED_ACCOUNT }}"
	}
	}
	The ``path_to_private_key_file`` is resolved by the hook using credentials from gcp_conn_id.
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on how Google Ads authentication flow works take a look at:
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on the Google Ads API, take a look at the API docs:
	https://developers.google.com/google-ads/api/docs/start

[wojsamjan]

Discussed within the team, for now I am going to remove API_KEY - it was needed for development purposes. Once the Dataplex API will be publicly available it will not be needed any more. I will commit changes and then draft this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[wojsamjan]

@mik-laj@vikramkoka guys could you do a review, would be great. Thank you

[turbaszek]

Looks good to me 👌

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

It needs at least rebase and checking if the errors were accidental.

[wojsamjan]

Add support for Google Dataplex. Includes operators, sensors, hooks, example dags, tests and docs.

Authored-by: Wojciech Januszek januszek@google.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

NIT: We prefer to use "%" format for logs, otherwise the string interpolation will be executed indepnendently of the logging level set (yep. I know INFO is default, but it can be changed to ERROR and then it is unnecessary to interpolate it)

[wojsamjan]

Ok, noted

[josh-fell]

Would you mind adding catchup=False? This has been added to all example DAGs to ward off any unexpected DagRuns for users if they copy this DAG for their use and modify start_date or schedule_interval without knowing about the catchup functionality.

[josh-fell]

Could you add parameter/type info in the docstring for this hook? It would be great to see these in the Airflow API documentation which is generated by the docstring.

[josh-fell]

Is this something that users should be able to configure in an Airflow Connection and passed to the hook or is the idea that this key must be configured within the environment itself?

If the latter, should there be a validation here to check that the api_key was provided rather than have the "INVALID API KEY" default value? Or is this default value checked somewhere downstream?

[wojsamjan]

In Airflow Connection I set standard Keyfile Path - pointing to: /files/airflow-breeze-config/keys/<KEY_FILE_NAME>.json On the other hand we have to set an API Key in Credentials on GCP side to connect with discovery API - otherwise we can not perform operations.
I have not seen an option to set this value inside the Airflow Connection - so I used an environment variable. I am open to any suggestions about where it should be stored and any other improvements.

[josh-fell]

Now that 2.2.3 has been released, typing for context can be:

fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:
fromairflow.utils.contextimportContext
...
defexecute(self, context: "Context") ->dict:
...

This can be applied to all of the operators too.

[turbaszek]

TIL 🚀 Thanks @josh-fell

[potiuk]

Yeah. I applied it globally during the Xmas break. Just wonder. Maybe we should add a pre-commit checking if the "old ways" are still used. WDYT @turbaszek@josh-fell ?

[josh-fell]

+1 Definitely worth automating.

[josh-fell]

Same comment here about context typing.

[turbaszek]

Is that necessary?

[wojsamjan]

Yes, API Key is needed to perform operations on dataplex

[turbaszek]

The authentication should be provided via dedicated connection. And as far as I remember GoogleBaseHook already provides all authentication methods supported by Google. If this is something only Dataplex specific we should introduce a new connection type. In this way users will have full control over the credentials. See for example google ads:

airflow/airflow/providers/google/ads/hooks/ads.py

Lines 43 to 68 in 7947b72

	This hook requires two connections:
	- gcp_conn_id - provides service account details (like any other GCP connection)
	- google_ads_conn_id - which contains information from Google Ads config.yaml file
	in the ``extras``. Example of the ``extras``:
	.. code-block:: json
	{
	"google_ads_client": {
	"developer_token": "{{ INSERT_TOKEN }}",
	"path_to_private_key_file": null,
	"delegated_account": "{{ INSERT_DELEGATED_ACCOUNT }}"
	}
	}
	The ``path_to_private_key_file`` is resolved by the hook using credentials from gcp_conn_id.
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on how Google Ads authentication flow works take a look at:
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on the Google Ads API, take a look at the API docs:
	https://developers.google.com/google-ads/api/docs/start

[wojsamjan]

Discussed within the team, for now I am going to remove API_KEY - it was needed for development purposes. Once the Dataplex API will be publicly available it will not be needed any more. I will commit changes and then draft this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[wojsamjan]

@mik-laj@vikramkoka guys could you do a review, would be great. Thank you

[turbaszek]

Looks good to me 👌

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

It needs at least rebase and checking if the errors were accidental.

[wojsamjan]

Add support for Google Dataplex. Includes operators, sensors, hooks, example dags, tests and docs.

Authored-by: Wojciech Januszek januszek@google.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

NIT: We prefer to use "%" format for logs, otherwise the string interpolation will be executed indepnendently of the logging level set (yep. I know INFO is default, but it can be changed to ERROR and then it is unnecessary to interpolate it)

[wojsamjan]

Ok, noted

[josh-fell]

Would you mind adding catchup=False? This has been added to all example DAGs to ward off any unexpected DagRuns for users if they copy this DAG for their use and modify start_date or schedule_interval without knowing about the catchup functionality.

[josh-fell]

Could you add parameter/type info in the docstring for this hook? It would be great to see these in the Airflow API documentation which is generated by the docstring.

[josh-fell]

Is this something that users should be able to configure in an Airflow Connection and passed to the hook or is the idea that this key must be configured within the environment itself?

If the latter, should there be a validation here to check that the api_key was provided rather than have the "INVALID API KEY" default value? Or is this default value checked somewhere downstream?

[wojsamjan]

In Airflow Connection I set standard Keyfile Path - pointing to: /files/airflow-breeze-config/keys/<KEY_FILE_NAME>.json On the other hand we have to set an API Key in Credentials on GCP side to connect with discovery API - otherwise we can not perform operations.
I have not seen an option to set this value inside the Airflow Connection - so I used an environment variable. I am open to any suggestions about where it should be stored and any other improvements.

[josh-fell]

Now that 2.2.3 has been released, typing for context can be:

fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:
fromairflow.utils.contextimportContext
...
defexecute(self, context: "Context") ->dict:
...

This can be applied to all of the operators too.

[turbaszek]

TIL 🚀 Thanks @josh-fell

[potiuk]

Yeah. I applied it globally during the Xmas break. Just wonder. Maybe we should add a pre-commit checking if the "old ways" are still used. WDYT @turbaszek@josh-fell ?

[josh-fell]

+1 Definitely worth automating.

[josh-fell]

Same comment here about context typing.

[turbaszek]

Is that necessary?

[wojsamjan]

Yes, API Key is needed to perform operations on dataplex

[turbaszek]

The authentication should be provided via dedicated connection. And as far as I remember GoogleBaseHook already provides all authentication methods supported by Google. If this is something only Dataplex specific we should introduce a new connection type. In this way users will have full control over the credentials. See for example google ads:

airflow/airflow/providers/google/ads/hooks/ads.py

Lines 43 to 68 in 7947b72

	This hook requires two connections:
	- gcp_conn_id - provides service account details (like any other GCP connection)
	- google_ads_conn_id - which contains information from Google Ads config.yaml file
	in the ``extras``. Example of the ``extras``:
	.. code-block:: json
	{
	"google_ads_client": {
	"developer_token": "{{ INSERT_TOKEN }}",
	"path_to_private_key_file": null,
	"delegated_account": "{{ INSERT_DELEGATED_ACCOUNT }}"
	}
	}
	The ``path_to_private_key_file`` is resolved by the hook using credentials from gcp_conn_id.
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on how Google Ads authentication flow works take a look at:
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on the Google Ads API, take a look at the API docs:
	https://developers.google.com/google-ads/api/docs/start

[wojsamjan]

Discussed within the team, for now I am going to remove API_KEY - it was needed for development purposes. Once the Dataplex API will be publicly available it will not be needed any more. I will commit changes and then draft this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[wojsamjan]

@mik-laj@vikramkoka guys could you do a review, would be great. Thank you

[turbaszek]

Looks good to me 👌

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

It needs at least rebase and checking if the errors were accidental.

[wojsamjan]

Add support for Google Dataplex. Includes operators, sensors, hooks, example dags, tests and docs.

Authored-by: Wojciech Januszek januszek@google.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

NIT: We prefer to use "%" format for logs, otherwise the string interpolation will be executed indepnendently of the logging level set (yep. I know INFO is default, but it can be changed to ERROR and then it is unnecessary to interpolate it)

[wojsamjan]

Ok, noted

[josh-fell]

Would you mind adding catchup=False? This has been added to all example DAGs to ward off any unexpected DagRuns for users if they copy this DAG for their use and modify start_date or schedule_interval without knowing about the catchup functionality.

[josh-fell]

Could you add parameter/type info in the docstring for this hook? It would be great to see these in the Airflow API documentation which is generated by the docstring.

[josh-fell]

Is this something that users should be able to configure in an Airflow Connection and passed to the hook or is the idea that this key must be configured within the environment itself?

If the latter, should there be a validation here to check that the api_key was provided rather than have the "INVALID API KEY" default value? Or is this default value checked somewhere downstream?

[wojsamjan]

In Airflow Connection I set standard Keyfile Path - pointing to: /files/airflow-breeze-config/keys/<KEY_FILE_NAME>.json On the other hand we have to set an API Key in Credentials on GCP side to connect with discovery API - otherwise we can not perform operations.
I have not seen an option to set this value inside the Airflow Connection - so I used an environment variable. I am open to any suggestions about where it should be stored and any other improvements.

[josh-fell]

Now that 2.2.3 has been released, typing for context can be:

fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:
fromairflow.utils.contextimportContext
...
defexecute(self, context: "Context") ->dict:
...

This can be applied to all of the operators too.

[turbaszek]

TIL 🚀 Thanks @josh-fell

[potiuk]

Yeah. I applied it globally during the Xmas break. Just wonder. Maybe we should add a pre-commit checking if the "old ways" are still used. WDYT @turbaszek@josh-fell ?

[josh-fell]

+1 Definitely worth automating.

[josh-fell]

Same comment here about context typing.

[turbaszek]

Is that necessary?

[wojsamjan]

Yes, API Key is needed to perform operations on dataplex

[turbaszek]

The authentication should be provided via dedicated connection. And as far as I remember GoogleBaseHook already provides all authentication methods supported by Google. If this is something only Dataplex specific we should introduce a new connection type. In this way users will have full control over the credentials. See for example google ads:

airflow/airflow/providers/google/ads/hooks/ads.py

Lines 43 to 68 in 7947b72

	This hook requires two connections:
	- gcp_conn_id - provides service account details (like any other GCP connection)
	- google_ads_conn_id - which contains information from Google Ads config.yaml file
	in the ``extras``. Example of the ``extras``:
	.. code-block:: json
	{
	"google_ads_client": {
	"developer_token": "{{ INSERT_TOKEN }}",
	"path_to_private_key_file": null,
	"delegated_account": "{{ INSERT_DELEGATED_ACCOUNT }}"
	}
	}
	The ``path_to_private_key_file`` is resolved by the hook using credentials from gcp_conn_id.
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on how Google Ads authentication flow works take a look at:
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on the Google Ads API, take a look at the API docs:
	https://developers.google.com/google-ads/api/docs/start

[wojsamjan]

Discussed within the team, for now I am going to remove API_KEY - it was needed for development purposes. Once the Dataplex API will be publicly available it will not be needed any more. I will commit changes and then draft this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[wojsamjan]

@mik-laj@vikramkoka guys could you do a review, would be great. Thank you

[turbaszek]

Looks good to me 👌

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

It needs at least rebase and checking if the errors were accidental.

[wojsamjan]

Add support for Google Dataplex. Includes operators, sensors, hooks, example dags, tests and docs.

Authored-by: Wojciech Januszek januszek@google.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

NIT: We prefer to use "%" format for logs, otherwise the string interpolation will be executed indepnendently of the logging level set (yep. I know INFO is default, but it can be changed to ERROR and then it is unnecessary to interpolate it)

[wojsamjan]

Ok, noted

[josh-fell]

Would you mind adding catchup=False? This has been added to all example DAGs to ward off any unexpected DagRuns for users if they copy this DAG for their use and modify start_date or schedule_interval without knowing about the catchup functionality.

[josh-fell]

Could you add parameter/type info in the docstring for this hook? It would be great to see these in the Airflow API documentation which is generated by the docstring.

[josh-fell]

Is this something that users should be able to configure in an Airflow Connection and passed to the hook or is the idea that this key must be configured within the environment itself?

If the latter, should there be a validation here to check that the api_key was provided rather than have the "INVALID API KEY" default value? Or is this default value checked somewhere downstream?

[wojsamjan]

In Airflow Connection I set standard Keyfile Path - pointing to: /files/airflow-breeze-config/keys/<KEY_FILE_NAME>.json On the other hand we have to set an API Key in Credentials on GCP side to connect with discovery API - otherwise we can not perform operations.
I have not seen an option to set this value inside the Airflow Connection - so I used an environment variable. I am open to any suggestions about where it should be stored and any other improvements.

[josh-fell]

Now that 2.2.3 has been released, typing for context can be:

fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:
fromairflow.utils.contextimportContext
...
defexecute(self, context: "Context") ->dict:
...

This can be applied to all of the operators too.

[turbaszek]

TIL 🚀 Thanks @josh-fell

[potiuk]

Yeah. I applied it globally during the Xmas break. Just wonder. Maybe we should add a pre-commit checking if the "old ways" are still used. WDYT @turbaszek@josh-fell ?

[josh-fell]

+1 Definitely worth automating.

[josh-fell]

Same comment here about context typing.

[turbaszek]

Is that necessary?

[wojsamjan]

Yes, API Key is needed to perform operations on dataplex

[turbaszek]

The authentication should be provided via dedicated connection. And as far as I remember GoogleBaseHook already provides all authentication methods supported by Google. If this is something only Dataplex specific we should introduce a new connection type. In this way users will have full control over the credentials. See for example google ads:

airflow/airflow/providers/google/ads/hooks/ads.py

Lines 43 to 68 in 7947b72

	This hook requires two connections:
	- gcp_conn_id - provides service account details (like any other GCP connection)
	- google_ads_conn_id - which contains information from Google Ads config.yaml file
	in the ``extras``. Example of the ``extras``:
	.. code-block:: json
	{
	"google_ads_client": {
	"developer_token": "{{ INSERT_TOKEN }}",
	"path_to_private_key_file": null,
	"delegated_account": "{{ INSERT_DELEGATED_ACCOUNT }}"
	}
	}
	The ``path_to_private_key_file`` is resolved by the hook using credentials from gcp_conn_id.
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on how Google Ads authentication flow works take a look at:
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on the Google Ads API, take a look at the API docs:
	https://developers.google.com/google-ads/api/docs/start

[wojsamjan]

Discussed within the team, for now I am going to remove API_KEY - it was needed for development purposes. Once the Dataplex API will be publicly available it will not be needed any more. I will commit changes and then draft this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[wojsamjan]

@mik-laj@vikramkoka guys could you do a review, would be great. Thank you

[turbaszek]

Looks good to me 👌

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

It needs at least rebase and checking if the errors were accidental.

[wojsamjan]

Add support for Google Dataplex. Includes operators, sensors, hooks, example dags, tests and docs.

Authored-by: Wojciech Januszek januszek@google.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

NIT: We prefer to use "%" format for logs, otherwise the string interpolation will be executed indepnendently of the logging level set (yep. I know INFO is default, but it can be changed to ERROR and then it is unnecessary to interpolate it)

[wojsamjan]

Ok, noted

[josh-fell]

Would you mind adding catchup=False? This has been added to all example DAGs to ward off any unexpected DagRuns for users if they copy this DAG for their use and modify start_date or schedule_interval without knowing about the catchup functionality.

[josh-fell]

Could you add parameter/type info in the docstring for this hook? It would be great to see these in the Airflow API documentation which is generated by the docstring.

[josh-fell]

Is this something that users should be able to configure in an Airflow Connection and passed to the hook or is the idea that this key must be configured within the environment itself?

If the latter, should there be a validation here to check that the api_key was provided rather than have the "INVALID API KEY" default value? Or is this default value checked somewhere downstream?

[wojsamjan]

In Airflow Connection I set standard Keyfile Path - pointing to: /files/airflow-breeze-config/keys/<KEY_FILE_NAME>.json On the other hand we have to set an API Key in Credentials on GCP side to connect with discovery API - otherwise we can not perform operations.
I have not seen an option to set this value inside the Airflow Connection - so I used an environment variable. I am open to any suggestions about where it should be stored and any other improvements.

[josh-fell]

Now that 2.2.3 has been released, typing for context can be:

fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:
fromairflow.utils.contextimportContext
...
defexecute(self, context: "Context") ->dict:
...

This can be applied to all of the operators too.

[turbaszek]

TIL 🚀 Thanks @josh-fell

[potiuk]

Yeah. I applied it globally during the Xmas break. Just wonder. Maybe we should add a pre-commit checking if the "old ways" are still used. WDYT @turbaszek@josh-fell ?

[josh-fell]

+1 Definitely worth automating.

[josh-fell]

Same comment here about context typing.

[turbaszek]

Is that necessary?

[wojsamjan]

Yes, API Key is needed to perform operations on dataplex

[turbaszek]

The authentication should be provided via dedicated connection. And as far as I remember GoogleBaseHook already provides all authentication methods supported by Google. If this is something only Dataplex specific we should introduce a new connection type. In this way users will have full control over the credentials. See for example google ads:

airflow/airflow/providers/google/ads/hooks/ads.py

Lines 43 to 68 in 7947b72

	This hook requires two connections:
	- gcp_conn_id - provides service account details (like any other GCP connection)
	- google_ads_conn_id - which contains information from Google Ads config.yaml file
	in the ``extras``. Example of the ``extras``:
	.. code-block:: json
	{
	"google_ads_client": {
	"developer_token": "{{ INSERT_TOKEN }}",
	"path_to_private_key_file": null,
	"delegated_account": "{{ INSERT_DELEGATED_ACCOUNT }}"
	}
	}
	The ``path_to_private_key_file`` is resolved by the hook using credentials from gcp_conn_id.
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on how Google Ads authentication flow works take a look at:
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on the Google Ads API, take a look at the API docs:
	https://developers.google.com/google-ads/api/docs/start

[wojsamjan]

Discussed within the team, for now I am going to remove API_KEY - it was needed for development purposes. Once the Dataplex API will be publicly available it will not be needed any more. I will commit changes and then draft this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[wojsamjan]

@mik-laj@vikramkoka guys could you do a review, would be great. Thank you

[turbaszek]

Looks good to me 👌

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

It needs at least rebase and checking if the errors were accidental.

[wojsamjan]

Add support for Google Dataplex. Includes operators, sensors, hooks, example dags, tests and docs.

Authored-by: Wojciech Januszek januszek@google.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

NIT: We prefer to use "%" format for logs, otherwise the string interpolation will be executed indepnendently of the logging level set (yep. I know INFO is default, but it can be changed to ERROR and then it is unnecessary to interpolate it)

[wojsamjan]

Ok, noted

[josh-fell]

Would you mind adding catchup=False? This has been added to all example DAGs to ward off any unexpected DagRuns for users if they copy this DAG for their use and modify start_date or schedule_interval without knowing about the catchup functionality.

[josh-fell]

Could you add parameter/type info in the docstring for this hook? It would be great to see these in the Airflow API documentation which is generated by the docstring.

[josh-fell]

Is this something that users should be able to configure in an Airflow Connection and passed to the hook or is the idea that this key must be configured within the environment itself?

If the latter, should there be a validation here to check that the api_key was provided rather than have the "INVALID API KEY" default value? Or is this default value checked somewhere downstream?

[wojsamjan]

In Airflow Connection I set standard Keyfile Path - pointing to: /files/airflow-breeze-config/keys/<KEY_FILE_NAME>.json On the other hand we have to set an API Key in Credentials on GCP side to connect with discovery API - otherwise we can not perform operations.
I have not seen an option to set this value inside the Airflow Connection - so I used an environment variable. I am open to any suggestions about where it should be stored and any other improvements.

[josh-fell]

Now that 2.2.3 has been released, typing for context can be:

fromtypingimportTYPE_CHECKINGifTYPE_CHECKING:
fromairflow.utils.contextimportContext
...
defexecute(self, context: "Context") ->dict:
...

This can be applied to all of the operators too.

[turbaszek]

TIL 🚀 Thanks @josh-fell

[potiuk]

Yeah. I applied it globally during the Xmas break. Just wonder. Maybe we should add a pre-commit checking if the "old ways" are still used. WDYT @turbaszek@josh-fell ?

[josh-fell]

+1 Definitely worth automating.

[josh-fell]

Same comment here about context typing.

[turbaszek]

Is that necessary?

[wojsamjan]

Yes, API Key is needed to perform operations on dataplex

[turbaszek]

The authentication should be provided via dedicated connection. And as far as I remember GoogleBaseHook already provides all authentication methods supported by Google. If this is something only Dataplex specific we should introduce a new connection type. In this way users will have full control over the credentials. See for example google ads:

airflow/airflow/providers/google/ads/hooks/ads.py

Lines 43 to 68 in 7947b72

	This hook requires two connections:
	- gcp_conn_id - provides service account details (like any other GCP connection)
	- google_ads_conn_id - which contains information from Google Ads config.yaml file
	in the ``extras``. Example of the ``extras``:
	.. code-block:: json
	{
	"google_ads_client": {
	"developer_token": "{{ INSERT_TOKEN }}",
	"path_to_private_key_file": null,
	"delegated_account": "{{ INSERT_DELEGATED_ACCOUNT }}"
	}
	}
	The ``path_to_private_key_file`` is resolved by the hook using credentials from gcp_conn_id.
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on how Google Ads authentication flow works take a look at:
	https://developers.google.com/google-ads/api/docs/client-libs/python/oauth-service
	.. seealso::
	For more information on the Google Ads API, take a look at the API docs:
	https://developers.google.com/google-ads/api/docs/start

[wojsamjan]

Discussed within the team, for now I am going to remove API_KEY - it was needed for development purposes. Once the Dataplex API will be publicly available it will not be needed any more. I will commit changes and then draft this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[wojsamjan]

@mik-laj@vikramkoka guys could you do a review, would be great. Thank you

[turbaszek]

Looks good to me 👌

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

It needs at least rebase and checking if the errors were accidental.