Ensure that Connection extra can get masked without causing an error

[ashb]

Fixes#54768

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[VladaZakharova]

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

[potiuk]

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

[potiuk]

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

[ashb]

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

[ashb]

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@taskdefmy_function() ->None:
conn=Connection.get("test")
print(f"{conn.conn_id=}{conn.password=}{conn.extra_dejson=}{conn=}")
withDAG("test_dag") asdag:
my_function()

[potiuk]

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

[amoghrajesh]

Damn!

[amoghrajesh]

@ashb mypy will need a fix:

 task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue" [arg-type]
comms.send(MaskSecret(value=secret, name=name))
^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

[potiuk]

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

[potiuk]

Ah... It's in `apache/airflow" so we can push to it directly

[VladaZakharova]

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

[gopidesupavan]

oh geeee...

[amoghrajesh]

@potiuk you can push directly to the PR. No need to copy it.

[potiuk]

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

[potiuk]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

[potiuk]

Let's just push fixups without -f : to not override each-others's changes.

[amoghrajesh]

Dropped you a message on slack, looking into mypy failures

[potiuk]

pushed mypy fix :)

[VladaZakharova]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

[potiuk]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

[VladaZakharova]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

[amoghrajesh]

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

[potiuk]

fixup with tests is here

[potiuk]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

[amoghrajesh]

Yeah I do not think it is directly linked to this PR

[kaxil]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

[amoghrajesh]

Looks good to me!

[kaxil]

Compat test change: #54776

[potiuk]

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

[ashb]

Fixes#54768

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[VladaZakharova]

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

[potiuk]

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

[potiuk]

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

[ashb]

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

[ashb]

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@taskdefmy_function() ->None:
conn=Connection.get("test")
print(f"{conn.conn_id=}{conn.password=}{conn.extra_dejson=}{conn=}")
withDAG("test_dag") asdag:
my_function()

[potiuk]

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

[amoghrajesh]

Damn!

[amoghrajesh]

@ashb mypy will need a fix:

 task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue" [arg-type]
comms.send(MaskSecret(value=secret, name=name))
^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

[potiuk]

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

[potiuk]

Ah... It's in `apache/airflow" so we can push to it directly

[VladaZakharova]

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

[gopidesupavan]

oh geeee...

[amoghrajesh]

@potiuk you can push directly to the PR. No need to copy it.

[potiuk]

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

[potiuk]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

[potiuk]

Let's just push fixups without -f : to not override each-others's changes.

[amoghrajesh]

Dropped you a message on slack, looking into mypy failures

[potiuk]

pushed mypy fix :)

[VladaZakharova]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

[potiuk]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

[VladaZakharova]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

[amoghrajesh]

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

[potiuk]

fixup with tests is here

[potiuk]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

[amoghrajesh]

Yeah I do not think it is directly linked to this PR

[kaxil]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

[amoghrajesh]

Looks good to me!

[kaxil]

Compat test change: #54776

[potiuk]

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

[ashb]

Fixes#54768

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[VladaZakharova]

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

[potiuk]

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

[potiuk]

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

[ashb]

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

[ashb]

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@taskdefmy_function() ->None:
conn=Connection.get("test")
print(f"{conn.conn_id=}{conn.password=}{conn.extra_dejson=}{conn=}")
withDAG("test_dag") asdag:
my_function()

[potiuk]

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

[amoghrajesh]

Damn!

[amoghrajesh]

@ashb mypy will need a fix:

 task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue" [arg-type]
comms.send(MaskSecret(value=secret, name=name))
^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

[potiuk]

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

[potiuk]

Ah... It's in `apache/airflow" so we can push to it directly

[VladaZakharova]

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

[gopidesupavan]

oh geeee...

[amoghrajesh]

@potiuk you can push directly to the PR. No need to copy it.

[potiuk]

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

[potiuk]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

[potiuk]

Let's just push fixups without -f : to not override each-others's changes.

[amoghrajesh]

Dropped you a message on slack, looking into mypy failures

[potiuk]

pushed mypy fix :)

[VladaZakharova]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

[potiuk]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

[VladaZakharova]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

[amoghrajesh]

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

[potiuk]

fixup with tests is here

[potiuk]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

[amoghrajesh]

Yeah I do not think it is directly linked to this PR

[kaxil]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

[amoghrajesh]

Looks good to me!

[kaxil]

Compat test change: #54776

[potiuk]

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

[ashb]

Fixes#54768

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[VladaZakharova]

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

[potiuk]

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

[potiuk]

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

[ashb]

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

[ashb]

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@taskdefmy_function() ->None:
conn=Connection.get("test")
print(f"{conn.conn_id=}{conn.password=}{conn.extra_dejson=}{conn=}")
withDAG("test_dag") asdag:
my_function()

[potiuk]

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

[amoghrajesh]

Damn!

[amoghrajesh]

@ashb mypy will need a fix:

 task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue" [arg-type]
comms.send(MaskSecret(value=secret, name=name))
^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

[potiuk]

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

[potiuk]

Ah... It's in `apache/airflow" so we can push to it directly

[VladaZakharova]

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

[gopidesupavan]

oh geeee...

[amoghrajesh]

@potiuk you can push directly to the PR. No need to copy it.

[potiuk]

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

[potiuk]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

[potiuk]

Let's just push fixups without -f : to not override each-others's changes.

[amoghrajesh]

Dropped you a message on slack, looking into mypy failures

[potiuk]

pushed mypy fix :)

[VladaZakharova]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

[potiuk]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

[VladaZakharova]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

[amoghrajesh]

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

[potiuk]

fixup with tests is here

[potiuk]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

[amoghrajesh]

Yeah I do not think it is directly linked to this PR

[kaxil]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

[amoghrajesh]

Looks good to me!

[kaxil]

Compat test change: #54776

[potiuk]

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

[ashb]

Fixes#54768

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[VladaZakharova]

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

[potiuk]

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

[potiuk]

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

[ashb]

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

[ashb]

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@taskdefmy_function() ->None:
conn=Connection.get("test")
print(f"{conn.conn_id=}{conn.password=}{conn.extra_dejson=}{conn=}")
withDAG("test_dag") asdag:
my_function()

[potiuk]

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

[amoghrajesh]

Damn!

[amoghrajesh]

@ashb mypy will need a fix:

 task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue" [arg-type]
comms.send(MaskSecret(value=secret, name=name))
^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

[potiuk]

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

[potiuk]

Ah... It's in `apache/airflow" so we can push to it directly

[VladaZakharova]

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

[gopidesupavan]

oh geeee...

[amoghrajesh]

@potiuk you can push directly to the PR. No need to copy it.

[potiuk]

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

[potiuk]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

[potiuk]

Let's just push fixups without -f : to not override each-others's changes.

[amoghrajesh]

Dropped you a message on slack, looking into mypy failures

[potiuk]

pushed mypy fix :)

[VladaZakharova]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

[potiuk]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

[VladaZakharova]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

[amoghrajesh]

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

[potiuk]

fixup with tests is here

[potiuk]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

[amoghrajesh]

Yeah I do not think it is directly linked to this PR

[kaxil]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

[amoghrajesh]

Looks good to me!

[kaxil]

Compat test change: #54776

[potiuk]

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

[ashb]

Fixes#54768

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[VladaZakharova]

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

[potiuk]

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

[potiuk]

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

[ashb]

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

[ashb]

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@taskdefmy_function() ->None:
conn=Connection.get("test")
print(f"{conn.conn_id=}{conn.password=}{conn.extra_dejson=}{conn=}")
withDAG("test_dag") asdag:
my_function()

[potiuk]

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

[amoghrajesh]

Damn!

[amoghrajesh]

@ashb mypy will need a fix:

 task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue" [arg-type]
comms.send(MaskSecret(value=secret, name=name))
^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

[potiuk]

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

[potiuk]

Ah... It's in `apache/airflow" so we can push to it directly

[VladaZakharova]

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

[gopidesupavan]

oh geeee...

[amoghrajesh]

@potiuk you can push directly to the PR. No need to copy it.

[potiuk]

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

[potiuk]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

[potiuk]

Let's just push fixups without -f : to not override each-others's changes.

[amoghrajesh]

Dropped you a message on slack, looking into mypy failures

[potiuk]

pushed mypy fix :)

[VladaZakharova]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

[potiuk]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

[VladaZakharova]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

[amoghrajesh]

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

[potiuk]

fixup with tests is here

[potiuk]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

[amoghrajesh]

Yeah I do not think it is directly linked to this PR

[kaxil]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

[amoghrajesh]

Looks good to me!

[kaxil]

Compat test change: #54776

[potiuk]

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

[ashb]

Fixes#54768

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[VladaZakharova]

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

[potiuk]

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

[potiuk]

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

[ashb]

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

[ashb]

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@taskdefmy_function() ->None:
conn=Connection.get("test")
print(f"{conn.conn_id=}{conn.password=}{conn.extra_dejson=}{conn=}")
withDAG("test_dag") asdag:
my_function()

[potiuk]

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

[amoghrajesh]

Damn!

[amoghrajesh]

@ashb mypy will need a fix:

 task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue" [arg-type]
comms.send(MaskSecret(value=secret, name=name))
^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

[potiuk]

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

[potiuk]

Ah... It's in `apache/airflow" so we can push to it directly

[VladaZakharova]

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

[gopidesupavan]

oh geeee...

[amoghrajesh]

@potiuk you can push directly to the PR. No need to copy it.

[potiuk]

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

[potiuk]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

[potiuk]

Let's just push fixups without -f : to not override each-others's changes.

[amoghrajesh]

Dropped you a message on slack, looking into mypy failures

[potiuk]

pushed mypy fix :)

[VladaZakharova]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

[potiuk]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

[VladaZakharova]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

[amoghrajesh]

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

[potiuk]

fixup with tests is here

[potiuk]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

[amoghrajesh]

Yeah I do not think it is directly linked to this PR

[kaxil]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

[amoghrajesh]

Looks good to me!

[kaxil]

Compat test change: #54776

[potiuk]

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test

[ashb]

Fixes#54768

This was caused by pydantic#9541 and improper testing on my part. Sorry
folks. Thsi happens because Iterable is too open-ended a type

Yes, this should absolutely have unit tests to go with the fix, but a fix is
better than nothing, and I'm about to leave on a camping holiday.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[VladaZakharova]

Hi
thank you for the fix
Did you check this on different types? can you please add unit tests for that?

[potiuk]

Damn. Pydantic runtime type usage has side effects - that change looks lile it changes nothing....

[potiuk]

Hi thank you for the fix Did you check this on different types? can you please add unit tests for that?

@ashb wrote:

Yes, this should absolutely have unit tests to go with the fix, but a fix is better than nothing, and I'm about to leave on a camping holiday.

I guess we should add unit tests as follow-up without breaking @ashb's plans.

[ashb]

Anyone hitting this on 3.0.5 andwho doesn't want to rollback to 3.0.4: a work around for now is to manually apply this patch to your Airflow.

If you are using docker to deploy your airflow then something like this should help hopefully:

USER root
RUN apt update && apt install -y patch patchutils
RUN set -ex; \
cd /usr/local/lib/python3.12/site-packages/airflow; \
curl -L https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/54769.patch \
| filterdiff -p1 -i 'task-sdk/src/airflow/*' | patch -p4 -u --verbose
USER airflow

The path to site-packages and the user to switch back to at the end would need adapting to your specific Dockerfile

[ashb]

Did you check this on different types? can you please add unit tests for that?

I manually tested it with this:

First set the conneciton:

export AIRFLOW_CONN_TEST='{"conn_type": "google_cloud_default", "extra": {"key_path": "/files/airflow-breeze-config/keys2/keys.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "project_id", "num_retries": 6}}

Than I ran this DAG:

@taskdefmy_function() ->None:
conn=Connection.get("test")
print(f"{conn.conn_id=}{conn.password=}{conn.extra_dejson=}{conn=}")
withDAG("test_dag") asdag:
my_function()

[potiuk]

I manually tested it with this:

I will take it from now on @ashb -> thanks for the repro scenarios

[amoghrajesh]

Damn!

[amoghrajesh]

@ashb mypy will need a fix:

 task-sdk/src/airflow/sdk/execution_time/secrets_masker.py:127: error: Argument "value" to "MaskSecret" has incompatible type "str | Iterable[Any]"; expected "JsonValue" [arg-type]
comms.send(MaskSecret(value=secret, name=name))
^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

[potiuk]

^~~~~~
task-sdk/src/airflow/sdk/execution_time/supervisor.py:1351: error: Argument 1 to "mask_secret" has incompatible type "list[JsonValue] | dict[str, JsonValue] | str | int | float | None"; expected
"str | dict[Any, Any] | Iterable[Any]" [arg-type]
mask_secret(msg.value, msg.name)
^~~~~~~~~
Found 2 errors in 2 files (checked 126 source files)

I'll copy the PR and fix it -> and I will ask others to help :) .. @ashb will not be available to fix it @amoghrajesh

[potiuk]

Ah... It's in `apache/airflow" so we can push to it directly

[VladaZakharova]

Originally, there was also the problem with triggerers with the following errors:

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

do you think we can also fix the types for masks to solve this problem too?

[gopidesupavan]

oh geeee...

[amoghrajesh]

@potiuk you can push directly to the PR. No need to copy it.

[potiuk]

@potiuk you can push directly to the PR. No need to copy it.

Yep. Will do

[potiuk]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

[potiuk]

Let's just push fixups without -f : to not override each-others's changes.

[amoghrajesh]

Dropped you a message on slack, looking into mypy failures

[potiuk]

pushed mypy fix :)

[VladaZakharova]

We can work together @amoghrajesh -> and add more related fixes possibly (looking at the last comment from @VladaZakharova ).

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

[potiuk]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

[VladaZakharova]

will be happy to discuss and help, because now it looks like if we extend this logic with types, maybe we can also add support of Enums and other types that are not standard. Because the current logic only supports limited number of objects. The problem with triggers was the first one, and now with this masking it's just making clear with connections that it doesn't work as expected

Let's focus on getting unit testing for that one first - and then we can see what we can do with Enums - this one is pretty burning :).

agree :)

[amoghrajesh]

Agreed too. Let's fix the error at hand first and we can always follow up for the non breaking / burning things...

[potiuk]

fixup with tests is here

[potiuk]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

[amoghrajesh]

Yeah I do not think it is directly linked to this PR

[kaxil]

Also @VladaZakharova -> those seem to be unrelated, it seems that simply we would have to add custom serializer to pydantic to handle those enums and classes that are being serialized. How do you arrive at those errors?

dataplex_data_profile: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataplex_data_quality: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_batch_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
dataproc_spark_deferrable: kills the triggerer (NotImplementedError: Objects of type <enum 'State'> are not supported)
example_gcp_transfer: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_transfer_gcs_to_gcs: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)
example_gcp_transfer_aws: kill the triggerer (NotImplementedError: Objects of type <class 'google.cloud.sotrage_transfer_v1.types.transfer_types.TransferOperation'> are not supported)

Yeah unrelated

[amoghrajesh]

Looks good to me!

[kaxil]

Compat test change: #54776

[potiuk]

Merging it 3.0.5 fails clearly because of yanking :) and #54766 is already on the way.

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test