Build deferred AWS hooks from the operator's own settings by SEPURI-SAI-KRISHNA · Pull Request #72171 · apache/airflow

SEPURI-SAI-KRISHNA · 2026-08-27T19:20:34Z

Addresses the bulk of the deferred hook-configuration gap tracked in #72144.

AwsBaseWaiterTrigger now builds the hook itself, from the parameters it already serializes, driven by an aws_hook_class attribute, the same arrangement AwsBaseHookMixin gives the operators. Subclasses name the hook they need instead of constructing it, so region_name, verify and botocore_config reach the triggerer by default rather than only where a
subclass remembered to thread them through.

That removes 40 bespoke hook() implementations, and takes the provider from 49 of 113 defer sites forwarding the full hook configuration to 110 of 113.

What is deliberately left out

Three services are reserved for the Contributors Workshop, at the request of the workshop organiser on #72144: sensors/batch.py, sensors/opensearch_serverless.py and operators/sagemaker_unified_studio_notebook.py. They are listed in PENDING_MIGRATION in the invariant test, which asserts each entry is still needed, a stale line fails the suite, so the allowlist cannot outlive the work it tracks.

SageMakerNotebookOperator defers to SageMakerNotebookJobTrigger, which is a plain BaseTrigger whose hook is addressed by execution name and takes no connection parameters at all. EksPodOperator defers to EksPodTrigger, a KubernetesPodTrigger that reaches the pod through a kubeconfig rather than a boto3 client. Neither is an instance of this bug; both are named explicitly in the test rather than passed over silently.

Invariant test

test_deferred_hook_configuration.py walks every self.defer(trigger=...) call in the provider and fails if one does not pass the hook configuration, plus asserts every AwsBaseWaiterTrigger subclass can actually build a hook. It resolves a trigger= expression to every construction it can evaluate to, so a trigger chosen in a conditional expression is checked on both branches, that is how the two EmrContainer sites were caught. A defer site whose trigger is a bare reference is asserted against an explicit allowlist rather than skipped. An operator added later that forgets the parameters fails in CI rather than in production.

Notes for review

Stacked on Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098. No file overlaps, but the invariant test asserts every non-allowlisted site forwards the configuration, and 13 of them are fixed by that PR. This should merge after it.
aws_hook_class binds the hook at class definition, so the @patch("...triggers.<module>.<Hook>") idiom no longer intercepts it for migrated triggers. No existing test needed changing as a result.
EmrContainerTrigger's hook takes an extra virtual_cluster_id, so it overrides _hook_parameters rather than using the default, the escape hatch the base class keeps for exactly this.
EksDeleteClusterTrigger bypasses the base __init__ and rolls its own serialize(), so it sets and serializes the two new parameters explicitly.
Verified against a full run of the operator, sensor and trigger suites: 2463 passed, 3 skipped, with an identical list of 5 failures and 30 collection errors before and after the change (all missing optional dependencies in the local environment, airflow_shared, common.messaging, openlineage).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.

Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.

An AWS operator always carries region_name, verify and botocore_config, but on deferral the trigger builds its own hook. Most triggers accepted none of those parameters and constructed the hook from aws_conn_id alone, so the triggerer silently fell back to boto3 defaults: a different region, default SSL verification, and none of the configured timeouts or retry policy. The task changed behaviour purely by virtue of deferring, and did so without any error. Fixing this service by service would have meant editing every trigger signature as well as every call site, so the hook is now built in one place from the parameters the base trigger already serializes. Subclasses name the hook they need instead of constructing it, which is the same arrangement the operators use. The accompanying invariant test walks every defer site in the provider and fails if one does not hand its hook configuration to the trigger, so an operator added later cannot reintroduce the gap unnoticed. Three services are deliberately left for the Contributors Workshop and are named in the test's allowlist rather than skipped silently.

SEPURI-SAI-KRISHNA · 2026-08-27T19:20:34Z

Addresses the bulk of the deferred hook-configuration gap tracked in #72144.

AwsBaseWaiterTrigger now builds the hook itself, from the parameters it already serializes, driven by an aws_hook_class attribute, the same arrangement AwsBaseHookMixin gives the operators. Subclasses name the hook they need instead of constructing it, so region_name, verify and botocore_config reach the triggerer by default rather than only where a
subclass remembered to thread them through.

That removes 40 bespoke hook() implementations, and takes the provider from 49 of 113 defer sites forwarding the full hook configuration to 110 of 113.

What is deliberately left out

Three services are reserved for the Contributors Workshop, at the request of the workshop organiser on #72144: sensors/batch.py, sensors/opensearch_serverless.py and operators/sagemaker_unified_studio_notebook.py. They are listed in PENDING_MIGRATION in the invariant test, which asserts each entry is still needed, a stale line fails the suite, so the allowlist cannot outlive the work it tracks.

SageMakerNotebookOperator defers to SageMakerNotebookJobTrigger, which is a plain BaseTrigger whose hook is addressed by execution name and takes no connection parameters at all. EksPodOperator defers to EksPodTrigger, a KubernetesPodTrigger that reaches the pod through a kubeconfig rather than a boto3 client. Neither is an instance of this bug; both are named explicitly in the test rather than passed over silently.

Invariant test

test_deferred_hook_configuration.py walks every self.defer(trigger=...) call in the provider and fails if one does not pass the hook configuration, plus asserts every AwsBaseWaiterTrigger subclass can actually build a hook. It resolves a trigger= expression to every construction it can evaluate to, so a trigger chosen in a conditional expression is checked on both branches, that is how the two EmrContainer sites were caught. A defer site whose trigger is a bare reference is asserted against an explicit allowlist rather than skipped. An operator added later that forgets the parameters fails in CI rather than in production.

Notes for review

Stacked on Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098. No file overlaps, but the invariant test asserts every non-allowlisted site forwards the configuration, and 13 of them are fixed by that PR. This should merge after it.
aws_hook_class binds the hook at class definition, so the @patch("...triggers.<module>.<Hook>") idiom no longer intercepts it for migrated triggers. No existing test needed changing as a result.
EmrContainerTrigger's hook takes an extra virtual_cluster_id, so it overrides _hook_parameters rather than using the default, the escape hatch the base class keeps for exactly this.
EksDeleteClusterTrigger bypasses the base __init__ and rolls its own serialize(), so it sets and serializes the two new parameters explicitly.
Verified against a full run of the operator, sensor and trigger suites: 2463 passed, 3 skipped, with an identical list of 5 failures and 30 collection errors before and after the change (all missing optional dependencies in the local environment, airflow_shared, common.messaging, openlineage).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.

Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.

An AWS operator always carries region_name, verify and botocore_config, but on deferral the trigger builds its own hook. Most triggers accepted none of those parameters and constructed the hook from aws_conn_id alone, so the triggerer silently fell back to boto3 defaults: a different region, default SSL verification, and none of the configured timeouts or retry policy. The task changed behaviour purely by virtue of deferring, and did so without any error. Fixing this service by service would have meant editing every trigger signature as well as every call site, so the hook is now built in one place from the parameters the base trigger already serializes. Subclasses name the hook they need instead of constructing it, which is the same arrangement the operators use. The accompanying invariant test walks every defer site in the provider and fails if one does not hand its hook configuration to the trigger, so an operator added later cannot reintroduce the gap unnoticed. Three services are deliberately left for the Contributors Workshop and are named in the test's allowlist rather than skipped silently.

SEPURI-SAI-KRISHNA · 2026-08-27T19:20:34Z

Addresses the bulk of the deferred hook-configuration gap tracked in #72144.

AwsBaseWaiterTrigger now builds the hook itself, from the parameters it already serializes, driven by an aws_hook_class attribute, the same arrangement AwsBaseHookMixin gives the operators. Subclasses name the hook they need instead of constructing it, so region_name, verify and botocore_config reach the triggerer by default rather than only where a
subclass remembered to thread them through.

That removes 40 bespoke hook() implementations, and takes the provider from 49 of 113 defer sites forwarding the full hook configuration to 110 of 113.

What is deliberately left out

Three services are reserved for the Contributors Workshop, at the request of the workshop organiser on #72144: sensors/batch.py, sensors/opensearch_serverless.py and operators/sagemaker_unified_studio_notebook.py. They are listed in PENDING_MIGRATION in the invariant test, which asserts each entry is still needed, a stale line fails the suite, so the allowlist cannot outlive the work it tracks.

SageMakerNotebookOperator defers to SageMakerNotebookJobTrigger, which is a plain BaseTrigger whose hook is addressed by execution name and takes no connection parameters at all. EksPodOperator defers to EksPodTrigger, a KubernetesPodTrigger that reaches the pod through a kubeconfig rather than a boto3 client. Neither is an instance of this bug; both are named explicitly in the test rather than passed over silently.

Invariant test

test_deferred_hook_configuration.py walks every self.defer(trigger=...) call in the provider and fails if one does not pass the hook configuration, plus asserts every AwsBaseWaiterTrigger subclass can actually build a hook. It resolves a trigger= expression to every construction it can evaluate to, so a trigger chosen in a conditional expression is checked on both branches, that is how the two EmrContainer sites were caught. A defer site whose trigger is a bare reference is asserted against an explicit allowlist rather than skipped. An operator added later that forgets the parameters fails in CI rather than in production.

Notes for review

Stacked on Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098. No file overlaps, but the invariant test asserts every non-allowlisted site forwards the configuration, and 13 of them are fixed by that PR. This should merge after it.
aws_hook_class binds the hook at class definition, so the @patch("...triggers.<module>.<Hook>") idiom no longer intercepts it for migrated triggers. No existing test needed changing as a result.
EmrContainerTrigger's hook takes an extra virtual_cluster_id, so it overrides _hook_parameters rather than using the default, the escape hatch the base class keeps for exactly this.
EksDeleteClusterTrigger bypasses the base __init__ and rolls its own serialize(), so it sets and serializes the two new parameters explicitly.
Verified against a full run of the operator, sensor and trigger suites: 2463 passed, 3 skipped, with an identical list of 5 failures and 30 collection errors before and after the change (all missing optional dependencies in the local environment, airflow_shared, common.messaging, openlineage).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.

Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.

An AWS operator always carries region_name, verify and botocore_config, but on deferral the trigger builds its own hook. Most triggers accepted none of those parameters and constructed the hook from aws_conn_id alone, so the triggerer silently fell back to boto3 defaults: a different region, default SSL verification, and none of the configured timeouts or retry policy. The task changed behaviour purely by virtue of deferring, and did so without any error. Fixing this service by service would have meant editing every trigger signature as well as every call site, so the hook is now built in one place from the parameters the base trigger already serializes. Subclasses name the hook they need instead of constructing it, which is the same arrangement the operators use. The accompanying invariant test walks every defer site in the provider and fails if one does not hand its hook configuration to the trigger, so an operator added later cannot reintroduce the gap unnoticed. Three services are deliberately left for the Contributors Workshop and are named in the test's allowlist rather than skipped silently.

SEPURI-SAI-KRISHNA · 2026-08-27T19:20:34Z

Addresses the bulk of the deferred hook-configuration gap tracked in #72144.

AwsBaseWaiterTrigger now builds the hook itself, from the parameters it already serializes, driven by an aws_hook_class attribute, the same arrangement AwsBaseHookMixin gives the operators. Subclasses name the hook they need instead of constructing it, so region_name, verify and botocore_config reach the triggerer by default rather than only where a
subclass remembered to thread them through.

That removes 40 bespoke hook() implementations, and takes the provider from 49 of 113 defer sites forwarding the full hook configuration to 110 of 113.

What is deliberately left out

Three services are reserved for the Contributors Workshop, at the request of the workshop organiser on #72144: sensors/batch.py, sensors/opensearch_serverless.py and operators/sagemaker_unified_studio_notebook.py. They are listed in PENDING_MIGRATION in the invariant test, which asserts each entry is still needed, a stale line fails the suite, so the allowlist cannot outlive the work it tracks.

SageMakerNotebookOperator defers to SageMakerNotebookJobTrigger, which is a plain BaseTrigger whose hook is addressed by execution name and takes no connection parameters at all. EksPodOperator defers to EksPodTrigger, a KubernetesPodTrigger that reaches the pod through a kubeconfig rather than a boto3 client. Neither is an instance of this bug; both are named explicitly in the test rather than passed over silently.

Invariant test

test_deferred_hook_configuration.py walks every self.defer(trigger=...) call in the provider and fails if one does not pass the hook configuration, plus asserts every AwsBaseWaiterTrigger subclass can actually build a hook. It resolves a trigger= expression to every construction it can evaluate to, so a trigger chosen in a conditional expression is checked on both branches, that is how the two EmrContainer sites were caught. A defer site whose trigger is a bare reference is asserted against an explicit allowlist rather than skipped. An operator added later that forgets the parameters fails in CI rather than in production.

Notes for review

Stacked on Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098. No file overlaps, but the invariant test asserts every non-allowlisted site forwards the configuration, and 13 of them are fixed by that PR. This should merge after it.
aws_hook_class binds the hook at class definition, so the @patch("...triggers.<module>.<Hook>") idiom no longer intercepts it for migrated triggers. No existing test needed changing as a result.
EmrContainerTrigger's hook takes an extra virtual_cluster_id, so it overrides _hook_parameters rather than using the default, the escape hatch the base class keeps for exactly this.
EksDeleteClusterTrigger bypasses the base __init__ and rolls its own serialize(), so it sets and serializes the two new parameters explicitly.
Verified against a full run of the operator, sensor and trigger suites: 2463 passed, 3 skipped, with an identical list of 5 failures and 30 collection errors before and after the change (all missing optional dependencies in the local environment, airflow_shared, common.messaging, openlineage).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.

Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.

An AWS operator always carries region_name, verify and botocore_config, but on deferral the trigger builds its own hook. Most triggers accepted none of those parameters and constructed the hook from aws_conn_id alone, so the triggerer silently fell back to boto3 defaults: a different region, default SSL verification, and none of the configured timeouts or retry policy. The task changed behaviour purely by virtue of deferring, and did so without any error. Fixing this service by service would have meant editing every trigger signature as well as every call site, so the hook is now built in one place from the parameters the base trigger already serializes. Subclasses name the hook they need instead of constructing it, which is the same arrangement the operators use. The accompanying invariant test walks every defer site in the provider and fails if one does not hand its hook configuration to the trigger, so an operator added later cannot reintroduce the gap unnoticed. Three services are deliberately left for the Contributors Workshop and are named in the test's allowlist rather than skipped silently.

SEPURI-SAI-KRISHNA · 2026-08-27T19:20:34Z

Addresses the bulk of the deferred hook-configuration gap tracked in #72144.

AwsBaseWaiterTrigger now builds the hook itself, from the parameters it already serializes, driven by an aws_hook_class attribute, the same arrangement AwsBaseHookMixin gives the operators. Subclasses name the hook they need instead of constructing it, so region_name, verify and botocore_config reach the triggerer by default rather than only where a
subclass remembered to thread them through.

That removes 40 bespoke hook() implementations, and takes the provider from 49 of 113 defer sites forwarding the full hook configuration to 110 of 113.

What is deliberately left out

Three services are reserved for the Contributors Workshop, at the request of the workshop organiser on #72144: sensors/batch.py, sensors/opensearch_serverless.py and operators/sagemaker_unified_studio_notebook.py. They are listed in PENDING_MIGRATION in the invariant test, which asserts each entry is still needed, a stale line fails the suite, so the allowlist cannot outlive the work it tracks.

SageMakerNotebookOperator defers to SageMakerNotebookJobTrigger, which is a plain BaseTrigger whose hook is addressed by execution name and takes no connection parameters at all. EksPodOperator defers to EksPodTrigger, a KubernetesPodTrigger that reaches the pod through a kubeconfig rather than a boto3 client. Neither is an instance of this bug; both are named explicitly in the test rather than passed over silently.

Invariant test

test_deferred_hook_configuration.py walks every self.defer(trigger=...) call in the provider and fails if one does not pass the hook configuration, plus asserts every AwsBaseWaiterTrigger subclass can actually build a hook. It resolves a trigger= expression to every construction it can evaluate to, so a trigger chosen in a conditional expression is checked on both branches, that is how the two EmrContainer sites were caught. A defer site whose trigger is a bare reference is asserted against an explicit allowlist rather than skipped. An operator added later that forgets the parameters fails in CI rather than in production.

Notes for review

Stacked on Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098. No file overlaps, but the invariant test asserts every non-allowlisted site forwards the configuration, and 13 of them are fixed by that PR. This should merge after it.
aws_hook_class binds the hook at class definition, so the @patch("...triggers.<module>.<Hook>") idiom no longer intercepts it for migrated triggers. No existing test needed changing as a result.
EmrContainerTrigger's hook takes an extra virtual_cluster_id, so it overrides _hook_parameters rather than using the default, the escape hatch the base class keeps for exactly this.
EksDeleteClusterTrigger bypasses the base __init__ and rolls its own serialize(), so it sets and serializes the two new parameters explicitly.
Verified against a full run of the operator, sensor and trigger suites: 2463 passed, 3 skipped, with an identical list of 5 failures and 30 collection errors before and after the change (all missing optional dependencies in the local environment, airflow_shared, common.messaging, openlineage).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.

Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.

An AWS operator always carries region_name, verify and botocore_config, but on deferral the trigger builds its own hook. Most triggers accepted none of those parameters and constructed the hook from aws_conn_id alone, so the triggerer silently fell back to boto3 defaults: a different region, default SSL verification, and none of the configured timeouts or retry policy. The task changed behaviour purely by virtue of deferring, and did so without any error. Fixing this service by service would have meant editing every trigger signature as well as every call site, so the hook is now built in one place from the parameters the base trigger already serializes. Subclasses name the hook they need instead of constructing it, which is the same arrangement the operators use. The accompanying invariant test walks every defer site in the provider and fails if one does not hand its hook configuration to the trigger, so an operator added later cannot reintroduce the gap unnoticed. Three services are deliberately left for the Contributors Workshop and are named in the test's allowlist rather than skipped silently.

SEPURI-SAI-KRISHNA · 2026-08-27T19:20:34Z

Addresses the bulk of the deferred hook-configuration gap tracked in #72144.

AwsBaseWaiterTrigger now builds the hook itself, from the parameters it already serializes, driven by an aws_hook_class attribute, the same arrangement AwsBaseHookMixin gives the operators. Subclasses name the hook they need instead of constructing it, so region_name, verify and botocore_config reach the triggerer by default rather than only where a
subclass remembered to thread them through.

That removes 40 bespoke hook() implementations, and takes the provider from 49 of 113 defer sites forwarding the full hook configuration to 110 of 113.

What is deliberately left out

Three services are reserved for the Contributors Workshop, at the request of the workshop organiser on #72144: sensors/batch.py, sensors/opensearch_serverless.py and operators/sagemaker_unified_studio_notebook.py. They are listed in PENDING_MIGRATION in the invariant test, which asserts each entry is still needed, a stale line fails the suite, so the allowlist cannot outlive the work it tracks.

SageMakerNotebookOperator defers to SageMakerNotebookJobTrigger, which is a plain BaseTrigger whose hook is addressed by execution name and takes no connection parameters at all. EksPodOperator defers to EksPodTrigger, a KubernetesPodTrigger that reaches the pod through a kubeconfig rather than a boto3 client. Neither is an instance of this bug; both are named explicitly in the test rather than passed over silently.

Invariant test

test_deferred_hook_configuration.py walks every self.defer(trigger=...) call in the provider and fails if one does not pass the hook configuration, plus asserts every AwsBaseWaiterTrigger subclass can actually build a hook. It resolves a trigger= expression to every construction it can evaluate to, so a trigger chosen in a conditional expression is checked on both branches, that is how the two EmrContainer sites were caught. A defer site whose trigger is a bare reference is asserted against an explicit allowlist rather than skipped. An operator added later that forgets the parameters fails in CI rather than in production.

Notes for review

Stacked on Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098. No file overlaps, but the invariant test asserts every non-allowlisted site forwards the configuration, and 13 of them are fixed by that PR. This should merge after it.
aws_hook_class binds the hook at class definition, so the @patch("...triggers.<module>.<Hook>") idiom no longer intercepts it for migrated triggers. No existing test needed changing as a result.
EmrContainerTrigger's hook takes an extra virtual_cluster_id, so it overrides _hook_parameters rather than using the default, the escape hatch the base class keeps for exactly this.
EksDeleteClusterTrigger bypasses the base __init__ and rolls its own serialize(), so it sets and serializes the two new parameters explicitly.
Verified against a full run of the operator, sensor and trigger suites: 2463 passed, 3 skipped, with an identical list of 5 failures and 30 collection errors before and after the change (all missing optional dependencies in the local environment, airflow_shared, common.messaging, openlineage).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.

Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.

An AWS operator always carries region_name, verify and botocore_config, but on deferral the trigger builds its own hook. Most triggers accepted none of those parameters and constructed the hook from aws_conn_id alone, so the triggerer silently fell back to boto3 defaults: a different region, default SSL verification, and none of the configured timeouts or retry policy. The task changed behaviour purely by virtue of deferring, and did so without any error. Fixing this service by service would have meant editing every trigger signature as well as every call site, so the hook is now built in one place from the parameters the base trigger already serializes. Subclasses name the hook they need instead of constructing it, which is the same arrangement the operators use. The accompanying invariant test walks every defer site in the provider and fails if one does not hand its hook configuration to the trigger, so an operator added later cannot reintroduce the gap unnoticed. Three services are deliberately left for the Contributors Workshop and are named in the test's allowlist rather than skipped silently.

SEPURI-SAI-KRISHNA · 2026-08-27T19:20:34Z

Addresses the bulk of the deferred hook-configuration gap tracked in #72144.

AwsBaseWaiterTrigger now builds the hook itself, from the parameters it already serializes, driven by an aws_hook_class attribute, the same arrangement AwsBaseHookMixin gives the operators. Subclasses name the hook they need instead of constructing it, so region_name, verify and botocore_config reach the triggerer by default rather than only where a
subclass remembered to thread them through.

That removes 40 bespoke hook() implementations, and takes the provider from 49 of 113 defer sites forwarding the full hook configuration to 110 of 113.

What is deliberately left out

Three services are reserved for the Contributors Workshop, at the request of the workshop organiser on #72144: sensors/batch.py, sensors/opensearch_serverless.py and operators/sagemaker_unified_studio_notebook.py. They are listed in PENDING_MIGRATION in the invariant test, which asserts each entry is still needed, a stale line fails the suite, so the allowlist cannot outlive the work it tracks.

SageMakerNotebookOperator defers to SageMakerNotebookJobTrigger, which is a plain BaseTrigger whose hook is addressed by execution name and takes no connection parameters at all. EksPodOperator defers to EksPodTrigger, a KubernetesPodTrigger that reaches the pod through a kubeconfig rather than a boto3 client. Neither is an instance of this bug; both are named explicitly in the test rather than passed over silently.

Invariant test

test_deferred_hook_configuration.py walks every self.defer(trigger=...) call in the provider and fails if one does not pass the hook configuration, plus asserts every AwsBaseWaiterTrigger subclass can actually build a hook. It resolves a trigger= expression to every construction it can evaluate to, so a trigger chosen in a conditional expression is checked on both branches, that is how the two EmrContainer sites were caught. A defer site whose trigger is a bare reference is asserted against an explicit allowlist rather than skipped. An operator added later that forgets the parameters fails in CI rather than in production.

Notes for review

Stacked on Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098. No file overlaps, but the invariant test asserts every non-allowlisted site forwards the configuration, and 13 of them are fixed by that PR. This should merge after it.
aws_hook_class binds the hook at class definition, so the @patch("...triggers.<module>.<Hook>") idiom no longer intercepts it for migrated triggers. No existing test needed changing as a result.
EmrContainerTrigger's hook takes an extra virtual_cluster_id, so it overrides _hook_parameters rather than using the default, the escape hatch the base class keeps for exactly this.
EksDeleteClusterTrigger bypasses the base __init__ and rolls its own serialize(), so it sets and serializes the two new parameters explicitly.
Verified against a full run of the operator, sensor and trigger suites: 2463 passed, 3 skipped, with an identical list of 5 failures and 30 collection errors before and after the change (all missing optional dependencies in the local environment, airflow_shared, common.messaging, openlineage).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.

Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.

An AWS operator always carries region_name, verify and botocore_config, but on deferral the trigger builds its own hook. Most triggers accepted none of those parameters and constructed the hook from aws_conn_id alone, so the triggerer silently fell back to boto3 defaults: a different region, default SSL verification, and none of the configured timeouts or retry policy. The task changed behaviour purely by virtue of deferring, and did so without any error. Fixing this service by service would have meant editing every trigger signature as well as every call site, so the hook is now built in one place from the parameters the base trigger already serializes. Subclasses name the hook they need instead of constructing it, which is the same arrangement the operators use. The accompanying invariant test walks every defer site in the provider and fails if one does not hand its hook configuration to the trigger, so an operator added later cannot reintroduce the gap unnoticed. Three services are deliberately left for the Contributors Workshop and are named in the test's allowlist rather than skipped silently.

SEPURI-SAI-KRISHNA · 2026-08-27T19:20:34Z

Addresses the bulk of the deferred hook-configuration gap tracked in #72144.

AwsBaseWaiterTrigger now builds the hook itself, from the parameters it already serializes, driven by an aws_hook_class attribute, the same arrangement AwsBaseHookMixin gives the operators. Subclasses name the hook they need instead of constructing it, so region_name, verify and botocore_config reach the triggerer by default rather than only where a
subclass remembered to thread them through.

That removes 40 bespoke hook() implementations, and takes the provider from 49 of 113 defer sites forwarding the full hook configuration to 110 of 113.

What is deliberately left out

Three services are reserved for the Contributors Workshop, at the request of the workshop organiser on #72144: sensors/batch.py, sensors/opensearch_serverless.py and operators/sagemaker_unified_studio_notebook.py. They are listed in PENDING_MIGRATION in the invariant test, which asserts each entry is still needed, a stale line fails the suite, so the allowlist cannot outlive the work it tracks.

SageMakerNotebookOperator defers to SageMakerNotebookJobTrigger, which is a plain BaseTrigger whose hook is addressed by execution name and takes no connection parameters at all. EksPodOperator defers to EksPodTrigger, a KubernetesPodTrigger that reaches the pod through a kubeconfig rather than a boto3 client. Neither is an instance of this bug; both are named explicitly in the test rather than passed over silently.

Invariant test

test_deferred_hook_configuration.py walks every self.defer(trigger=...) call in the provider and fails if one does not pass the hook configuration, plus asserts every AwsBaseWaiterTrigger subclass can actually build a hook. It resolves a trigger= expression to every construction it can evaluate to, so a trigger chosen in a conditional expression is checked on both branches, that is how the two EmrContainer sites were caught. A defer site whose trigger is a bare reference is asserted against an explicit allowlist rather than skipped. An operator added later that forgets the parameters fails in CI rather than in production.

Notes for review

Stacked on Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098. No file overlaps, but the invariant test asserts every non-allowlisted site forwards the configuration, and 13 of them are fixed by that PR. This should merge after it.
aws_hook_class binds the hook at class definition, so the @patch("...triggers.<module>.<Hook>") idiom no longer intercepts it for migrated triggers. No existing test needed changing as a result.
EmrContainerTrigger's hook takes an extra virtual_cluster_id, so it overrides _hook_parameters rather than using the default, the escape hatch the base class keeps for exactly this.
EksDeleteClusterTrigger bypasses the base __init__ and rolls its own serialize(), so it sets and serializes the two new parameters explicitly.
Verified against a full run of the operator, sensor and trigger suites: 2463 passed, 3 skipped, with an identical list of 5 failures and 30 collection errors before and after the change (all missing optional dependencies in the local environment, airflow_shared, common.messaging, openlineage).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.

Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.

An AWS operator always carries region_name, verify and botocore_config, but on deferral the trigger builds its own hook. Most triggers accepted none of those parameters and constructed the hook from aws_conn_id alone, so the triggerer silently fell back to boto3 defaults: a different region, default SSL verification, and none of the configured timeouts or retry policy. The task changed behaviour purely by virtue of deferring, and did so without any error. Fixing this service by service would have meant editing every trigger signature as well as every call site, so the hook is now built in one place from the parameters the base trigger already serializes. Subclasses name the hook they need instead of constructing it, which is the same arrangement the operators use. The accompanying invariant test walks every defer site in the provider and fails if one does not hand its hook configuration to the trigger, so an operator added later cannot reintroduce the gap unnoticed. Three services are deliberately left for the Contributors Workshop and are named in the test's allowlist rather than skipped silently.

SEPURI-SAI-KRISHNA requested a review from o-nikolas as a code owner August 27, 2026 19:20

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Aug 27, 2026

SEPURI-SAI-KRISHNA mentioned this pull request Aug 27, 2026
AWS deferrable operators drop hook configuration when handing off to triggers #72144
Open
2 tasks

This was referenced Aug 30, 2026
Forward hook config to BatchJobTrigger from the BatchSensor#72278
Open
Forward hook config to the SageMakerUnifiedStudioNotebookTrigger#72279
Open
Forward hook config to OpenSearchServerlessCollectionActiveTrigger#72280
Open

SEPURI-SAI-KRISHNA added 3 commits August 31, 2026 12:56