Use the operator's AWS settings for deferred SageMaker tasks by SEPURI-SAI-KRISHNA · Pull Request #71857 · apache/airflow

SEPURI-SAI-KRISHNA · 2026-08-20T04:15:45Z

SageMakerBaseOperator extends AwsBaseOperator, so every SageMaker operator carries the region_name, verify and botocore_config the user configured. When these operators defer, they build a SageMakerTrigger / SageMakerPipelineTrigger and pass only aws_conn_id, so the trigger builds its hook without those settings.

The result is that the deferred half of the task talks to AWS differently from the operator: a job submitted in an explicitly requested region is then polled in the connection's default region, a custom botocore_config (timeouts, retry policy) is ignored while waiting, and an explicit verify for SSL verification is dropped once the task is handed to the triggerer.

Both triggers already accept all three arguments and forward them to AwsBaseWaiterTrigger, so only the operators needed changing.

This continues the same cleanup as #52904 (Glue), #67508 (Batch), #67876 (Redshift) and #71646 (Neptune).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

SageMaker operators inherit region_name, verify and botocore_config from AwsBaseOperator, but did not hand them to the trigger they defer to. The deferred half of the task then reached AWS with different settings than the operator itself, so an explicitly requested region, a custom botocore config or an explicit SSL verify setting stopped applying once the task was handed to the triggerer.

SEPURI-SAI-KRISHNA · 2026-08-20T04:15:45Z

SageMakerBaseOperator extends AwsBaseOperator, so every SageMaker operator carries the region_name, verify and botocore_config the user configured. When these operators defer, they build a SageMakerTrigger / SageMakerPipelineTrigger and pass only aws_conn_id, so the trigger builds its hook without those settings.

The result is that the deferred half of the task talks to AWS differently from the operator: a job submitted in an explicitly requested region is then polled in the connection's default region, a custom botocore_config (timeouts, retry policy) is ignored while waiting, and an explicit verify for SSL verification is dropped once the task is handed to the triggerer.

Both triggers already accept all three arguments and forward them to AwsBaseWaiterTrigger, so only the operators needed changing.

This continues the same cleanup as #52904 (Glue), #67508 (Batch), #67876 (Redshift) and #71646 (Neptune).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

SageMaker operators inherit region_name, verify and botocore_config from AwsBaseOperator, but did not hand them to the trigger they defer to. The deferred half of the task then reached AWS with different settings than the operator itself, so an explicitly requested region, a custom botocore config or an explicit SSL verify setting stopped applying once the task was handed to the triggerer.

SEPURI-SAI-KRISHNA · 2026-08-20T04:15:45Z

SageMakerBaseOperator extends AwsBaseOperator, so every SageMaker operator carries the region_name, verify and botocore_config the user configured. When these operators defer, they build a SageMakerTrigger / SageMakerPipelineTrigger and pass only aws_conn_id, so the trigger builds its hook without those settings.

The result is that the deferred half of the task talks to AWS differently from the operator: a job submitted in an explicitly requested region is then polled in the connection's default region, a custom botocore_config (timeouts, retry policy) is ignored while waiting, and an explicit verify for SSL verification is dropped once the task is handed to the triggerer.

Both triggers already accept all three arguments and forward them to AwsBaseWaiterTrigger, so only the operators needed changing.

This continues the same cleanup as #52904 (Glue), #67508 (Batch), #67876 (Redshift) and #71646 (Neptune).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

SageMaker operators inherit region_name, verify and botocore_config from AwsBaseOperator, but did not hand them to the trigger they defer to. The deferred half of the task then reached AWS with different settings than the operator itself, so an explicitly requested region, a custom botocore config or an explicit SSL verify setting stopped applying once the task was handed to the triggerer.

SEPURI-SAI-KRISHNA · 2026-08-20T04:15:45Z

SageMakerBaseOperator extends AwsBaseOperator, so every SageMaker operator carries the region_name, verify and botocore_config the user configured. When these operators defer, they build a SageMakerTrigger / SageMakerPipelineTrigger and pass only aws_conn_id, so the trigger builds its hook without those settings.

The result is that the deferred half of the task talks to AWS differently from the operator: a job submitted in an explicitly requested region is then polled in the connection's default region, a custom botocore_config (timeouts, retry policy) is ignored while waiting, and an explicit verify for SSL verification is dropped once the task is handed to the triggerer.

Both triggers already accept all three arguments and forward them to AwsBaseWaiterTrigger, so only the operators needed changing.

This continues the same cleanup as #52904 (Glue), #67508 (Batch), #67876 (Redshift) and #71646 (Neptune).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

SageMaker operators inherit region_name, verify and botocore_config from AwsBaseOperator, but did not hand them to the trigger they defer to. The deferred half of the task then reached AWS with different settings than the operator itself, so an explicitly requested region, a custom botocore config or an explicit SSL verify setting stopped applying once the task was handed to the triggerer.

SEPURI-SAI-KRISHNA · 2026-08-20T04:15:45Z

SageMakerBaseOperator extends AwsBaseOperator, so every SageMaker operator carries the region_name, verify and botocore_config the user configured. When these operators defer, they build a SageMakerTrigger / SageMakerPipelineTrigger and pass only aws_conn_id, so the trigger builds its hook without those settings.

The result is that the deferred half of the task talks to AWS differently from the operator: a job submitted in an explicitly requested region is then polled in the connection's default region, a custom botocore_config (timeouts, retry policy) is ignored while waiting, and an explicit verify for SSL verification is dropped once the task is handed to the triggerer.

Both triggers already accept all three arguments and forward them to AwsBaseWaiterTrigger, so only the operators needed changing.

This continues the same cleanup as #52904 (Glue), #67508 (Batch), #67876 (Redshift) and #71646 (Neptune).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

SageMaker operators inherit region_name, verify and botocore_config from AwsBaseOperator, but did not hand them to the trigger they defer to. The deferred half of the task then reached AWS with different settings than the operator itself, so an explicitly requested region, a custom botocore config or an explicit SSL verify setting stopped applying once the task was handed to the triggerer.

SEPURI-SAI-KRISHNA · 2026-08-20T04:15:45Z

SageMakerBaseOperator extends AwsBaseOperator, so every SageMaker operator carries the region_name, verify and botocore_config the user configured. When these operators defer, they build a SageMakerTrigger / SageMakerPipelineTrigger and pass only aws_conn_id, so the trigger builds its hook without those settings.

The result is that the deferred half of the task talks to AWS differently from the operator: a job submitted in an explicitly requested region is then polled in the connection's default region, a custom botocore_config (timeouts, retry policy) is ignored while waiting, and an explicit verify for SSL verification is dropped once the task is handed to the triggerer.

Both triggers already accept all three arguments and forward them to AwsBaseWaiterTrigger, so only the operators needed changing.

This continues the same cleanup as #52904 (Glue), #67508 (Batch), #67876 (Redshift) and #71646 (Neptune).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

SageMaker operators inherit region_name, verify and botocore_config from AwsBaseOperator, but did not hand them to the trigger they defer to. The deferred half of the task then reached AWS with different settings than the operator itself, so an explicitly requested region, a custom botocore config or an explicit SSL verify setting stopped applying once the task was handed to the triggerer.

SEPURI-SAI-KRISHNA · 2026-08-20T04:15:45Z

SageMakerBaseOperator extends AwsBaseOperator, so every SageMaker operator carries the region_name, verify and botocore_config the user configured. When these operators defer, they build a SageMakerTrigger / SageMakerPipelineTrigger and pass only aws_conn_id, so the trigger builds its hook without those settings.

The result is that the deferred half of the task talks to AWS differently from the operator: a job submitted in an explicitly requested region is then polled in the connection's default region, a custom botocore_config (timeouts, retry policy) is ignored while waiting, and an explicit verify for SSL verification is dropped once the task is handed to the triggerer.

Both triggers already accept all three arguments and forward them to AwsBaseWaiterTrigger, so only the operators needed changing.

This continues the same cleanup as #52904 (Glue), #67508 (Batch), #67876 (Redshift) and #71646 (Neptune).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

SageMaker operators inherit region_name, verify and botocore_config from AwsBaseOperator, but did not hand them to the trigger they defer to. The deferred half of the task then reached AWS with different settings than the operator itself, so an explicitly requested region, a custom botocore config or an explicit SSL verify setting stopped applying once the task was handed to the triggerer.

SEPURI-SAI-KRISHNA · 2026-08-20T04:15:45Z

SageMakerBaseOperator extends AwsBaseOperator, so every SageMaker operator carries the region_name, verify and botocore_config the user configured. When these operators defer, they build a SageMakerTrigger / SageMakerPipelineTrigger and pass only aws_conn_id, so the trigger builds its hook without those settings.

The result is that the deferred half of the task talks to AWS differently from the operator: a job submitted in an explicitly requested region is then polled in the connection's default region, a custom botocore_config (timeouts, retry policy) is ignored while waiting, and an explicit verify for SSL verification is dropped once the task is handed to the triggerer.

Both triggers already accept all three arguments and forward them to AwsBaseWaiterTrigger, so only the operators needed changing.

This continues the same cleanup as #52904 (Glue), #67508 (Batch), #67876 (Redshift) and #71646 (Neptune).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

SageMaker operators inherit region_name, verify and botocore_config from AwsBaseOperator, but did not hand them to the trigger they defer to. The deferred half of the task then reached AWS with different settings than the operator itself, so an explicitly requested region, a custom botocore config or an explicit SSL verify setting stopped applying once the task was handed to the triggerer.

SEPURI-SAI-KRISHNA requested a review from o-nikolas as a code owner August 20, 2026 04:15

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Aug 20, 2026

vincbeck approved these changes Aug 20, 2026
View reviewed changes

vincbeck merged commit 08153a8 into apache:mainAug 21, 2026
83 checks passed

SEPURI-SAI-KRISHNA deleted the fix-sagemaker-trigger-hook-config branch August 26, 2026 03:44

SEPURI-SAI-KRISHNA mentioned this pull request Aug 26, 2026
Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks #72098
Open
1 task

o-nikolas mentioned this pull request Aug 26, 2026
Status of testing Providers that were prepared on August 25, 2026 #72117
Closed
12 tasks

SEPURI-SAI-KRISHNA mentioned this pull request Aug 27, 2026
AWS deferrable operators drop hook configuration when handing off to triggers #72144
Open
2 tasks