Uh oh!
There was an error while loading. Please reload this page.
fix(core): resolve default training role from sagemaker config - #6228
fix(core): resolve default training role from sagemaker config#6228jam-jee wants to merge 1 commit into
Conversation
Training (and feature_store) role resolution only inferred a role from the caller identity, so a caller authenticating as an IAM user or the account root - whose identity has no backing role - hit "No IAM role could be resolved from your caller identity" and was forced to pass role= on every call, even when a default execution role was configured in the SageMaker intelligent-defaults config. resolve_and_validate_role now consults the config default (SageMaker.TrainingJob.RoleArn / FeatureGroup.RoleArn) before falling back to caller-identity inference, matching the pattern already used by processing and model monitor. Resolution order is now: explicit role -> config default -> caller identity -> raise. Behavior for a bare IAM user with no configured default is unchanged (still raises the same error). --- X-AI-Prompt: Deep-dive RLVR trainer default-role resolution failing for IAM users; add a sagemaker-config default-role fallback X-AI-Tool: Kiro
Claude finished @jam-jee's task in 1m 45s —— View job Review: resolve default training role from SageMaker configI read the full diff, the surrounding A few observations, none blocking: 1. Behavior change for assumed-role callers (medium — worth documenting, not a bug)
That is arguably correct, but the PR body's claim that the change "cannot break existing callers" is a bit overstated — it changes the resolved role for that (probably small) population. Consider softening the wording and/or calling out the precedence in the user-facing docstring so the switch isn't surprising. 2. Config isn't consulted when |
Problem
When a trainer (e.g.
RLVRTrainer,SFTTrainer) is used without an explicitrole=, and the caller authenticates as an IAM user (or the account root)rather than an assumed role, the SDK fails with:
The user must then pass
role=ROLE_ARNexplicitly on every call, even when theyhave already configured a default execution role in the SageMaker config.
Why it matters
Running the SDK under long-term IAM-user credentials (access key + secret) on a
laptop, CI runner, or on-prem host is common outside SageMaker Studio/notebooks.
Every such caller is forced to hand-thread a role ARN into each trainer call, and
the SageMaker intelligent-defaults config (
SageMaker.TrainingJob.RoleArn) thatexists precisely to hold that default was silently ignored on this path, unlike
processing and model monitor which already honor it.
Fix (symptom -> root cause -> change)
configured default role.
resolve_and_validate_rolejumped straight from "no explicit role"to caller-identity inference (
_resolve_caller_role_arn), which returnsNonefor a user/root ARN, and then raised. It never consulted the config default.
role=is provided, resolve the role type's config default(
SageMaker.TrainingJob.RoleArnfortraining,FeatureGroup.RoleArnforfeature_store) viaresolve_value_from_configbefore falling back tocaller-identity inference. The resolved config role goes through the same
read-only permission/trust validation as any other role.
New resolution order: explicit
role=-> config default -> caller identity -> raise.Config reads are defensive (any failure is swallowed and only a concrete string
ARN is trusted), so the change cannot break existing callers. Behavior for a bare
IAM user with no configured default is unchanged: it still raises the same
error.
Tests
Added to
sagemaker-core/tests/unit/helper/test_iam_role_resolver.py:test_config_default_role_used_when_caller_is_iam_user- an IAM-user caller witha configured default training role resolves to it instead of failing.
test_config_default_role_takes_precedence_over_caller_role- a configureddefault wins over the caller's own backing role.
test_iam_user_without_config_default_still_raises- locks in the unchangedbehavior when no default is configured.
Manual verification
N/A - unit coverage exercises all three branches (config hit, precedence, and the
preserved raise). Full resolver suite: 76 passed.
sagemaker-trainunit suite:147 passed, 1 skipped.
Screenshots
N/A - no user-visible UI change.