sagemaker-core 2.15.0: role validation raises false-positive `RoleValidationError` under condition-based SCPs (IAM simulator can't evaluate conditional SCPs)

[akwoprosper]

PySDK Version

• PySDK V2 (2.x)
• PySDK V3 (3.x)

Reported against the sagemaker-core distribution, version 2.15.0 (repo tag v3.15.0).

Describe the bug

sagemaker-core 2.15.0 added a client-side permission pre-check that runs during high-level construction (e.g. ModelTrainer(...) → TrainDefaults.get_role) before any training job is submitted: resolve_and_validate_role → _evaluate_permissions → iam:SimulatePrincipalPolicy. It raises RoleValidationError on any non-allowed simulate verdict — and that verdict includes the AWS Organizations / SCP layer (OrganizationsDecisionDetail.AllowedByOrganizations).

Per AWS docs, the IAM policy simulator does not evaluate SCPs that have any conditions. So in an account whose organization uses condition-based SCPs, SimulatePrincipalPolicy returns AllowedByOrganizations: false (with EvalDecision: implicitDeny, MatchedStatements: []) for actions that are actually permitted at run time. The pre-check treats this as a definitive denial and raises — a false positive — even though the execution role is correctly configured and the real API call would succeed.

Two observable consequences:

1. Creating a brand-new, fully-permissioned role does not help — the simulate is denied at the org layer regardless of the role's own policies.
2. The same role works fine from a notebook / via a direct create_training_job call, because those paths don't run this client-side pre-check.

2.15.0 is currently the latest published release, so there is no fixed version to upgrade to.

To reproduce

Prerequisites: an AWS account under an organization with at least one condition-based SCP; a training execution role that trusts sagemaker.amazonaws.com and grants the training smoke-test actions at Resource: *; a calling identity that can call iam:SimulatePrincipalPolicy.

pip install 'sagemaker-core==2.15.0'

from sagemaker.core.helper.iam_role_resolver import IamRoleResolver

# Also reproducible via ModelTrainer(...) construction with role_arn set to the same role.
IamRoleResolver().resolve_and_validate_role(
 role_arn="arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role>",
 role_type="training",
)

Result:

RoleValidationError: IAM role 'arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role>' cannot be used for 'training' workloads.
Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions, ec2:DescribeNetworkInterfaces,
ec2:DescribeSecurityGroups, ec2:DescribeSubnets, ec2:DescribeVpcs,
ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:GetAuthorizationToken,
ecr:GetDownloadUrlForLayer

Confirm the verdict is an org-layer artifact rather than a real permission gap:

aws iam simulate-principal-policy \
 --policy-source-arn arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role> \
 --action-names cloudwatch:PutMetricData ec2:CreateNetworkInterface sagemaker:CreateTrainingJob

{
 "EvalActionName": "cloudwatch:PutMetricData",
 "EvalDecision": "implicitDeny",
 "MatchedStatements": [],
 "OrganizationsDecisionDetail": { "AllowedByOrganizations": "false" }
 // ...same for the other actions, including sagemaker:CreateTrainingJob —
 // yet CreateTrainingJob calls from this role succeed at run time (visible in CloudTrail).
}

The identical role runs the same workload successfully from a notebook / via direct API, so the real run-time evaluation permits these actions.

Expected behavior

The pre-check should not hard-fail on an Organizations/SCP-layer denial that the IAM policy simulator cannot faithfully evaluate. Because the simulator ignores condition-based SCPs, an AllowedByOrganizations: false result (with no matched explicit identity Deny) is unverifiable, not authoritative — it should be treated the same as the existing "caller can't call simulate → warn and proceed" path, letting the real API call be the source of truth. An explicit opt-out (e.g. validate_role=False or an env var) would also let users bypass the client-side check without modifying IAM.

Screenshots or logs

.../site-packages/sagemaker/core/helper/iam_role_resolver.py:573 in resolve_and_validate_role
570 # Permission check (definitive denial blocks; unverifiable warns)
571 verdict, denied = _evaluate_permissions(iam_client, role_arn, rol...
572 if verdict is False:
> 573 raise RoleValidationError(
574 _build_validation_error_message(role_arn, role_type, miss...

System information

• SageMaker Python SDK version: sagemaker-core 2.15.0 (repo tag v3.15.0)
• Framework name or algorithm: N/A — fails during role validation, before framework/job selection (framework-agnostic)
• Framework version: N/A
• Python version: 3.12
• CPU or GPU: N/A (fails before job submission)
• Custom Docker image (Y/N): N/A

Additional context

Introduced in sagemaker-core 2.15.0 — file sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py, added in commit dba1127a ("New release (#5969)"), first tag v3.15.0. Authoring PRs: #2041 (added SimulatePrincipalPolicy-based resolve_or_create_role) → #2080 (replaced it with the raising resolve_and_validate_role). #2080 notes it gates only on *-resource "smoke test" actions to avoid false denials on resource-scoped actions, but does not account for the Organizations/SCP layer the simulate call implicitly evaluates — which is the source of this false positive.

Suggested fix: in _evaluate_permissions, when an action is implicitDeny with no matched identity/SCP statement and OrganizationsDecisionDetail.AllowedByOrganizations == false, treat it as unverifiable (warn + proceed) rather than a missing permission; and/or add an explicit opt-out.

Relevant docs:

• IAM policy simulator can't test SCPs with conditions
• IAM policy evaluation logic (explicit Deny overrides Allow)

Workarounds (both verified):

1. Attach an explicit Deny on iam:SimulatePrincipalPolicy to the identity running the SDK — it then skips the pre-check, warns, and proceeds (an explicit Deny is needed to override any existing Allow).
2. Pin sagemaker-core<2.15.0, which predates the pre-check.

[hiSandog]

Could the exact RoleValidationError path from “sagemaker-core 2.15.0: role validation raises false-positive 'RoleValidationError' under condition-based SCPs (IAM…” be retained in the test suite when this is fixed? The report already isolates a useful boundary, and asserting its expected result would help prevent recurrence.

[thejesterscap]

A second trigger for the same RoleValidationError, with a different root cause

This hits the same pre-flight and exception as the report above, but the evidence here doesn't point to an Organizations/SCP-conditional denial — nothing about IAM policy changed between a successful run and a failing one, only the caller's ability to call iam:SimulatePrincipalPolicy did. That isolates the cause to the fixed smoke-test action list itself:

RoleValidationError: IAM role 'arn:aws:iam::…:role/…' cannot be used for 'training' workloads.
Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions,
ec2:DescribeNetworkInterfaces, ec2:DescribeSecurityGroups, ec2:DescribeSubnets,
ec2:DescribeVpcs

Every listed permission is for VPC-attached training. Our jobs set no VpcConfig, so none is needed, and the role deliberately does not carry them.

The part that makes it hard to diagnose

The check has three outcomes, and which one you get depends on the caller's permissions, not on the role being validated:

caller can iam:SimulatePrincipalPolicy?	result
no	verdict None → warn and proceed	fail-open
yes, all allowed	proceed
yes, any denied	RoleValidationError	fail-closed

So the SDK logs Could not verify permissions for role … Proceeding with it. and works. Granting the permission that silences that warning is what breaks it. We granted iam:SimulatePrincipalPolicy specifically to make the pre-flight informative, and it took down every dispatch for 76 minutes.

Cause

sagemaker/core/helper/iam_role_resolver.py. _get_smoke_test_actions() collects actions from a bundled training policy template — the AmazonSageMakerFullAccess shape — and _evaluate_permissions() simulates those, not the permissions the submitted job actually requires. A least-privilege non-VPC role therefore cannot return "allowed".

The function's own docstring states the intent this violates:

Gating only on *-resource actions keeps validation a meaningful "is this a usable SageMaker execution role?" smoke test without false denials.

Restricting to *-resource actions avoids one class of false denial (resource-scoped actions simulated without ResourceArns). It does not avoid this one: the VPC/CloudWatch actions are *-resource in the template, and are simply not required by every training job.

Evidence it is a false denial

Three facts, same role throughout:

• a pipeline execution submitted before the grant trained to completion on that role;
• PutRolePolicy adding iam:SimulatePrincipalPolicy landed at a known timestamp (CloudTrail);
• the next submission, with a docstring-only diff to our pipeline code, died on RoleValidationError.

The only variable was the caller's ability to simulate.

On the pending fix (#6029)

Read the diff before posting this: _is_unverifiable_denial treats an implicitDeny as unverifiable only when OrganizationsDecisionDetail.AllowedByOrganizations or PermissionsBoundaryDecisionDetail.AllowedByPermissionsBoundary comes back explicitly False. Our denial has neither in play — no SCP or permissions boundary is withholding these permissions, the role's own identity policy simply never granted them, by design. #6029 as written would still classify this as denied and still raise. It does add a validate_role=False / SAGEMAKER_VALIDATE_ROLE opt-out, which would give a supported bypass once merged — today the only remedies are widening the role beyond least privilege, revoking iam:SimulatePrincipalPolicy from the caller, or monkey-patching.

Expected behaviour

Do not fail closed on permissions the submitted job does not use. Options, roughly in order of preference: derive the required set from the job's own configuration (VPC actions only when VpcConfig is set); treat a denial as a warning and let the service be the authority; or use #6029's opt-out once it merges.

Versions

package	version
sagemaker	3.18.0
sagemaker-core	2.18.0
sagemaker-train	1.18.0
sagemaker-mlops	1.18.0
sagemaker-serve	1.18.0
boto3 / botocore	1.43.53
pydantic	2.13.4
Python	3.12.3

Note: sagemaker.__version__ no longer exists on v3 — the 3.x wheel is a namespace shim — so version-report snippets that read it raise AttributeError. The table above is per-package.