[v3] FrameworkProcessor and ModelTrainer: 4 regressions (including dropping CodeArtifact support) from v2 migration

[humanzz]

PySDK Version

• PySDK V2 (2.x)
• PySDK V3 (3.x)

Additional context

The following four issues were found during a migration from sagemaker==2.257.1 to sagemaker==3.7.1, moving training jobs from sagemaker.pytorch.PyTorch to sagemaker.train.ModelTrainer and processing jobs from sagemaker.pytorch.processing.PyTorchProcessor to sagemaker.core.processing.FrameworkProcessor.

---

Describe the bug

Four regressions found when migrating from v2 to v3, specifically around FrameworkProcessor (processing jobs) and ModelTrainer (training jobs). All four worked correctly in v2.

System information

• SageMaker Python SDK version: 3.7.1
• Framework name: PyTorch
• Framework version: 2.10
• Python version: 3.13
• CPU or GPU: Both
• Custom Docker image: N

---

Bug 1: wait=True does not respect sagemaker session

Affects: ModelTrainer.train(wait=True) and FrameworkProcessor.run(wait=True)

ProcessingJob.refresh() and TrainingJob.refresh() use Base.get_sagemaker_client() — a global/default client — instead of the sagemaker_session passed to the processor/trainer. This fails with NoCredentialsError when using assumed-role sessions (via STS).

To reproduce:

import boto3
from sagemaker.core.helper.session_helper import Session
from sagemaker.core.processing import FrameworkProcessor
from sagemaker.train import ModelTrainer
from sagemaker.core.training.configs import Compute, SourceCode

# Assumed-role session
sts = boto3.client("sts")
assumed = sts.assume_role(RoleArn="arn:aws:iam::123456789:role/MyRole", RoleSessionName="test")
creds = assumed["Credentials"]
assumed_session = boto3.Session(
 aws_access_key_id=creds["AccessKeyId"],
 aws_secret_access_key=creds["SecretAccessKey"],
 aws_session_token=creds["SessionToken"],
 region_name="us-west-2",
)
sm_session = Session(boto_session=assumed_session)

# FrameworkProcessor — job created OK, wait fails
processor = FrameworkProcessor(
 image_uri="763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:2.10-cpu-py313",
 command=["python3"],
 role="arn:aws:iam::123456789:role/MyRole",
 instance_count=1,
 instance_type="ml.m5.xlarge",
 sagemaker_session=sm_session,
)
processor.run(code="my_script.py", source_dir="src", wait=True)
# → NoCredentialsError

# ModelTrainer — same issue
trainer = ModelTrainer(
 training_image="763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:2.10-cpu-py313",
 role="arn:aws:iam::123456789:role/MyRole",
 source_code=SourceCode(entry_script="train.py", source_dir="src"),
 compute=Compute(instance_type="ml.m5.xlarge", instance_count=1),
 sagemaker_session=sm_session,
)
trainer.train(wait=True)
# → NoCredentialsError

Root cause (sagemaker/core/resources.py):

# ProcessingJob.refresh() / TrainingJob.refresh()
client = Base.get_sagemaker_client() # ← ignores the session
response = client.describe_processing_job(**operation_input_args)

v2 behaviour — ProcessingJob.wait() used the session directly:

def wait(self, logs=True):
 if logs:
 self.sagemaker_session.logs_for_processing_job(self.job_name, wait=True)
 else:
 self.sagemaker_session.wait_for_processing_job(self.job_name)

---

Bug 2: FrameworkProcessor.code_location is accepted but ignored

FrameworkProcessor.__init__ accepts code_location and stores it as self.code_location. The docstring states it controls where code is uploaded. However, _package_code ignores it and always uploads to self.sagemaker_session.default_bucket().

To reproduce:

processor = FrameworkProcessor(
 image_uri="763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:2.10-cpu-py313",
 command=["python3"],
 role="arn:aws:iam::123456789:role/MyRole",
 instance_count=1,
 instance_type="ml.m5.xlarge",
 code_location="s3://my-custom-bucket", # ← ignored
)
processor.run(code="my_script.py", source_dir="src", wait=False)
# Code uploads to s3://sagemaker-us-west-2-123456789/... instead of s3://my-custom-bucket/...

Root cause (sagemaker-core/src/sagemaker/core/processing.py, _package_code):

s3_uri = s3.s3_path_join(
 "s3://",
 self.sagemaker_session.default_bucket(), # ← always uses default bucket
 self.sagemaker_session.default_bucket_prefix or "",
 job_name, "source", "sourcedir.tar.gz",
)

self.code_location is never referenced.

v2 behaviour — FrameworkProcessor delegated to an estimator that honored code_location:

# v2 FrameworkProcessor._create_estimator
return self.estimator_cls(
 ...
 code_location=self.code_location,
 ...
)

Note: ModelTrainer does not offer code_location at all — it always uses session.default_bucket(). Suggested fix: either remove code_location from FrameworkProcessor to align with ModelTrainer, or update _package_code to use it when set.

---

Bugs 3 and 4 are both about losing the ability to install requirements.txt dependencies from a CodeArtifact repository.

1. PyTorch training and inference containers supported this via the CA_REPOSITORY_ARN environment variable (see [feature-request] Support installing dependencies in requirements.txt from CodeArtifact for both training/inference SageMaker Containers deep-learning-containers#2509 for details)
2. feature: Add optional CodeArtifact login to FrameworkProcessing job script #4145 extended that support to processing jobs by exposing codeartifact_repo_arn on FrameworkProcessor.run()

For context, the new ray-based inference containers are also adding codeartifact support via CA_REPOSITORY_ARN environment variable as can be seen in https://github.com/aws/deep-learning-containers/blob/0fc07f317a4db68ff728274070fbe332dde1ca26/scripts/ray/sagemaker_serve.py#L68

---

Bug 3: CodeArtifact support missing from FrameworkProcessor

In v2, FrameworkProcessor.run() accepted codeartifact_repo_arn (added in PR #4145). This configured pip inside the container to authenticate with CodeArtifact before installing requirements.txt.

In v3, FrameworkProcessor.run() does not accept codeartifact_repo_arn, and _generate_framework_script has no CodeArtifact support. The generated runproc.sh runs pip install -r requirements.txt without authentication, failing for packages hosted on private CodeArtifact repositories.

To reproduce:

processor = FrameworkProcessor(
 image_uri="763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:2.10-cpu-py313",
 command=["python3"],
 role="arn:aws:iam::123456789:role/MyRole",
 instance_count=1,
 instance_type="ml.m5.xlarge",
)

# v2 supported: processor.run(..., codeartifact_repo_arn="arn:aws:codeartifact:us-west-2:123:repository/domain/repo")
# v3 does not — parameter doesn't exist

processor.run(code="my_script.py", source_dir="src", wait=False)
# Container fails: pip install -r requirements.txt → package not found (private CodeArtifact repo)

v2 behaviour — _generate_framework_script injected CodeArtifact login into runproc.sh:

if [[ -f 'requirements.txt' ]]; then
 if ! hash aws 2>/dev/null; then
 echo "AWS CLI is not installed. Skipping CodeArtifact login."
 else
 aws codeartifact login --tool pip --domain {domain} --domain-owner {owner} --repository {repository} --region {region}
 fi
 pip install -r requirements.txt
fi

Suggested fix: Port codeartifact_repo_arn and _get_codeartifact_command from v2's PR #4145 into v3's FrameworkProcessor.

---

Bug 4: ModelTrainer bypasses sagemaker-training-toolkit, losing CodeArtifact support for requirements.txt

Affects: ModelTrainer.train() with SourceCode(requirements="requirements.txt")

ModelTrainer overrides the container's ENTRYPOINT with its own sm_train.sh driver script. This bypasses the sagemaker-training-toolkit installed in the container, which handled requirements.txt installation with CodeArtifact support (via the CA_REPOSITORY_ARN environment variable, added in sagemaker-training-toolkit#187).

The v3 sm_train.sh template for requirements installation is a bare pip install:

# from sagemaker.train.templates.INSTALL_REQUIREMENTS
echo "Installing requirements"
$SM_PIP_CMD install -r {requirements_file}

It does not check for CA_REPOSITORY_ARN or configure pip to use CodeArtifact. The env var is passed to the container but nothing reads it.

To reproduce:

from sagemaker.train import ModelTrainer
from sagemaker.core.training.configs import Compute, SourceCode

trainer = ModelTrainer(
 training_image="763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:2.10-cpu-py313",
 role="arn:aws:iam::123456789:role/MyRole",
 source_code=SourceCode(
 entry_script="train.py",
 source_dir="src",
 requirements="requirements.txt", # ← installed without CodeArtifact
 ),
 compute=Compute(instance_type="ml.m5.xlarge", instance_count=1),
 environment={"CA_REPOSITORY_ARN": "arn:aws:codeartifact:us-west-2:123:repository/domain/repo"},
)
trainer.train(wait=False)
# Container runs: pip install -r requirements.txt (using public PyPI, not CodeArtifact)
# Fails in VPC-isolated environments where PyPI is unreachable

v2 behaviour — the PyTorch estimator used the container's native entrypoint, which invoked sagemaker-training-toolkit. The toolkit checked CA_REPOSITORY_ARN, ran aws codeartifact login --tool pip, then installed requirements. This was added in sagemaker-training-toolkit v4.7.0 and tracked in deep-learning-containers#2509.

Suggested fix: The INSTALL_REQUIREMENTS template in sagemaker.train.templates should check for CA_REPOSITORY_ARN and configure pip accordingly before installing, matching the behaviour of sagemaker-training-toolkit.

[humanzz]

The following is a bit more context from an LLM-analysis I did on the codebase, along with historical context given I contributed the codeartifact support before, and proposal for solutions.

CodeArtifact & SageMaker: How It Worked, What Broke, and How to Fix It

---

1. The Problem CodeArtifact Solves
2. How CodeArtifact Support Was Built (2023)
3. The SDK v2 State: How Each Job Type Worked
4. What Broke in SDK v3
5. The Ideal Long-Term Solution (SDK + DLC Updates)
6. The Stopgap Solution (SDK-Only Updates)
7. Appendix: Container Inventory
8. Appendix: The Three Existing Implementations

---

The Problem CodeArtifact Solves

SageMaker training, processing, and inference containers support installing additional Python dependencies from a requirements.txt file bundled with user code. By default, pip install resolves packages from the public PyPI index, which requires internet access.

CodeArtifact solves this by providing a private PyPI mirror. With CodeArtifact:

• Containers don't need internet access (improved security posture)
• Only approved packages are installable (policy enforcement)

The mechanism is simple: set the CA_REPOSITORY_ARN environment variable on the SageMaker job, and the container authenticates with CodeArtifact before running pip install.

---

How CodeArtifact Support Was Built (2023)

CodeArtifact support was contributed through a series of PRs across multiple repositories (see deep-learning-containers#2509 for the original feature request):

1. sagemaker-training-toolkit#187 — Training containers read CA_REPOSITORY_ARN, use boto3 to get an auth token, build an authenticated pip index URL, pass it to pip install -i <url>
2. sagemaker-inference-toolkit#130 — Same approach for inference containers
3. sagemaker-pytorch-inference-toolkit#150 — Made PyTorch inference reuse the base toolkit's logic instead of duplicating it
4. deep-learning-containers#3227 — Updated PyTorch inference container image to pick up the new toolkit
5. sagemaker-python-sdk#4145 — Added codeartifact_repo_arn to SDK v2's FrameworkProcessor.run() for processing jobs

The core authentication logic (parse ARN → boto3 get_authorization_token → build authenticated URL) was implemented identically in both sagemaker-training-toolkit and sagemaker-inference-toolkit. The new DLC ray inference containers (scripts/ray/sagemaker_serve.py) later independently reimplemented the same pattern.

---

The SDK v2 State: How Each Job Type Worked

The Clean Model: Container Owns the Lifecycle

In SDK v2, the SDK's role was simple for training, inference, and tuning: upload user code, set environment variables, and let the container handle everything else. The container's native entrypoint invoked the appropriate toolkit, which handled:

1. Downloading and extracting user code
2. Reading CA_REPOSITORY_ARN from the environment
3. Authenticating with CodeArtifact via boto3
4. Installing requirements.txt with the authenticated pip index
5. Running the user's entry script

This worked because the toolkits were installed in the containers, boto3 was available, and the SDK didn't interfere with the container's entrypoint.

The Exception: Processing Jobs

Processing jobs were the one case where the SDK did override the container's behavior. FrameworkProcessor generated its own bash script (runproc.sh) that handled code extraction and dependency installation. Because this script bypassed the toolkit, it needed its own CodeArtifact implementation.

Since runproc.sh is bash, the SDK v2 used AWS CLI (aws codeartifact login --tool pip) rather than boto3. This was added in PR #4145.

SDK v2 Summary

Job Type	Class	Who owns the lifecycle?	Entrypoint	CodeArtifact mechanism
Training	PyTorch estimator	Container (sagemaker-training-toolkit)	Container's native	boto3 (pip install -i <url>)
Inference	PyTorchModel	Container (sagemaker-inference-toolkit)	Container's native	boto3 (pip install -i <url>)
Tuning	HyperparameterTuner	Container (sagemaker-training-toolkit)	Container's native	boto3 (pip install -i <url>)
Processing	FrameworkProcessor	SDK (generated runproc.sh)	SDK overrides	AWS CLI (aws codeartifact login)

The key insight: only processing jobs bypassed the container's toolkit, and that's the only place the SDK needed its own CodeArtifact implementation.

---

What Broke in SDK v3

Training: ModelTrainer Now Overrides the Entrypoint (Bug 4)

The SDK v3 ModelTrainer follows the same pattern that FrameworkProcessor used — it overrides the container's entrypoint with its own SDK-generated script (sm_train.sh):

# sagemaker-train/src/sagemaker/train/constants.py
DEFAULT_CONTAINER_ENTRYPOINT = ["/bin/bash"]
DEFAULT_CONTAINER_ARGUMENTS = ["-c", "chmod +x .../sm_train.sh && .../sm_train.sh"]

This bypasses sagemaker-training-toolkit entirely. The SDK's sm_train.sh reimplements environment setup, requirements installation, and script execution — but without CodeArtifact support:

# sagemaker-train/src/sagemaker/train/templates.py — INSTALL_REQUIREMENTS (no CodeArtifact)
echo "Installing requirements"
$SM_PIP_CMD install -r {requirements_file}

Source: constants.py, templates.py

The CA_REPOSITORY_ARN env var reaches the container but nothing reads it.

Processing: codeartifact_repo_arn Not Ported (Bug 3)

The SDK v3 FrameworkProcessor was rewritten in sagemaker-core but the codeartifact_repo_arn parameter from PR #4145 was not ported. The generated runproc.sh does a bare pip install without authentication.

Tuning: Not Affected (Due to an API Limitation)

The SDK v3 Tuner wraps ModelTrainer but cannot override the container entrypoint because the AWS CreateHyperParameterTuningJob API doesn't support it.

The CreateHyperParameterTuningJob API has a TrainingJobDefinition field, which might suggest it supports the same options as CreateTrainingJob. However, the types differ despite confusingly similar naming:

API	Field path	Actual type	Has ContainerEntrypoint?
CreateTrainingJob	AlgorithmSpecification	AlgorithmSpecification	✅ Yes (8 fields)
CreateHyperParameterTuningJob	TrainingJobDefinition.AlgorithmSpecification	HyperParameterAlgorithmSpecification	❌ No (4 fields)

Both fields are named AlgorithmSpecification, but they map to different types. HyperParameterAlgorithmSpecification only has TrainingImage, TrainingInputMode, AlgorithmName, and MetricDefinitions. This applies to both TrainingJobDefinition (singular) and TrainingJobDefinitions (plural, for multi-algo tuning).

Because of this, the Tuner falls back to the SDK v2-style approach: setting sagemaker_program and sagemaker_submit_directory as hyperparameters, letting the container's native toolkit handle everything — including CodeArtifact.

Future risk: If AWS adds ContainerEntrypoint to HyperParameterAlgorithmSpecification, the SDK team may update the Tuner to match ModelTrainer's approach, breaking CodeArtifact for tuning too.

Inference: Not Affected (Yet)

The SDK v3 does not override inference container entrypoints. sagemaker-inference-toolkit still handles CA_REPOSITORY_ARN at model server startup. However, if a future SDK v3 inference path follows the ModelTrainer pattern, it would break here too.

SDK v3 Summary

Job Type	Class	Who owns the lifecycle?	Entrypoint	CodeArtifact
Training	ModelTrainer	SDK (generated sm_train.sh)	SDK overrides	❌ Broken
Inference	ModelBuilder	Container (toolkit)	Container's native	✅ Works
Tuning	Tuner (wraps ModelTrainer)	Container (toolkit, due to API limitation)	Container's native	✅ Works (for now)
Processing	FrameworkProcessor	SDK (generated runproc.sh)	SDK overrides	❌ Broken (param not ported)

The pattern is clear: every time the SDK overrides the container entrypoint, CodeArtifact breaks.

---

The Ideal Long-Term Solution (SDK + DLC Updates)

The Root Problem

CodeArtifact authentication logic has been copy-pasted across 4+ codebases:

• sagemaker-training-toolkit (boto3)
• sagemaker-inference-toolkit (boto3)
• sagemaker-python-sdk SDK v2 FrameworkProcessor (AWS CLI)
• deep-learning-containers ray sagemaker_serve.py (boto3)

Each time the SDK takes over a container's entrypoint, this logic needs to be reimplemented again. This doesn't scale.

Proposed Architecture

A dedicated, shared Python package (or well-known module) that handles "configure pip for CodeArtifact from environment variables":

sagemaker-pip-auth (or similar)
├── Depends on boto3 (declared in package dependencies)
├── Reads CA_REPOSITORY_ARN from environment
├── Uses boto3 to get auth token + repository endpoint
├── Returns authenticated pip index URL
└── Provides a CLI entry point for bash callers

Because the package declares boto3 as a dependency, installing it in a container automatically brings boto3 along — containers don't need to independently manage boto3.

Requirements:

1. This package is installed in all SageMaker containers (like the toolkits are today)
2. The toolkits, the SDK's generated scripts, and any future entrypoint overrides all call this one package

Usage from Python (toolkits, SDK driver scripts):

from sagemaker_pip_auth import get_codeartifact_index
index = get_codeartifact_index() # Returns URL or None if CA_REPOSITORY_ARN not set
if index:
 subprocess.check_call([sys.executable, "-m", "pip", "install", "-r", "requirements.txt", "-i", index])

Usage from bash (SDK-generated scripts like runproc.sh, sm_train.sh):

# The package provides a CLI entry point
if [ -n "$CA_REPOSITORY_ARN" ]; then
 CA_INDEX=$(python3 -m sagemaker_pip_auth)
 $SM_PIP_CMD install -r requirements.txt -i "$CA_INDEX"
fi

What This Requires

Action	Owner	Dependency
Create the shared package (with boto3 dependency)	SDK or DLC team	None
Install the package in containers	DLC team	Package must exist first
Update toolkits to use it	DLC team	Package must be in containers
Update SDK scripts to use it	SDK team	Package must be in containers

This requires cross-team coordination, but it's the right long-term answer.

---

The Stopgap Solution (SDK-Only Updates)

Given the current state — the SDK uploads custom entrypoint scripts that bypass the toolkit — we need a solution that works now, without waiting for container changes.

Design: A Self-Contained Python Script

A single Python script, uploaded by the SDK as part of its driver payload, that handles CodeArtifact authentication. It:

1. Checks if CA_REPOSITORY_ARN is set — if not, does nothing
2. Tries boto3 first (the established pattern used by all toolkits)
3. Falls back to AWS CLI if boto3 is not available
4. Hard fails if CA_REPOSITORY_ARN is set but neither boto3 nor AWS CLI is available — silent failure is worse than a clear error

#!/usr/bin/env python3
"""Configure pip for CodeArtifact and install requirements.

Reads CA_REPOSITORY_ARN from environment. Tries boto3 first (matching
sagemaker-training-toolkit), falls back to AWS CLI, hard fails if neither
is available when CA_REPOSITORY_ARN is set.

Usage:
 python3 install_requirements.py requirements.txt
"""
import os
import re
import subprocess
import sys

_ARN_RE = re.compile(
 r"arn:([^:]+):codeartifact:([^:]+):([^:]+):repository/([^/]+)/(.+)"
)


def _get_index_boto3(region, account, domain, repo):
 import boto3
 ca = boto3.client("codeartifact", region_name=region)
 token = ca.get_authorization_token(
 domain=domain, domainOwner=account
 )["authorizationToken"]
 endpoint = ca.get_repository_endpoint(
 domain=domain, domainOwner=account, repository=repo, format="pypi"
 )["repositoryEndpoint"]
 return re.sub(
 "https://", f"https://aws:{token}@",
 re.sub(f"{repo}/?$", f"{repo}/simple/", endpoint),
 )


def _login_awscli(region, account, domain, repo):
 subprocess.check_call([
 "aws", "codeartifact", "login", "--tool", "pip",
 "--domain", domain, "--domain-owner", account,
 "--repository", repo, "--region", region,
 ])


def configure_pip():
 """Returns an authenticated index URL, or None if CA not configured."""
 arn = os.environ.get("CA_REPOSITORY_ARN")
 if not arn:
 return None

 m = _ARN_RE.match(arn)
 if not m:
 raise ValueError(f"Invalid CA_REPOSITORY_ARN: {arn}")
 _, region, account, domain, repo = m.groups()

 # Try boto3 first (matches sagemaker-training-toolkit approach)
 try:
 return _get_index_boto3(region, account, domain, repo)
 except ImportError:
 pass

 # Fall back to AWS CLI
 try:
 _login_awscli(region, account, domain, repo)
 return None # CLI modifies global pip config
 except FileNotFoundError:
 pass

 # Hard fail — CA is configured but we can't authenticate
 print(
 "ERROR: CA_REPOSITORY_ARN is set but neither boto3 nor AWS CLI "
 "is available to authenticate with CodeArtifact.",
 file=sys.stderr,
 )
 sys.exit(1)


def main():
 req_file = sys.argv[1] if len(sys.argv) > 1 else "requirements.txt"
 pip_cmd = [sys.executable, "-m", "pip", "install", "-r", req_file]
 index = configure_pip()
 if index:
 pip_cmd.extend(["-i", index])
 subprocess.check_call(pip_cmd)


if __name__ == "__main__":
 main()

How It's Used

For training (ModelTrainer's sm_train.sh):

The script is uploaded as part of sm_drivers/scripts/. The INSTALL_REQUIREMENTS template becomes:

echo "Installing requirements"
$SM_PYTHON_CMD /opt/ml/input/data/sm_drivers/scripts/install_requirements.py {requirements_file}

For processing (FrameworkProcessor's runproc.sh):

The script is embedded in the source tarball or uploaded alongside it. The generated script becomes:

if [[ -f 'requirements.txt' ]]; then
 python3 install_requirements.py requirements.txt
fi

Same script, both paths. No duplication.

What This Doesn't Require

• ❌ No container changes
• ❌ No cross-team coordination
• ❌ No new packages to install in containers
• ❌ No changes to sagemaker-training-toolkit or sagemaker-inference-toolkit

Behavior Matrix

CA_REPOSITORY_ARN set?	boto3 available?	AWS CLI available?	Result
No	—	—	Normal pip install (no CodeArtifact)
Yes	Yes	—	pip install -i <authenticated_url> (boto3)
Yes	No	Yes	aws codeartifact login then pip install (CLI)
Yes	No	No	Hard fail with clear error message

API Enhancement (Optional Layer)

On top of the stopgap script, the SDK could expose codeartifact_repo_arn as a first-class parameter:

# On SourceCode (for ModelTrainer)
SourceCode(entry_script="train.py", source_dir="src",
 requirements="requirements.txt",
 codeartifact_repo_arn="arn:aws:codeartifact:...")

# On FrameworkProcessor.run()
processor.run(code="script.py", source_dir="src",
 codeartifact_repo_arn="arn:aws:codeartifact:...")

This would:

1. Validate the ARN at submission time (fail fast, not at container runtime)
2. Auto-set CA_REPOSITORY_ARN in the container environment
3. Ensure the install_requirements.py script is included in the upload

---

Appendix: Container Inventory

Current Containers (DLC master branch)

Container	boto3	awscli	sagemaker-training-toolkit	CodeArtifact?
PyTorch Training 2.7–2.9	✅ explicit	✅ explicit	✅ explicit	✅ via toolkit
PyTorch Inference (master)	✅ via toolkit	✅ explicit	— (inference toolkit)	✅ via toolkit

New Containers (DLC main branch)

Container	boto3	awscli	Toolkit	CodeArtifact?
Base v1/v2	❌	❌	❌	❌
Ray Inference	✅ (requirements.lock)	✅ (requirements.lock)	❌	✅ via sagemaker_serve.py
vLLM / SGLang	❌	❌	❌	❌ (no requirements.txt support)

The new DLC base images do NOT include boto3 or awscli. This is why the stopgap script's fallback chain (boto3 → CLI → hard fail) matters — and why the long-term solution uses a shared package that declares boto3 as its own dependency, so installing the package in a container automatically brings boto3 along.

---

Appendix: The Three Existing Implementations

All three use the same core logic — parse the ARN, get a token, build an authenticated URL — but implemented independently:

1. sagemaker-training-toolkit (Python/boto3)

# sagemaker_training/modules.py
# https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/modules.py
client = boto3.client("codeartifact", region_name=region)
token = client.get_authorization_token(domain=domain, domainOwner=owner)["authorizationToken"]
endpoint = client.get_repository_endpoint(domain=domain, domainOwner=owner, repository=repository, format="pypi")["repositoryEndpoint"]
# → pip install -r requirements.txt -i https://aws:{token}@{endpoint}/{repo}/simple/

2. SDK v2 FrameworkProcessor (Bash/AWS CLI)

# Generated runproc.sh
aws codeartifact login --tool pip --domain DOMAIN --domain-owner ACCOUNT --repository REPO --region REGION
pip install -r requirements.txt

3. DLC ray inference container (Python/boto3)

# scripts/ray/sagemaker_serve.py
# https://github.com/aws/deep-learning-containers/blob/main/scripts/ray/sagemaker_serve.py
ca = boto3.client("codeartifact", region_name=region)
token = ca.get_authorization_token(domain=domain, domainOwner=account, durationSeconds=43200)["authorizationToken"]
endpoint = ca.get_repository_endpoint(domain=domain, domainOwner=account, repository=repository, format="pypi")["repositoryEndpoint"]
# → pip install -r requirements.txt --index-url https://aws:{token}@{endpoint}/simple/

The stopgap script consolidates these into one implementation with graceful fallback.

[d-vesely]

I would like to add that I described another regression of the FrameworkProcessor in Issue #5735, regarding the source_dir argument, which used to allow S3 URIs, but now requires a local path.