SageMaker HyperPod command-line interface

The Amazon SageMaker HyperPod command-line interface (HyperPod CLI) is a tool that helps manage clusters, training jobs, and inference endpoints on the SageMaker HyperPod clusters orchestrated by Amazon EKS.

This documentation serves as a reference for the available HyperPod CLI commands. For a comprehensive user guide, see Orchestrating SageMaker HyperPod clusters with Amazon EKS in the Amazon SageMaker Developer Guide.

Note: Old hyperpodCLI V2 has been moved to release_v2 branch. Please refer release_v2 branch for usage.

Table of Contents

• Overview
• Prerequisites
• Platform Support
• ML Framework Support
• Installation
• Usage
  • Getting Started
  • CLI
    • Cluster Management
    • Training
    • Inference
      • Jumpstart Endpoint
      • Custom Endpoint
    • Space
  • SDK
    • Cluster Management
    • Training
    • Inference
    • Space
• Examples

Overview

The SageMaker HyperPod CLI is a tool that helps create training jobs and inference endpoint deployments to the Amazon SageMaker HyperPod clusters orchestrated by Amazon EKS. It provides a set of commands for managing the full lifecycle of jobs, including create, describe, list, and delete operations, as well as accessing pod and operator logs where applicable. The CLI is designed to abstract away the complexity of working directly with Kubernetes for these core actions of managing jobs on SageMaker HyperPod clusters orchestrated by Amazon EKS.

Prerequisites

Region Configuration

Important: For commands that accept the --region option, if no region is explicitly provided, the command will use the default region from your AWS credentials configuration.

Prerequisites for Training

• HyperPod CLI currently supports starting PyTorchJobs. To start a job, you need to install Training Operator first.
  • You can follow pytorch operator doc to install it.

Prerequisites for Inference

• HyperPod CLI supports creating Inference Endpoints through jumpstart and through custom Endpoint config
  • You can follow inference operator doc to install it.

Platform Support

SageMaker HyperPod CLI currently supports Linux and MacOS platforms. Windows platform is not supported now.

ML Framework Support

SageMaker HyperPod CLI currently supports start training job with:

• PyTorch ML Framework. Version requirements: PyTorch >= 1.10

Installation

1. Make sure that your local python version is 3.8, 3.9, 3.10 or 3.11.

2. Install the sagemaker-hyperpod-cli package.

   pip install sagemaker-hyperpod

3. Verify if the installation succeeded by running the following command.

   hyp --help

Usage

The HyperPod CLI provides the following commands:

• Getting Started
• CLI
  • Cluster Management
  • Training
  • Inference
    • Jumpstart Endpoint
    • Custom Endpoint
• SDK
  • Cluster Management
  • Training
  • Inference

Getting Started

Getting Cluster information

This command lists the available SageMaker HyperPod clusters and their capacity information.

hyp list-cluster

Option	Type	Description
--region <region>	Optional	The region that the SageMaker HyperPod and EKS clusters are located. If not specified, it will be set to the region from the current AWS account credentials.
--namespace <namespace>	Optional	The namespace that users want to check the quota with. Only the SageMaker managed namespaces are supported.
--output <json|table>	Optional	The output format. Available values are table and json. The default value is json.
--debug	Optional	Enable debug mode for detailed logging.

Connecting to a Cluster

This command configures the local Kubectl environment to interact with the specified SageMaker HyperPod cluster and namespace.

hyp set-cluster-context --cluster-name <cluster-name>

Option	Type	Description
--cluster-name <cluster-name>	Required	The SageMaker HyperPod cluster name to configure with.
--namespace <namespace>	Optional	The namespace that you want to connect to. If not specified, Hyperpod cli commands will auto discover the accessible namespace.
--region <region>	Optional	The AWS region where the HyperPod cluster resides.
--debug	Optional	Enable debug mode for detailed logging.

Getting Cluster Context

Get all the context related to the current set Cluster

hyp get-cluster-context

Option	Type	Description
--debug	Optional	Enable debug mode for detailed logging.

CLI

Cluster Management

Important: For commands that accept the --region option, if no region is explicitly provided, the command will use the default region from your AWS credentials configuration.

Cluster stack names must be unique within each AWS region. If you attempt to create a cluster stack with a name that already exists in the same region, the deployment will fail.

Initialize Cluster Configuration

Initialize a new cluster configuration in the current directory:

hyp init cluster-stack

Important: The resource_name_prefix parameter in the generated config.yaml file serves as the primary identifier for all AWS resources created during deployment. Each deployment must use a unique resource name prefix to avoid conflicts. This prefix is automatically appended with a unique identifier during cluster creation to ensure resource uniqueness.

Configure Cluster Parameters

Configure cluster parameters interactively or via command line:

hyp configure --resource-name-prefix my-cluster --stage prod

Validate Configuration

Validate the configuration file syntax:

hyp validate

Create Cluster Stack

Create the cluster stack using the configured parameters:

hyp create --region <region>

Note: The region flag is optional. If not provided, the command will use the default region from your AWS credentials configuration.

List Cluster Stacks

hyp list cluster-stack

Option	Type	Description
--region <region>	Optional	The AWS region to list stacks from.
--status "['CREATE_COMPLETE', 'UPDATE_COMPLETE']"	Optional	Filter by stack status.
--debug	Optional	Enable debug mode for detailed logging.

Describe Cluster Stack

hyp describe cluster-stack <stack-name>

Option	Type	Description
--region <region>	Optional	The AWS region where the stack exists.
--debug	Optional	Enable debug mode for detailed logging.

Delete Cluster Stack

Delete a HyperPod cluster stack. Removes the specified CloudFormation stack and all associated AWS resources. This operation cannot be undone.

 hyp delete cluster-stack <stack-name>

Option	Type	Description
--region <region>	Required	The AWS region where the stack exists.
--retain-resources S3Bucket-TrainingData,EFSFileSystem-Models	Optional	Comma-separated list of logical resource IDs to retain during deletion (only works on DELETE_FAILED stacks). Resource names are shown in failed deletion output, or use AWS CLI: aws cloudformation list-stack-resources STACK_NAME --region REGION.
--debug	Optional	Enable debug mode for detailed logging.

Update Existing Cluster

hyp update cluster --cluster-name my-cluster \
--instance-groups '[{"InstanceCount":2,"InstanceGroupName":"worker-nodes","InstanceType":"ml.m5.large"}]' \
--node-recovery Automatic

Reset Configuration

Reset configuration to default values:

hyp reset

Training

Option 1: Create Pytorch job through init experience

Initialize Pytorch Job Configuration

Initialize a new pytorch job configuration in the current directory:

hyp init hyp-pytorch-job

Configure Pytorch Job Parameters

Configure pytorch job parameters interactively or via command line:

hyp configure --job-name my-pytorch-job

Validate Configuration

Validate the configuration file syntax:

hyp validate

Create Pytorch Job

Create the pytorch job using the configured parameters:

hyp create

Option 2: Create Pytorch job through create command

hyp create hyp-pytorch-job \
--version 1.0 \
--job-name test-pytorch-job \
--image pytorch/pytorch:latest \
--command '[python, train.py]' \
--args '[--epochs=10, --batch-size=32]' \
--environment '{"PYTORCH_CUDA_ALLOC_CONF": "max_split_size_mb:32"}' \
--pull-policy "IfNotPresent" \
--instance-type ml.p4d.24xlarge \
--tasks-per-node 8 \
--label-selector '{"accelerator": "nvidia", "network": "efa"}' \
--deep-health-check-passed-nodes-only true \
--scheduler-type "kueue" \
--queue-name "training-queue" \
--priority "high" \
--max-retry 3 \
--accelerators 8 \
--vcpu 96.0 \
--memory 1152.0 \
--accelerators-limit 8 \
--vcpu-limit 96.0 \
--memory-limit 1152.0 \
--preferred-topology "topology.kubernetes.io/zone=us-west-2a" \
--volume name=model-data,type=hostPath,mount_path=/data,path=/data \
--volume name=training-output,type=pvc,mount_path=/data2,claim_name=my-pvc,read_only=false

Example with accelerator parititons:

hyp create hyp-pytorch-job \
--version 1.1 \
--job-name test-pytorch-job \
--image pytorch/pytorch:latest \
--command '[python, train.py]' \
--args '[--epochs=10, --batch-size=32]' \
--environment '{"PYTORCH_CUDA_ALLOC_CONF": "max_split_size_mb:32"}' \
--pull-policy "IfNotPresent" \
--instance-type ml.p4d.24xlarge \
--tasks-per-node 8 \
--label-selector '{"accelerator": "nvidia", "network": "efa"}' \
--deep-health-check-passed-nodes-only true \
--scheduler-type "kueue" \
--queue-name "training-queue" \
--priority "high" \
--max-retry 3 \
--accelerator-partition-type "mig-1g.5gb" \
--accelerator-partition-count 2 \
--accelerator-partition-limit 4 \
--vcpu 96.0 \
--memory 1152.0 \
--vcpu-limit 96.0 \
--memory-limit 1152.0 \
--preferred-topology "topology.kubernetes.io/zone=us-west-2a" \
--volume name=model-data,type=hostPath,mount_path=/data,path=/data \
--volume name=training-output,type=pvc,mount_path=/data2,claim_name=my-pvc,read_only=false

Parameter	Type	Required	Description
--job-name	TEXT	Yes	Unique name for the training job (1-63 characters, alphanumeric with hyphens)
--image	TEXT	Yes	Docker image URI containing your training code
--namespace	TEXT	No	Kubernetes namespace
--command	ARRAY	No	Command to run in the container (array of strings)
--args	ARRAY	No	Arguments for the entry script (array of strings)
--environment	OBJECT	No	Environment variables as key-value pairs
--pull-policy	TEXT	No	Image pull policy (Always, Never, IfNotPresent)
--instance-type	TEXT	No	Instance type for training
--node-count	INTEGER	No	Number of nodes (minimum: 1)
--tasks-per-node	INTEGER	No	Number of tasks per node (minimum: 1)
--label-selector	OBJECT	No	Node label selector as key-value pairs
--deep-health-check-passed-nodes-only	BOOLEAN	No	Schedule pods only on nodes that passed deep health check (default: false)
--scheduler-type	TEXT	No	Scheduler type
--queue-name	TEXT	No	Queue name for job scheduling (1-63 characters, alphanumeric with hyphens)
--priority	TEXT	No	Priority class for job scheduling
--max-retry	INTEGER	No	Maximum number of job retries (minimum: 0)
--volume	ARRAY	No	List of volume configurations (Refer Volume Configuration for detailed parameter info)
--service-account-name	TEXT	No	Service account name
--accelerators	INTEGER	No	Number of accelerators a.k.a GPUs or Trainium Chips
--vcpu	FLOAT	No	Number of vCPUs
--memory	FLOAT	No	Amount of memory in GiB
--accelerators-limit	INTEGER	No	Limit for the number of accelerators a.k.a GPUs or Trainium Chips
--vcpu-limit	FLOAT	No	Limit for the number of vCPUs
--memory-limit	FLOAT	No	Limit for the amount of memory in GiB
--accelerator-partition-type	TEXT	No	Type of accelerator partition (e.g., mig-1g.5gb, mig-2g.10gb, mig-3g.20gb, mig-4g.20gb, mig-7g.40gb)
--accelerator-partition-count	INTEGER	No	Number of accelerator partitions to request (minimum: 1)
--accelerator-partition-limit	INTEGER	No	Limit for the number of accelerator partitions (minimum: 1)
--preferred-topology	TEXT	No	Preferred topology annotation for scheduling
--required-topology	TEXT	No	Required topology annotation for scheduling
--max-node-count	INTEGER	No	Maximum number of nodes
--elastic-replica-increment-step	INTEGER	No	Scaling step size for elastic training. Provide either this or elastic-replica-discrete-values
--elastic-graceful-shutdown-timeout-in-seconds	INTEGER	No	Graceful shutdown timeout in seconds for elastic scaling operations
--elastic-scaling-timeout-in-seconds	INTEGER	No	Scaling timeout for elastic training
--elastic-scale-up-snooze-time-in-seconds	INTEGER	No	Timeout period after job restart during which no scale up/workload admission is allowed
--elastic-replica-discrete-values	ARRAY	No	Alternative to elastic-replica-increment-step. Provides exact values for total replicas count (array of integers)
--debug	FLAG	No	Enable debug mode (default: false)

List Available Accelerator Partition Types

This command lists the available accelerator partition types on the cluster for a specific instance type.

hyp list-accelerator-partition-type --instance-type <instance-type>

List Training Jobs

hyp list hyp-pytorch-job

Describe a Training Job

hyp describe hyp-pytorch-job --job-name <job-name>

Listing Pods

This command lists all the pods associated with a specific training job.

hyp list-pods hyp-pytorch-job --job-name <job-name>

• job-name (string) - Required. The name of the job to list pods for.

Accessing Logs

This command retrieves the logs for a specific pod within a training job.

hyp get-logs hyp-pytorch-job --pod-name <pod-name> --job-name <job-name>

Parameter	Required	Description
--job-name	Yes	The name of the job to get the log for.
--pod-name	Yes	The name of the pod to get the log from.
--namespace	No	The namespace of the job. Defaults to 'default'.
--container	No	The container name to get logs from.

Get Operator Logs

hyp get-operator-logs hyp-pytorch-job --since-hours 0.5

Delete a Training Job

hyp delete hyp-pytorch-job --job-name <job-name>

Recipe Job

Use hyp-recipe-job to submit fine-tuning and evaluation jobs using pre-built recipes from SageMaker JumpStart Hub — no YAML authoring required.

Initialize Recipe Job Configuration

mkdir my-recipe-job &&cd my-recipe-job
# Option A: HuggingFace model ID
hyp init hyp-recipe-job . \
--huggingface-model-id Qwen/Qwen3-0.6B \
--technique SFT \
--instance-type ml.g5.48xlarge
# Option B: JumpStart model ID
hyp init hyp-recipe-job . \
--model-id huggingface-reasoning-qwen3-06b \
--technique SFT \
--instance-type ml.g5.48xlarge

Supported job types:

• Fine-tuning: SFT, DPO, CPT, PPO, RLAIF, RLVR
• Evaluation: deterministic, LLMAJ

Note: If you omit --instance-type, the CLI will automatically query your HyperPod clusters and find clusters with instance types supported by the selected recipe and technique. You will be presented with a list of compatible clusters to choose from.

Configure Recipe Job Parameters

hyp configure \
--name my-recipe-job \
--namespace default \
--data-path /data/recipes-data/sft/train.jsonl \
--global-batch-size 8 \
--learning-rate 0.0001 \
--max-epochs 1 \
--output-path /data/output/my-model \
--instance-type ml.g5.48xlarge

Validate Configuration

hyp validate

Reset Configuration

To reset config.yaml back to its default values:

hyp reset

Submit Recipe Job

hyp create

List Recipe Jobs

hyp list hyp-recipe-job --namespace default

Describe a Recipe Job

hyp describe hyp-recipe-job --job-name <job-name> --namespace default

List Pods for a Recipe Job

hyp list-pods hyp-recipe-job --job-name <job-name> --namespace default

Get Logs from a Recipe Job Pod

hyp get-logs hyp-recipe-job --job-name <job-name> --pod-name <pod-name> --namespace default

Get Operator Logs

hyp get-operator-logs hyp-recipe-job

Delete a Recipe Job

hyp delete hyp-recipe-job --job-name <job-name> --namespace default

Inference

Jumpstart Endpoint Creation

Option 1: Create jumpstart endpoint through init experience

Initialize Jumpstart Endpoint Configuration

Initialize a new jumpstart endpoint configuration in the current directory:

hyp init hyp-jumpstart-endpoint

Configure Jumpstart Endpoint Parameters

Configure jumpstart endpoint parameters interactively or via command line:

hyp configure --endpoint-name my-jumpstart-endpoint

Validate Configuration

Validate the configuration file syntax:

hyp validate

Create Jumpstart Endpoint

Create the jumpstart endpoint using the configured parameters:

hyp create

Option 2: Create jumpstart endpoint through create command

Pre-trained Jumpstart models can be gotten from https://sagemaker.readthedocs.io/en/v2.82.0/doc_utils/jumpstart.html and fed into the call for creating the endpoint

hyp create hyp-jumpstart-endpoint \
--version 1.2 \
--model-id jumpstart-model-id\
--instance-type ml.g5.8xlarge \
--endpoint-name endpoint-jumpstart

Parameter	Type	Required	Description
--model-id	TEXT	Yes	JumpStart model identifier (1-63 characters, alphanumeric with hyphens)
--instance-type	TEXT	Yes	EC2 instance type for inference (must start with "ml.")
--namespace	TEXT	No	Kubernetes namespace
--metadata-name	TEXT	No	Name of the jumpstart endpoint object
--accept-eula	BOOLEAN	No	Whether model terms of use have been accepted (default: false)
--model-version	TEXT	No	Semantic version of the model (e.g., "1.0.0", 5-14 characters)
--endpoint-name	TEXT	No	Name of SageMaker endpoint (1-63 characters, alphanumeric with hyphens)
--tls-certificate-output-s3-uri	TEXT	No	S3 URI to write the TLS certificate
--debug	FLAG	No	Enable debug mode (default: false)
--version	TEXT	No	Schema version to use (default: "1.2")
--accelerator-partition-type	TEXT	No	MIG profile for GPU partitioning (must start with "mig-")
--accelerator-partition-validation	BOOLEAN	No	Enable MIG validation (default: true)
--replicas	INTEGER	No	Number of inference server replicas (default: 1)
--max-deploy-time-in-seconds	INTEGER	No	Maximum deployment time in seconds (default: 3600)
--execution-role	TEXT	No	IAM role ARN for deploying and managing the inference server
--env	JSON	No	Environment variables as JSON, e.g. '{"KEY":"value"}'
--metrics-enabled	BOOLEAN	No	Enable metrics collection
--metrics-scrape-interval-seconds	INTEGER	No	Scrape interval for metrics collection
--model-metrics-path	TEXT	No	Path where the model exposes metrics
--model-metrics-port	INTEGER	No	Port where the model exposes metrics
--additional-configs	JSON	No	Additional model configs as JSON key-value pairs
--gated-model-download-role	TEXT	No	IAM role ARN for downloading gated models
--model-hub-name	TEXT	No	Name of the model hub
--intelligent-routing-enabled	BOOLEAN	No	Enable intelligent routing
--routing-strategy	TEXT	No	Routing strategy: prefixaware, kvaware, session, or roundrobin
--enable-l1-cache	BOOLEAN	No	Enable L1 cache (CPU offloading)
--enable-l2-cache	BOOLEAN	No	Enable L2 cache
--l2-cache-backend	TEXT	No	L2 cache backend type
--l2-cache-local-url	TEXT	No	L2 cache URL to local storage
--cache-config-file	TEXT	No	KV cache configuration file path
--load-balancer-health-check-path	TEXT	No	Health check path for the ALB target group
--load-balancer-routing-algorithm	TEXT	No	Routing algorithm: least_outstanding_requests or round_robin
--custom-certificate-acm-arn	TEXT	No	ACM certificate ARN for custom TLS
--custom-certificate-domain-name	TEXT	No	Domain name for the custom TLS certificate
--auto-scaling-spec	JSON	No	Full autoScalingSpec JSON for autoscaling configuration
--dns-hosted-zone-id	TEXT	No	Route53 Hosted Zone ID for DNS automation
--data-capture	JSON	No	Data capture configuration JSON for SageMaker, LoadBalancer, and Model Pod tiers

Invoke a JumpstartModel Endpoint

hyp invoke hyp-jumpstart-endpoint \
--endpoint-name endpoint-jumpstart \
--body '{"inputs":"What is the capital of USA?"}'

Managing an Endpoint

hyp list hyp-jumpstart-endpoint
hyp describe hyp-jumpstart-endpoint --name endpoint-jumpstart

List Pods

hyp list-pods hyp-jumpstart-endpoint

Get Logs

hyp get-logs hyp-jumpstart-endpoint --pod-name <pod-name>

Get Operator Logs

hyp get-operator-logs hyp-jumpstart-endpoint --since-hours 0.5

Deleting an Endpoint

hyp delete hyp-jumpstart-endpoint --name endpoint-jumpstart

Custom Endpoint Creation

Option 1: Create custom endpoint through init experience

Initialize Custom Endpoint Configuration

Initialize a new custom endpoint configuration in the current directory:

hyp init hyp-custom-endpoint

Configure Custom Endpoint Parameters

Configure custom endpoint parameters interactively or via command line:

hyp configure --endpoint-name my-custom-endpoint

Validate Configuration

Validate the configuration file syntax:

hyp validate

Create Custom Endpoint

Create the custom endpoint using the configured parameters:

hyp create

Option 2: Create custom endpoint through create command

hyp create hyp-custom-endpoint \
--version 1.2 \
--endpoint-name endpoint-custom \
--model-name my-pytorch-model \
--model-source-type s3 \
--model-location my-pytorch-training \
--model-volume-mount-name test-volume \
--s3-bucket-name your-bucket \
--s3-region us-east-1 \
--instance-type ml.g5.8xlarge \
--image-uri 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:latest \
--container-port 8080

Parameter	Type	Required	Description
--model-name	TEXT	Yes	Name of model to create on SageMaker (1-63 characters, alphanumeric with hyphens)
--model-source-type	TEXT	Yes	Model source type: "s3", "fsx", "huggingface", or "kubernetesVolume"
--image-uri	TEXT	Yes	Docker image URI for inference
--container-port	INTEGER	Yes	Port on which model server listens (1-65535)
--model-volume-mount-name	TEXT	Yes	Name of the model volume mount
--namespace	TEXT	No	Kubernetes namespace
--metadata-name	TEXT	No	Name of the custom endpoint object
--endpoint-name	TEXT	No	Name of SageMaker endpoint (1-63 characters, alphanumeric with hyphens)
--version	TEXT	No	Schema version to use (default: "1.2")
--instance-type	TEXT	No	EC2 instance type (mutually exclusive with --instance-types)
--instance-types	TEXT	No	Comma-separated list of instance types in order of preference
--env	JSON	No	Environment variables as JSON, e.g. '{"KEY":"value"}'
--metrics-enabled	BOOLEAN	No	Enable metrics collection (default: false)
--metrics-scrape-interval-seconds	INTEGER	No	Scrape interval for metrics collection
--model-metrics-path	TEXT	No	Path where the model exposes metrics
--model-metrics-port	INTEGER	No	Port where the model exposes metrics
--model-version	TEXT	No	Version of the model (semantic version format)
--model-location	TEXT	No	Specific model data location
--prefetch-enabled	BOOLEAN	No	Whether to pre-fetch model data (default: false)
--tls-certificate-output-s3-uri	TEXT	No	S3 URI for TLS certificate output
--fsx-dns-name	TEXT	No	FSx File System DNS Name
--fsx-file-system-id	TEXT	No	FSx File System ID
--fsx-mount-name	TEXT	No	FSx File System Mount Name
--s3-bucket-name	TEXT	No	S3 bucket location
--s3-region	TEXT	No	S3 bucket region
--huggingface-model-id	TEXT	No	HuggingFace Hub model identifier (e.g. "meta-llama/Llama-3.1-8B-Instruct")
--huggingface-commit-sha	TEXT	No	Git commit SHA for the model revision (40-char hex)
--huggingface-token-secret-name	TEXT	No	Name of the K8s Secret containing the HuggingFace API token
--huggingface-token-secret-key	TEXT	No	Key in the K8s Secret for the HuggingFace API token
--model-volume-mount-path	TEXT	No	Path inside container for model volume (default: "/opt/ml/model")
--resources-limits	JSON	No	Resource limits, e.g. '{"nvidia.com/gpu":"1"}'
--resources-requests	JSON	No	Resource requests, e.g. '{"cpu":"1","memory":"2Gi"}'
--replicas	INTEGER	No	Number of inference server replicas (default: 1)
--initial-replica-count	INTEGER	No	Number of desired pods (defaults to 1)
--max-deploy-time-in-seconds	INTEGER	No	Maximum deployment time in seconds (default: 3600)
--worker-args	TEXT	No	Comma-separated arguments to the entrypoint
--worker-command	TEXT	No	Comma-separated entrypoint command array
--working-dir	TEXT	No	Working directory of the container
--invocation-endpoint	TEXT	No	Invocation endpoint path (default: "invocations")
--intelligent-routing-enabled	BOOLEAN	No	Enable intelligent routing
--routing-strategy	TEXT	No	Routing strategy: prefixaware, kvaware, session, or roundrobin
--enable-l1-cache	BOOLEAN	No	Enable L1 cache (CPU offloading)
--enable-l2-cache	BOOLEAN	No	Enable L2 cache
--l2-cache-backend	TEXT	No	L2 cache backend type
--l2-cache-local-url	TEXT	No	L2 cache URL to local storage
--cache-config-file	TEXT	No	KV cache configuration file path
--load-balancer-health-check-path	TEXT	No	Health check path for the ALB target group
--load-balancer-routing-algorithm	TEXT	No	Routing algorithm: least_outstanding_requests or round_robin
--max-concurrent-requests	INTEGER	No	Maximum concurrent requests per pod
--max-queue-size	INTEGER	No	Maximum request queue size
--overflow-status-code	INTEGER	No	HTTP status code when request limits exceeded (default: 429)
--custom-certificate-acm-arn	TEXT	No	ACM certificate ARN for custom TLS
--custom-certificate-domain-name	TEXT	No	Domain name for the custom TLS certificate
--kubernetes	JSON	No	Kubernetes customizations (initContainers, volumes, schedulerName, serviceAccountName)
--node-affinity	JSON	No	Node affinity JSON for advanced scheduling
--tags	JSON	No	Tags as JSON key-value pairs
--probes	JSON	No	Container probes JSON (livenessProbe, readinessProbe, startupProbe)
--auto-scaling-spec	JSON	No	Full autoScalingSpec JSON (overrides individual CloudWatch fields)
--dns-hosted-zone-id	TEXT	No	Route53 Hosted Zone ID for DNS automation
--data-capture	JSON	No	Data capture configuration JSON for SageMaker, LoadBalancer, and Model Pod tiers
--dimensions	JSON	No	CloudWatch Metric dimensions as key-value pairs
--metric-collection-period	INTEGER	No	Period for CloudWatch query (default: 300)
--metric-collection-start-time	INTEGER	No	StartTime for CloudWatch query (default: 300)
--metric-name	TEXT	No	Metric name to query for CloudWatch trigger
--metric-stat	TEXT	No	Statistics metric for CloudWatch (default: "Average")
--metric-type	TEXT	No	Type of metric for HPA ("Value" or "Average", default: "Average")
--min-value	NUMBER	No	Minimum metric value for empty CloudWatch response (default: 0)
--cloud-watch-trigger-name	TEXT	No	Name for the CloudWatch trigger
--cloud-watch-trigger-namespace	TEXT	No	AWS CloudWatch namespace for the metric
--target-value	NUMBER	No	Target value for the CloudWatch metric
--use-cached-metrics	BOOLEAN	No	Enable caching of metric values (default: true)
--debug	FLAG	No	Enable debug mode (default: false)

Invoke a Custom Inference Endpoint

hyp invoke hyp-custom-endpoint \
--endpoint-name endpoint-custom-pytorch \
--body '{"inputs":"What is the capital of USA?"}'

Managing an Endpoint

hyp list hyp-custom-endpoint
hyp describe hyp-custom-endpoint --name endpoint-custom

List Pods

hyp list-pods hyp-custom-endpoint

Get Logs

hyp get-logs hyp-custom-endpoint --pod-name <pod-name>

Get Operator Logs

hyp get-operator-logs hyp-custom-endpoint --since-hours 0.5

Deleting an Endpoint

hyp delete hyp-custom-endpoint --name endpoint-custom

Space

Create a Space

hyp create hyp-space \
--name myspace \
--namespace default \
--display-name "My Space"

Parameter	Type	Required	Description
--name	TEXT	Yes	Space name
--display-name	TEXT	Yes	Display Name of the space
--namespace	TEXT	No	Kubernetes namespace
--image	TEXT	No	Image specifies the container image to use
--desired-status	TEXT	No	DesiredStatus specifies the desired operational status
--ownership-type	TEXT	No	OwnershipType specifies who can modify the space. 'Public' means anyone with RBAC permissions can update/delete the space. 'OwnerOnly' means only the creator can update/delete the space.
--node-selector	TEXT	No	NodeSelector specifies node selection constraints for the space pod (JSON string)
--affinity	TEXT	No	Affinity specifies node affinity and anti-affinity rules for the space pod (JSON string)
--tolerations	TEXT	No	Tolerations specifies tolerations for the space pod to schedule on nodes with matching taints (JSON string)
--lifecycle	TEXT	No	Lifecycle specifies actions that the management system should take in response to container lifecycle events (JSON string)
--app-type	TEXT	No	AppType specifies the application type for this workspace
--service-account-name	TEXT	No	ServiceAccountName specifies the name of the ServiceAccount to use for the workspace pod
--queue-name	TEXT	No	Queue name for space scheduling (1-63 characters, alphanumeric with hyphens). Required when task governance is enabled on HyperPod EKS clusters.
--priority	TEXT	No	Priority class for space scheduling. Sets the kueue.x-k8s.io/priority-class label.
--access-type	TEXT	No	AccessType specifies who can connect to the workspace ('Public' or 'OwnerOnly')
--env	TEXT	No	Environment variables for the workspace container (JSON string, list of {name, value} objects)
--access-strategy	TEXT	No	References a WorkspaceAccessStrategy. Format: --access-strategy name=,namespace=
--pod-security-context	TEXT	No	Pod-level security context. Overrides template defaults when specified (JSON string)
--container-security-context	TEXT	No	Container-level security context for the main workspace container. Overrides template defaults (JSON string)
--init-containers	TEXT	No	Init containers to run before the workspace container starts (JSON string, max 10)
--idle-shutdown	TEXT	No	Idle shutdown configuration. Format: --idle-shutdown enabled=,idleTimeoutInMinutes=,detection=
--template-ref	TEXT	No	TemplateRef references a WorkspaceTemplate to use as base configuration. Format: --template-ref name=,namespace=
--container-config	TEXT	No	Container configuration. Format: --container-config command=,args=<arg1;arg2>
--storage	TEXT	No	Storage configuration. Format: --storage storageClassName=,size=,mountPath=
--volume	TEXT	No	Volume configuration. Format: --volume name=,mountPath=,persistentVolumeClaimName=<pvc_name>. Use multiple --volume flags for multiple volumes.
--accelerator-partition-count	TEXT	No	Fractional GPU partition count, e.g. '1'
--accelerator-partition-type	TEXT	No	Fractional GPU partition type, e.g. 'mig-3g.20gb'
--gpu-limit	TEXT	No	GPU resource limit, e.g. '1'
--gpu	TEXT	No	GPU resource request, e.g. '1'
--memory-limit	TEXT	No	Memory resource limit, e.g. '2Gi'
--memory	TEXT	No	Memory resource request, e.g. '2Gi'
--cpu-limit	TEXT	No	CPU resource limit, e.g. '500m'
--cpu	TEXT	No	CPU resource request, e.g. '500m'

List Spaces

# List spaces in default namespace
hyp list hyp-space
# List spaces in specific namespace
hyp list hyp-space --namespace my-namespace
# List spaces across all namespaces
hyp list hyp-space --all-namespaces
# List spaces with JSON output
hyp list hyp-space --output json

Describe a Space

hyp describe hyp-space --name myspace

Update a Space

hyp update hyp-space \
--name myspace \
--display-name "Updated Space Name"

Start/Stop a Space

hyp start hyp-space --name myspace
hyp stop hyp-space --name myspace

Get Logs

hyp get-logs hyp-space --name myspace

Delete a Space

hyp delete hyp-space --name myspace

Port Forward to a Space

Port forward to access a space from your local machine:

# Port forward with default port (8888)
hyp portforward hyp-space --name myspace
# Port forward with custom local port
hyp portforward hyp-space --name myspace --local-port 8080

Access the space via http://localhost:<local-port> after port forwarding is established. Press Ctrl+C to stop port forwarding.

Space Template Management

Create reusable space templates:

hyp create hyp-space-template --file template.yaml
hyp list hyp-space-template --all-namespaces
hyp describe hyp-space-template --name <template-name>
hyp update hyp-space-template --name <template-name> --file updated-template.yaml
hyp delete hyp-space-template --name <template-name>

Space Access

Create remote access to spaces. The --connection-type accepts web-ui or any {ide}-remote pattern (e.g. vscode-remote, kiro-remote, cursor-remote):

hyp create hyp-space-access --name myspace --connection-type vscode-remote
hyp create hyp-space-access --name myspace --connection-type kiro-remote
hyp create hyp-space-access --name myspace --connection-type cursor-remote
hyp create hyp-space-access --name myspace --connection-type web-ui

SDK

Along with the CLI, we also have SDKs available that can perform the cluster management, training and inference functionalities that the CLI performs

Cluster Management SDK

Creating a Cluster Stack

fromsagemaker.hyperpod.cluster_management.hp_cluster_stackimportHpClusterStack# Initialize cluster stack configurationcluster_stack=HpClusterStack(
stage="prod",
resource_name_prefix="my-hyperpod",
hyperpod_cluster_name="my-hyperpod-cluster",
eks_cluster_name="my-hyperpod-eks",
# Infrastructure componentscreate_vpc_stack=True,
create_eks_cluster_stack=True,
create_hyperpod_cluster_stack=True,
# Network configurationvpc_cidr="10.192.0.0/16",
availability_zone_ids=["use2-az1", "use2-az2"],
# Instance group configurationinstance_group_settings=[
{
"InstanceCount": 1,
"InstanceGroupName": "controller-group",
"InstanceType": "ml.t3.medium",
"TargetAvailabilityZoneId": "use2-az2"
}
]
)
# Create the cluster stackresponse=cluster_stack.create(region="us-east-2")

Listing Cluster Stacks

# List all cluster stacksstacks=HpClusterStack.list(region="us-east-2")
print(f"Found {len(stacks['StackSummaries'])} stacks")

Describing a Cluster Stack

# Describe a specific cluster stackstack_info=HpClusterStack.describe("my-stack-name", region="us-east-2")
print(f"Stack status: {stack_info['Stacks'][0]['StackStatus']}")

Monitoring Cluster Status

fromsagemaker.hyperpod.cluster_management.hp_cluster_stackimportHpClusterStackstack=HpClusterStack()
response=stack.create(region="us-west-2")
status=stack.get_status(region="us-west-2")
print(status)

Deleting a Cluster Stack

# Delete with custom loggerimportlogginglogger=logging.getLogger(__name__)
HpClusterStack.delete("my-stack-name", region="us-west-2", logger=logger)
# Delete with retained resources (only works on DELETE_FAILED stacks)HpClusterStack.delete("my-stack-name", retain_resources=["S3Bucket", "EFSFileSystem"])

Training SDK

Creating a Training Job

fromsagemaker.hyperpod.training.hyperpod_pytorch_jobimportHyperPodPytorchJobfromsagemaker.hyperpod.training.config.hyperpod_pytorch_job_unified_configimport (
ReplicaSpec, Template, Spec, Containers, Resources, RunPolicy
)
fromsagemaker.hyperpod.common.config.metadataimportMetadata# Define job specificationsnproc_per_node="1"# Number of processes per nodereplica_specs= [
ReplicaSpec
(
name="pod", # Replica nametemplate=Template
(
spec=Spec
(
containers=
[
Containers
(
# Container namename="container-name", # Training imageimage="123456789012.dkr.ecr.us-west-2.amazonaws.com/my-training-image:latest", # Always pull imageimage_pull_policy="Always", resources=Resources\
(
# No GPUs requestedrequests={"nvidia.com/gpu": "0"}, # No GPU limitlimits={"nvidia.com/gpu": "0"}, ),
# Command to runcommand=["python", "train.py"], # Script argumentsargs=["--epochs", "10", "--batch-size", "32"], )
]
)
),
)
]
# Keep pods after completionrun_policy=RunPolicy(clean_pod_policy="None") # Create and start the PyTorch jobpytorch_job=HyperPodPytorchJob
(
# Job namemetadata=Metadata(name="demo"), # Processes per nodenproc_per_node=nproc_per_node, # Replica specificationsreplica_specs=replica_specs, # Run policyrun_policy=run_policy, )
# Launch the jobpytorch_job.create() 

List Training Jobs

fromsagemaker.hyperpod.trainingimportHyperPodPytorchJobimportyaml# List all PyTorch jobsjobs=HyperPodPytorchJob.list()
print(yaml.dump(jobs))

Describe a Training Job

fromsagemaker.hyperpod.trainingimportHyperPodPytorchJob# Get an existing jobjob=HyperPodPytorchJob.get(name="my-pytorch-job")
print(job)

List Pods for a Training Job

fromsagemaker.hyperpod.trainingimportHyperPodPytorchJob# List Pods for an existing jobjob=HyperPodPytorchJob.get(name="my-pytorch-job")
print(job.list_pods())

Get Logs from a Pod

fromsagemaker.hyperpod.trainingimportHyperPodPytorchJob# Get pod logs for a jobjob=HyperPodPytorchJob.get(name="my-pytorch-job")
print(job.get_logs_from_pod("pod-name"))

Get Training Operator Logs

fromsagemaker.hyperpod.trainingimportHyperPodPytorchJob# Get training operator logsjob=HyperPodPytorchJob.get(name="my-pytorch-job")
print(job.get_operator_logs(since_hours=0.1))

Delete a Training Job

fromsagemaker.hyperpod.trainingimportHyperPodPytorchJob# Get an existing jobjob=HyperPodPytorchJob.get(name="my-pytorch-job")
# Delete the jobjob.delete()

Inference SDK

Creating a JumpstartModel Endpoint

Pre-trained Jumpstart models can be gotten from https://sagemaker.readthedocs.io/en/v2.82.0/doc_utils/jumpstart.html and fed into the call for creating the endpoint

fromsagemaker.hyperpod.inference.config.hp_jumpstart_endpoint_configimportModel, Server, SageMakerEndpoint, TlsConfigfromsagemaker.hyperpod.inference.hp_jumpstart_endpointimportHPJumpStartEndpointmodel=Model(
model_id='deepseek-llm-r1-distill-qwen-1-5b'
)
server=Server(
instance_type='ml.g5.8xlarge',
)
endpoint_name=SageMakerEndpoint(name='<my-endpoint-name>')
js_endpoint=HPJumpStartEndpoint(
model=model,
server=server,
sage_maker_endpoint=endpoint_name
)
js_endpoint.create()

Creating a Custom Inference Endpoint (with S3)

fromsagemaker.hyperpod.inference.config.hp_endpoint_configimportCloudWatchTrigger, Dimensions, AutoScalingSpec, Metrics, S3Storage, ModelSourceConfig, TlsConfig, EnvironmentVariables, ModelInvocationPort, ModelVolumeMount, Resources, Workerfromsagemaker.hyperpod.inference.hp_endpointimportHPEndpointmodel_source_config=ModelSourceConfig(
model_source_type='s3',
model_location="<my-model-folder-in-s3>",
s3_storage=S3Storage(
bucket_name='<my-model-artifacts-bucket>',
region='us-east-2',
),
)
environment_variables= [
EnvironmentVariables(name="HF_MODEL_ID", value="/opt/ml/model"),
EnvironmentVariables(name="SAGEMAKER_PROGRAM", value="inference.py"),
EnvironmentVariables(name="SAGEMAKER_SUBMIT_DIRECTORY", value="/opt/ml/model/code"),
EnvironmentVariables(name="MODEL_CACHE_ROOT", value="/opt/ml/model"),
EnvironmentVariables(name="SAGEMAKER_ENV", value="1"),
]
worker=Worker(
image='763104351884.dkr.ecr.us-east-2.amazonaws.com/huggingface-pytorch-tgi-inference:2.4.0-tgi2.3.1-gpu-py311-cu124-ubuntu22.04-v2.0',
model_volume_mount=ModelVolumeMount(
name='model-weights',
),
model_invocation_port=ModelInvocationPort(container_port=8080),
resources=Resources(
requests={"cpu": "30000m", "nvidia.com/gpu": 1, "memory": "100Gi"},
limits={"nvidia.com/gpu": 1}
),
environment_variables=environment_variables,
)
tls_config=TlsConfig(tls_certificate_output_s3_uri='s3://<my-tls-bucket-name>')
custom_endpoint=HPEndpoint(
endpoint_name='<my-endpoint-name>',
instance_type='ml.g5.8xlarge',
model_name='deepseek15b-test-model-name', tls_config=tls_config,
model_source_config=model_source_config,
worker=worker,
)
custom_endpoint.create()

List Endpoints

fromsagemaker.hyperpod.inference.hp_jumpstart_endpointimportHPJumpStartEndpointfromsagemaker.hyperpod.inference.hp_endpointimportHPEndpoint# List JumpStart endpointsjumpstart_endpoints=HPJumpStartEndpoint.list()
print(jumpstart_endpoints)
# List custom endpointscustom_endpoints=HPEndpoint.list()
print(custom_endpoints)

Describe an Endpoint

fromsagemaker.hyperpod.inference.hp_jumpstart_endpointimportHPJumpStartEndpointfromsagemaker.hyperpod.inference.hp_endpointimportHPEndpoint# Get JumpStart endpoint detailsjumpstart_endpoint=HPJumpStartEndpoint.get(name="js-endpoint-name", namespace="test")
print(jumpstart_endpoint)
# Get custom endpoint detailscustom_endpoint=HPEndpoint.get(name="endpoint-custom")
print(custom_endpoint)

Invoke an Endpoint

fromsagemaker.hyperpod.inference.hp_jumpstart_endpointimportHPJumpStartEndpointfromsagemaker.hyperpod.inference.hp_endpointimportHPEndpointdata='{"inputs":"What is the capital of USA?"}'jumpstart_endpoint=HPJumpStartEndpoint.get(name="endpoint-jumpstart")
response=jumpstart_endpoint.invoke(body=data).body.read()
print(response)
custom_endpoint=HPEndpoint.get(name="endpoint-custom")
response=custom_endpoint.invoke(body=data).body.read()
print(response)

List Pods

fromsagemaker.hyperpod.inference.hp_jumpstart_endpointimportHPJumpStartEndpointfromsagemaker.hyperpod.inference.hp_endpointimportHPEndpoint# List pods js_pods=HPJumpStartEndpoint.list_pods()
print(js_pods)
c_pods=HPEndpoint.list_pods()
print(c_pods)

Get Logs

fromsagemaker.hyperpod.inference.hp_jumpstart_endpointimportHPJumpStartEndpointfromsagemaker.hyperpod.inference.hp_endpointimportHPEndpoint# Get logs from pod js_logs=HPJumpStartEndpoint.get_logs(pod=<pod-name>)
print(js_logs)
c_logs=HPEndpoint.get_logs(pod=<pod-name>)
print(c_logs)

Get Operator Logs

fromsagemaker.hyperpod.inference.hp_jumpstart_endpointimportHPJumpStartEndpointfromsagemaker.hyperpod.inference.hp_endpointimportHPEndpoint# Invoke JumpStart endpointprint(HPJumpStartEndpoint.get_operator_logs(since_hours=0.1))
# Invoke custom endpointprint(HPEndpoint.get_operator_logs(since_hours=0.1))

Delete an Endpoint

fromsagemaker.hyperpod.inference.hp_jumpstart_endpointimportHPJumpStartEndpointfromsagemaker.hyperpod.inference.hp_endpointimportHPEndpoint# Delete JumpStart endpointjumpstart_endpoint=HPJumpStartEndpoint.get(name="endpoint-jumpstart")
jumpstart_endpoint.delete()
# Delete custom endpointcustom_endpoint=HPEndpoint.get(name="endpoint-custom")
custom_endpoint.delete()

Observability - Getting Monitoring Information

fromsagemaker.hyperpod.observability.utilsimportget_monitoring_configmonitor_config=get_monitoring_config()

Space SDK

Creating a Space

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpacefromhyperpod_space_template.v1_1.modelimportSpaceConfig# Create space configurationspace_config=SpaceConfig(
name="myspace",
namespace="default",
display_name="My Space",
)
# Create and start the spacespace=HPSpace(config=space_config)
space.create()

List Spaces

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# List all spaces in default namespacespaces=HPSpace.list()
forspaceinspaces:
print(f"Space: {space.config.name}, Status: {space.status}")
# List spaces in specific namespacespaces=HPSpace.list(namespace="your-namespace")

Get a Space

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# Get specific spacespace=HPSpace.get(name="myspace", namespace="default")
print(f"Space name: {space.config.name}")
print(f"Display name: {space.config.display_name}")

Update a Space

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# Get existing spacespace=HPSpace.get(name="myspace")
# Update space configurationspace.update(
display_name="Updated Space Name",
)

Start/Stop a Space

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# Get existing spacespace=HPSpace.get(name="myspace")
# Start the spacespace.start()
# Stop the spacespace.stop()

Get Space Logs

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# Get space and retrieve logsspace=HPSpace.get(name="myspace")
# Get logs from default pod and containerlogs=space.get_logs()
print(logs)

List Space Pods

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# Get space and list associated podsspace=HPSpace.get(name="myspace")
pods=space.list_pods()
forpodinpods:
print(f"Pod: {pod}")

Create Space Access

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# Get existing spacespace=HPSpace.get(name="myspace")
# Create VS Code remote accessvscode_access=space.create_space_access(connection_type="vscode-remote")
print(f"VS Code URL: {vscode_access['SpaceConnectionUrl']}")
# Create Kiro remote accesskiro_access=space.create_space_access(connection_type="kiro-remote")
print(f"Kiro URL: {kiro_access['SpaceConnectionUrl']}")
# Create web UI accessweb_access=space.create_space_access(connection_type="web-ui")
print(f"Web UI URL: {web_access['SpaceConnectionUrl']}")

Delete a Space

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# Get existing spacespace=HPSpace.get(name="myspace")
# Delete the spacespace.delete()

Port Forward to a Space

fromsagemaker.hyperpod.space.hyperpod_spaceimportHPSpace# Get existing spacespace=HPSpace.get(name="myspace")
# Port forward with default remote port (8888)space.portforward_space(local_port="8080")
# Port forward with custom remote portspace.portforward_space(local_port="8080", remote_port="8888")

Access the space via http://localhost:<local-port> after port forwarding is established. Press Ctrl+C to stop port forwarding.

Space Template Management

fromsagemaker.hyperpod.space.hyperpod_space_templateimportHPSpaceTemplate# Create space template from YAML filetemplate=HPSpaceTemplate(file_path="template.yaml")
template.create()
# List all space templatestemplates=HPSpaceTemplate.list()
fortemplateintemplates:
print(f"Template: {template.name}")
# Get specific space templatetemplate=HPSpaceTemplate.get(name="my-template")
print(template.to_yaml())
# Update space templatetemplate.update(file_path="updated-template.yaml")
# Delete space templatetemplate.delete()

Examples

This repository provides both a full end-to-end example walkthrough of using the CLI for real-world training and inference workloads as well as standalone example notebooks for individual features.

End-to-End Walkthrough

End-to-End Walkthrough Example

Standalone Examples

Cluster Management Example Notebooks

CLI Cluster Management Example

SDK Cluster Management Example

Training Example Notebooks

CLI Training Init Experience Example

CLI Training Example

SDK Training Example

Inference Example Notebooks

CLI

CLI Inference Jumpstart Model Init Experience Example

CLI Inference JumpStart Model Example

CLI Inference FSX Model Example

CLI Inference S3 Model Init Experience Example

CLI Inference S3 Model Example

SDK

SDK Inference JumpStart Model Example

SDK Inference FSX Model Example

SDK Inference S3 Model Example

Disclaimer

• This CLI and SDK requires access to the user's file system to set and get context and function properly. It needs to read configuration files such as kubeconfig to establish the necessary environment settings.

Working behind a proxy server ?

• Follow these steps from here to set up HTTP proxy connections