fix: change AMI ids in tests to be dynamic based on regions#1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chuyang-deng merged 10 commits into from
Aug 28, 2019
Merged

fix: change AMI ids in tests to be dynamic based on regions #1004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a877ca9
update: change directory_path for EFS usage example in overview doc
caxiaohu Aug 23, 2019
2ca2c9e
update: changed EFS directory path instructions in doc and docstring …
caxiaohu Aug 23, 2019
a4ec855
update: changed AMI ids to be dynamic based on regions
caxiaohu Aug 27, 2019
1bdf0ad
fix: change Amazon AMI ids to be dynamic based on regions
caxiaohu Aug 27, 2019
c43ff38
fix: change Amazon AMI ids to be dynamic based on regions using searc…
caxiaohu Aug 27, 2019
f41d3c5
update: move region closer to its usage
caxiaohu Aug 27, 2019
0393384
update: make filter as constant variable
caxiaohu Aug 27, 2019
6b6e070
update: remove else
caxiaohu Aug 27, 2019
b80ccbc
update: add constant variable for search AMI name
caxiaohu Aug 27, 2019
6a95b89
Merge branch 'master' into fix_region_issue
chuyang-deng Aug 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions tests/integ/__init__.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -71,8 +71,6 @@
NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]
NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]

EFS_TEST_ENABLED_REGION = ["us-west-2"]

logging.getLogger("boto3").setLevel(logging.INFO)
logging.getLogger("botocore").setLevel(logging.INFO)

Expand Down
37 changes: 28 additions & 9 deletions tests/integ/file_system_input_utils.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,7 @@

import collections
import logging
from operator import itemgetter
import os
from os import path
import stat
Expand All@@ -27,13 +28,12 @@
from tests.integ.vpc_test_utils import check_or_create_vpc_resources_efs_fsx

VPC_NAME = "sagemaker-efs-fsx-vpc"
ALINUX_AMI_NAME_FILTER = "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
EFS_CREATION_TOKEN = str(uuid.uuid4())
PREFIX = "ec2_fs_key_"
KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
ROLE_NAME = "SageMakerRole"
REGION = "us-west-2"
EC2_INSTANCE_TYPE = "t2.micro"
AMI_ID = "ami-082b5a644766e0e6f"
MIN_COUNT = 1
MAX_COUNT = 1

Expand DownExpand Up@@ -69,12 +69,13 @@ def set_up_efs_fsx(sagemaker_session):
_check_or_create_key_pair(sagemaker_session)
_check_or_create_iam_profile_and_attach_role(sagemaker_session)
subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
sagemaker_session, REGION, VPC_NAME
sagemaker_session, VPC_NAME
)

ami_id = _ami_id_for_region(sagemaker_session)
ec2_instance = _create_ec2_instance(
sagemaker_session,
AMI_ID,
ami_id,
EC2_INSTANCE_TYPE,
KEY_NAME,
MIN_COUNT,
Expand All@@ -100,16 +101,34 @@ def set_up_efs_fsx(sagemaker_session):
mount_efs_target_id,
)

region = sagemaker_session.boto_region_name
try:
connected_instance = _connect_ec2_instance(ec2_instance)
_upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id)
_upload_data_and_mount_fs(
connected_instance, file_system_efs_id, file_system_fsx_id, region
)
except Exception:
tear_down(sagemaker_session, fs_resources)
raise

return fs_resources


def _ami_id_for_region(sagemaker_session):
ec2_client = sagemaker_session.boto_session.client("ec2")
filters = [
{"Name": "name", "Values": [ALINUX_AMI_NAME_FILTER]},
{"Name": "state", "Values": ["available"]},
]
response = ec2_client.describe_images(Filters=filters)
image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)

if len(image_details) == 0:
raise Exception("AMI was not found based on current search criteria: {}".format(filters))

return image_details[0]["ImageId"]


def _connect_ec2_instance(ec2_instance):
public_ip_address = ec2_instance.public_ip_address
connected_instance = Connection(
Expand All@@ -118,7 +137,7 @@ def _connect_ec2_instance(ec2_instance):
return connected_instance


def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id):
def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id, region):
connected_instance.put(FS_MOUNT_SCRIPT, ".")
connected_instance.run("mkdir temp_tf; mkdir temp_one_p", in_stream=False)
for dir_name, subdir_list, file_list in os.walk(MNIST_LOCAL_DATA):
Expand All@@ -127,7 +146,7 @@ def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_syste
connected_instance.put(local_file, "temp_tf/")
connected_instance.put(ONE_P_LOCAL_DATA, "temp_one_p/")
connected_instance.run(
"sudo sh fs_mount_setup.sh {} {}".format(file_system_efs_id, file_system_fsx_id),
"sudo sh fs_mount_setup.sh {} {} {}".format(file_system_efs_id, file_system_fsx_id, region),
in_stream=False,
)

Expand DownExpand Up@@ -168,7 +187,7 @@ def _check_or_create_efs(sagemaker_session):

def _create_efs_mount(sagemaker_session, file_system_id):
subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
sagemaker_session, REGION, VPC_NAME
sagemaker_session, VPC_NAME
)
efs_client = sagemaker_session.boto_session.client("efs")
mount_response = efs_client.create_mount_target(
Expand All@@ -188,7 +207,7 @@ def _create_efs_mount(sagemaker_session, file_system_id):
def _check_or_create_fsx(sagemaker_session):
fsx_client = sagemaker_session.boto_session.client("fsx")
subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
sagemaker_session, REGION, VPC_NAME
sagemaker_session, VPC_NAME
)
create_response = fsx_client.create_file_system(
FileSystemType="LUSTRE",
Expand Down
34 changes: 8 additions & 26 deletions tests/integ/test_kmeans_efs_fsx.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,6 @@

import pytest

import tests.integ
from sagemaker import KMeans
from sagemaker.amazon.amazon_estimator import FileSystemRecordSet
from sagemaker.parameter import IntegerParameter, CategoricalParameter
Expand All@@ -25,7 +24,6 @@
from tests.integ.s3_utils import assert_s3_files_exist
from tests.integ.timeout import timeout

TRAIN_INSTANCE_TYPE = "ml.c4.xlarge"
TRAIN_INSTANCE_COUNT = 1
OBJECTIVE_METRIC_NAME = "test:msd"
EFS_DIR_PATH = "/one_p_mnist"
Expand All@@ -46,19 +44,15 @@ def efs_fsx_setup(sagemaker_session):
tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
subnets = [efs_fsx_setup.subnet_id]
security_group_ids = efs_fsx_setup.security_group_ids
role = efs_fsx_setup.role_name
kmeans = KMeans(
role=role,
train_instance_count=TRAIN_INSTANCE_COUNT,
train_instance_type=TRAIN_INSTANCE_TYPE,
train_instance_type=cpu_instance_type,
k=K,
sagemaker_session=sagemaker_session,
subnets=subnets,
Expand All@@ -80,19 +74,15 @@ def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
subnets = [efs_fsx_setup.subnet_id]
security_group_ids = efs_fsx_setup.security_group_ids
role = efs_fsx_setup.role_name
kmeans = KMeans(
role=role,
train_instance_count=TRAIN_INSTANCE_COUNT,
train_instance_type=TRAIN_INSTANCE_TYPE,
train_instance_type=cpu_instance_type,
k=K,
sagemaker_session=sagemaker_session,
subnets=subnets,
Expand All@@ -114,18 +104,14 @@ def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
subnets = [efs_fsx_setup.subnet_id]
security_group_ids = efs_fsx_setup.security_group_ids
role = efs_fsx_setup.role_name
kmeans = KMeans(
role=role,
train_instance_count=TRAIN_INSTANCE_COUNT,
train_instance_type=TRAIN_INSTANCE_TYPE,
train_instance_type=cpu_instance_type,
k=K,
sagemaker_session=sagemaker_session,
subnets=subnets,
Expand DownExpand Up@@ -174,18 +160,14 @@ def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
assert best_training_job


@pytest.mark.skipif(
tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
subnets = [efs_fsx_setup.subnet_id]
security_group_ids = efs_fsx_setup.security_group_ids
role = efs_fsx_setup.role_name
kmeans = KMeans(
role=role,
train_instance_count=TRAIN_INSTANCE_COUNT,
train_instance_type=TRAIN_INSTANCE_TYPE,
train_instance_type=cpu_instance_type,
k=K,
sagemaker_session=sagemaker_session,
subnets=subnets,
Expand Down
34 changes: 8 additions & 26 deletions tests/integ/test_tf_efs_fsx.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,6 @@

import pytest

import tests.integ
from sagemaker.inputs import FileSystemInput
from sagemaker.parameter import IntegerParameter
from sagemaker.tensorflow import TensorFlow
Expand All@@ -32,7 +31,6 @@
MNIST_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tensorflow_mnist")
SCRIPT = os.path.join(MNIST_RESOURCE_PATH, "mnist.py")
TFS_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tfs", "tfs-test-entrypoint-with-handler")
INSTANCE_TYPE = "ml.c4.xlarge"
EFS_DIR_PATH = "/tensorflow"
FSX_DIR_PATH = "/fsx/tensorflow"
MAX_JOBS = 2
Expand All@@ -49,11 +47,7 @@ def efs_fsx_setup(sagemaker_session):
tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_efs(efs_fsx_setup, sagemaker_session):
def test_mnist_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
role = efs_fsx_setup.role_name
subnets = [efs_fsx_setup.subnet_id]
security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -62,7 +56,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
entry_point=SCRIPT,
role=role,
train_instance_count=1,
train_instance_type=INSTANCE_TYPE,
train_instance_type=cpu_instance_type,
sagemaker_session=sagemaker_session,
script_mode=True,
framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -85,11 +79,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
)


@pytest.mark.skipif(
tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
def test_mnist_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
role = efs_fsx_setup.role_name
subnets = [efs_fsx_setup.subnet_id]
security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -98,7 +88,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
entry_point=SCRIPT,
role=role,
train_instance_count=1,
train_instance_type=INSTANCE_TYPE,
train_instance_type=cpu_instance_type,
sagemaker_session=sagemaker_session,
script_mode=True,
framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -121,11 +111,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
)


@pytest.mark.skipif(
tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
role = efs_fsx_setup.role_name
subnets = [efs_fsx_setup.subnet_id]
security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -134,7 +120,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
entry_point=SCRIPT,
role=role,
train_instance_count=1,
train_instance_type=INSTANCE_TYPE,
train_instance_type=cpu_instance_type,
script_mode=True,
sagemaker_session=sagemaker_session,
py_version=PY_VERSION,
Expand DownExpand Up@@ -169,11 +155,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
assert best_training_job


@pytest.mark.skipif(
tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
role = efs_fsx_setup.role_name
subnets = [efs_fsx_setup.subnet_id]
security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -182,7 +164,7 @@ def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
entry_point=SCRIPT,
role=role,
train_instance_count=1,
train_instance_type=INSTANCE_TYPE,
train_instance_type=cpu_instance_type,
script_mode=True,
sagemaker_session=sagemaker_session,
py_version=PY_VERSION,
Expand Down
12 changes: 5 additions & 7 deletions tests/integ/vpc_test_utils.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -62,7 +62,7 @@ def _route_table_id(ec2_client, vpc_id):
return desc["RouteTables"][0]["RouteTableId"]


def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NAME):
def check_or_create_vpc_resources_efs_fsx(sagemaker_session, name=VPC_NAME):
# use lock to prevent race condition when tests are running concurrently
with lock.lock(LOCK_PATH):
ec2_client = sagemaker_session.boto_session.client("ec2")
Expand All@@ -74,13 +74,11 @@ def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NA
_security_group_ids_by_vpc_id(sagemaker_session, vpc_id),
)
else:
return _create_vpc_with_name_efs_fsx(ec2_client, region, name)
return _create_vpc_with_name_efs_fsx(ec2_client, name)


def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(
ec2_client, region, name
)
def _create_vpc_with_name_efs_fsx(ec2_client, name):
vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(ec2_client, name)
ec2_client.modify_vpc_attribute(EnableDnsHostnames={"Value": True}, VpcId=vpc_id)

ig = ec2_client.create_internet_gateway()
Expand DownExpand Up@@ -121,7 +119,7 @@ def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
return [subnet_id_a], [security_group_id]


def _create_vpc_resources(ec2_client, region, name):
def _create_vpc_resources(ec2_client, name):
vpc_id = ec2_client.create_vpc(CidrBlock="10.0.0.0/16")["Vpc"]["VpcId"]
print("created vpc: {}".format(vpc_id))

Expand Down
7 changes: 4 additions & 3 deletions tests/scripts/fs_mount_setup.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -16,18 +16,19 @@
# Mounting EFS and FSx for Lustre file systems for integration Tests
FILE_SYSTEM_EFS_ID=$1
FILE_SYSTEM_FSX_ID=$2
REGION=$3

echo "Mounting EFS File Systems"
sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
sudo yum install -y amazon-efs-utils

@laurenyulaurenyuAug 27, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do different regions have different versions released?

@caxiaohucaxiaohuAug 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.
The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.
But it will automatically select the correct version if i don't specify

sudo mkdir efs
sudo mount -t efs "$FILE_SYSTEM_EFS_ID":/ efs
sudo mkdir efs/tensorflow
sudo mkdir efs/one_p_mnist

echo "Mounting FSx for Lustre File System"
sudo amazon-linux-extras install -y lustre2.10
sudo yum install -y lustre-client
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx.us-west-2.amazonaws.com@tcp:/fsx /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx."$REGION".amazonaws.com@tcp:/fsx /mnt/fsx
sudo mkdir /mnt/fsx/tensorflow
sudo mkdir /mnt/fsx/one_p_mnist

Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
fix: change AMI ids in tests to be dynamic based on regions#1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

chuyang-deng
merged 10 commits into

from

Aug 28, 2019
 Merged


fix: change AMI ids in tests to be dynamic based on regions
#1004
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


10 commits


Select commit
Hold shift + click to select a range
a877ca9

update: change directory_path for EFS usage example in overview doc


caxiaohu Aug 23, 2019
2ca2c9e

update: changed EFS directory path instructions in doc and docstring …


caxiaohu Aug 23, 2019
a4ec855

update: changed AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
1bdf0ad

fix: change Amazon AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
c43ff38

fix: change Amazon AMI ids to be dynamic based on regions using searc…


caxiaohu Aug 27, 2019
f41d3c5

update: move region closer to its usage


caxiaohu Aug 27, 2019
0393384

update: make filter as constant variable


caxiaohu Aug 27, 2019
6b6e070

update: remove else


caxiaohu Aug 27, 2019
b80ccbc

update: add constant variable for search AMI name


caxiaohu Aug 27, 2019
6a95b89

Merge branch 'master' into fix_region_issue


chuyang-deng Aug 28, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 0 additions & 2 deletions
tests/integ/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -71,8 +71,6 @@
NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]
NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]

EFS_TEST_ENABLED_REGION = ["us-west-2"]

logging.getLogger("boto3").setLevel(logging.INFO)
logging.getLogger("botocore").setLevel(logging.INFO)

Expand Down

37 changes: 28 additions & 9 deletions
tests/integ/file_system_input_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,7 @@

import collections
import logging
from operator import itemgetter
import os
from os import path
import stat
Expand All@@ -27,13 +28,12 @@
from tests.integ.vpc_test_utils import check_or_create_vpc_resources_efs_fsx

VPC_NAME = "sagemaker-efs-fsx-vpc"
ALINUX_AMI_NAME_FILTER = "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
EFS_CREATION_TOKEN = str(uuid.uuid4())
PREFIX = "ec2_fs_key_"
KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
ROLE_NAME = "SageMakerRole"
REGION = "us-west-2"
EC2_INSTANCE_TYPE = "t2.micro"
AMI_ID = "ami-082b5a644766e0e6f"
MIN_COUNT = 1
MAX_COUNT = 1

Expand DownExpand Up@@ -69,12 +69,13 @@ def set_up_efs_fsx(sagemaker_session):
 _check_or_create_key_pair(sagemaker_session)
 _check_or_create_iam_profile_and_attach_role(sagemaker_session)
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )

 ami_id = _ami_id_for_region(sagemaker_session)
 ec2_instance = _create_ec2_instance(
 sagemaker_session,
AMI_ID,
ami_id,
 EC2_INSTANCE_TYPE,
 KEY_NAME,
 MIN_COUNT,
Expand All@@ -100,16 +101,34 @@ def set_up_efs_fsx(sagemaker_session):
 mount_efs_target_id,
 )

 region = sagemaker_session.boto_region_name
 try:
 connected_instance = _connect_ec2_instance(ec2_instance)
 _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id)
 _upload_data_and_mount_fs(
 connected_instance, file_system_efs_id, file_system_fsx_id, region
 )
 except Exception:
 tear_down(sagemaker_session, fs_resources)
 raise

 return fs_resources


def _ami_id_for_region(sagemaker_session):
 ec2_client = sagemaker_session.boto_session.client("ec2")
 filters = [
 {"Name": "name", "Values": [ALINUX_AMI_NAME_FILTER]},
 {"Name": "state", "Values": ["available"]},
 ]
 response = ec2_client.describe_images(Filters=filters)
 image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)

 if len(image_details) == 0:
 raise Exception("AMI was not found based on current search criteria: {}".format(filters))

 return image_details[0]["ImageId"]


def _connect_ec2_instance(ec2_instance):
 public_ip_address = ec2_instance.public_ip_address
 connected_instance = Connection(
Expand All@@ -118,7 +137,7 @@ def _connect_ec2_instance(ec2_instance):
 return connected_instance


def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id):
def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id, region):
 connected_instance.put(FS_MOUNT_SCRIPT, ".")
 connected_instance.run("mkdir temp_tf; mkdir temp_one_p", in_stream=False)
 for dir_name, subdir_list, file_list in os.walk(MNIST_LOCAL_DATA):
Expand All@@ -127,7 +146,7 @@ def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_syste
 connected_instance.put(local_file, "temp_tf/")
 connected_instance.put(ONE_P_LOCAL_DATA, "temp_one_p/")
 connected_instance.run(
 "sudo sh fs_mount_setup.sh {} {}".format(file_system_efs_id, file_system_fsx_id),
 "sudo sh fs_mount_setup.sh {} {} {}".format(file_system_efs_id, file_system_fsx_id, region),
 in_stream=False,
 )

Expand DownExpand Up@@ -168,7 +187,7 @@ def _check_or_create_efs(sagemaker_session):

def _create_efs_mount(sagemaker_session, file_system_id):
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 efs_client = sagemaker_session.boto_session.client("efs")
 mount_response = efs_client.create_mount_target(
Expand All@@ -188,7 +207,7 @@ def _create_efs_mount(sagemaker_session, file_system_id):
def _check_or_create_fsx(sagemaker_session):
 fsx_client = sagemaker_session.boto_session.client("fsx")
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 create_response = fsx_client.create_file_system(
 FileSystemType="LUSTRE",
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_kmeans_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,6 @@

import pytest

import tests.integ
from sagemaker import KMeans
from sagemaker.amazon.amazon_estimator import FileSystemRecordSet
from sagemaker.parameter import IntegerParameter, CategoricalParameter
Expand All@@ -25,7 +24,6 @@
from tests.integ.s3_utils import assert_s3_files_exist
from tests.integ.timeout import timeout

TRAIN_INSTANCE_TYPE = "ml.c4.xlarge"
TRAIN_INSTANCE_COUNT = 1
OBJECTIVE_METRIC_NAME = "test:msd"
EFS_DIR_PATH = "/one_p_mnist"
Expand All@@ -46,19 +44,15 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -80,19 +74,15 @@ def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -114,18 +104,14 @@ def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand DownExpand Up@@ -174,18 +160,14 @@ def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_tf_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,6 @@

import pytest

import tests.integ
from sagemaker.inputs import FileSystemInput
from sagemaker.parameter import IntegerParameter
from sagemaker.tensorflow import TensorFlow
Expand All@@ -32,7 +31,6 @@
MNIST_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tensorflow_mnist")
SCRIPT = os.path.join(MNIST_RESOURCE_PATH, "mnist.py")
TFS_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tfs", "tfs-test-entrypoint-with-handler")
INSTANCE_TYPE = "ml.c4.xlarge"
EFS_DIR_PATH = "/tensorflow"
FSX_DIR_PATH = "/fsx/tensorflow"
MAX_JOBS = 2
Expand All@@ -49,11 +47,7 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_efs(efs_fsx_setup, sagemaker_session):
def test_mnist_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -62,7 +56,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -85,11 +79,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
def test_mnist_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -98,7 +88,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -121,11 +111,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -134,7 +120,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand DownExpand Up@@ -169,11 +155,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -182,7 +164,7 @@ def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand Down

12 changes: 5 additions & 7 deletions
tests/integ/vpc_test_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -62,7 +62,7 @@ def _route_table_id(ec2_client, vpc_id):
 return desc["RouteTables"][0]["RouteTableId"]


def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NAME):
def check_or_create_vpc_resources_efs_fsx(sagemaker_session, name=VPC_NAME):
 # use lock to prevent race condition when tests are running concurrently
 with lock.lock(LOCK_PATH):
 ec2_client = sagemaker_session.boto_session.client("ec2")
Expand All@@ -74,13 +74,11 @@ def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NA
 _security_group_ids_by_vpc_id(sagemaker_session, vpc_id),
 )
 else:
 return _create_vpc_with_name_efs_fsx(ec2_client, region, name)
 return _create_vpc_with_name_efs_fsx(ec2_client, name)


def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(
 ec2_client, region, name
 )
def _create_vpc_with_name_efs_fsx(ec2_client, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(ec2_client, name)
 ec2_client.modify_vpc_attribute(EnableDnsHostnames={"Value": True}, VpcId=vpc_id)

 ig = ec2_client.create_internet_gateway()
Expand DownExpand Up@@ -121,7 +119,7 @@ def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 return [subnet_id_a], [security_group_id]


def _create_vpc_resources(ec2_client, region, name):
def _create_vpc_resources(ec2_client, name):
 vpc_id = ec2_client.create_vpc(CidrBlock="10.0.0.0/16")["Vpc"]["VpcId"]
 print("created vpc: {}".format(vpc_id))

Expand Down

7 changes: 4 additions & 3 deletions
tests/scripts/fs_mount_setup.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,18 +16,19 @@
# Mounting EFS and FSx for Lustre file systems for integration Tests
FILE_SYSTEM_EFS_ID=$1
FILE_SYSTEM_FSX_ID=$2
REGION=$3

echo "Mounting EFS File Systems"
sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
sudo yum install -y amazon-efs-utils
@laurenyulaurenyuAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do different regions have different versions released?
@caxiaohucaxiaohuAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.

The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.

But it will automatically select the correct version if i don't specify
sudo mkdir efs
sudo mount -t efs "$FILE_SYSTEM_EFS_ID":/ efs
sudo mkdir efs/tensorflow
sudo mkdir efs/one_p_mnist

echo "Mounting FSx for Lustre File System"
sudo amazon-linux-extras install -y lustre2.10
sudo yum install -y lustre-client
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx.us-west-2.amazonaws.com@tcp:/fsx /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx."$REGION".amazonaws.com@tcp:/fsx /mnt/fsx
sudo mkdir /mnt/fsx/tensorflow
sudo mkdir /mnt/fsx/one_p_mnist

Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix: change AMI ids in tests to be dynamic based on regions#1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

chuyang-deng
merged 10 commits into

from

Aug 28, 2019
 Merged


fix: change AMI ids in tests to be dynamic based on regions
#1004
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


10 commits


Select commit
Hold shift + click to select a range
a877ca9

update: change directory_path for EFS usage example in overview doc


caxiaohu Aug 23, 2019
2ca2c9e

update: changed EFS directory path instructions in doc and docstring …


caxiaohu Aug 23, 2019
a4ec855

update: changed AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
1bdf0ad

fix: change Amazon AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
c43ff38

fix: change Amazon AMI ids to be dynamic based on regions using searc…


caxiaohu Aug 27, 2019
f41d3c5

update: move region closer to its usage


caxiaohu Aug 27, 2019
0393384

update: make filter as constant variable


caxiaohu Aug 27, 2019
6b6e070

update: remove else


caxiaohu Aug 27, 2019
b80ccbc

update: add constant variable for search AMI name


caxiaohu Aug 27, 2019
6a95b89

Merge branch 'master' into fix_region_issue


chuyang-deng Aug 28, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 0 additions & 2 deletions
tests/integ/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -71,8 +71,6 @@
NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]
NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]

EFS_TEST_ENABLED_REGION = ["us-west-2"]

logging.getLogger("boto3").setLevel(logging.INFO)
logging.getLogger("botocore").setLevel(logging.INFO)

Expand Down

37 changes: 28 additions & 9 deletions
tests/integ/file_system_input_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,7 @@

import collections
import logging
from operator import itemgetter
import os
from os import path
import stat
Expand All@@ -27,13 +28,12 @@
from tests.integ.vpc_test_utils import check_or_create_vpc_resources_efs_fsx

VPC_NAME = "sagemaker-efs-fsx-vpc"
ALINUX_AMI_NAME_FILTER = "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
EFS_CREATION_TOKEN = str(uuid.uuid4())
PREFIX = "ec2_fs_key_"
KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
ROLE_NAME = "SageMakerRole"
REGION = "us-west-2"
EC2_INSTANCE_TYPE = "t2.micro"
AMI_ID = "ami-082b5a644766e0e6f"
MIN_COUNT = 1
MAX_COUNT = 1

Expand DownExpand Up@@ -69,12 +69,13 @@ def set_up_efs_fsx(sagemaker_session):
 _check_or_create_key_pair(sagemaker_session)
 _check_or_create_iam_profile_and_attach_role(sagemaker_session)
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )

 ami_id = _ami_id_for_region(sagemaker_session)
 ec2_instance = _create_ec2_instance(
 sagemaker_session,
AMI_ID,
ami_id,
 EC2_INSTANCE_TYPE,
 KEY_NAME,
 MIN_COUNT,
Expand All@@ -100,16 +101,34 @@ def set_up_efs_fsx(sagemaker_session):
 mount_efs_target_id,
 )

 region = sagemaker_session.boto_region_name
 try:
 connected_instance = _connect_ec2_instance(ec2_instance)
 _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id)
 _upload_data_and_mount_fs(
 connected_instance, file_system_efs_id, file_system_fsx_id, region
 )
 except Exception:
 tear_down(sagemaker_session, fs_resources)
 raise

 return fs_resources


def _ami_id_for_region(sagemaker_session):
 ec2_client = sagemaker_session.boto_session.client("ec2")
 filters = [
 {"Name": "name", "Values": [ALINUX_AMI_NAME_FILTER]},
 {"Name": "state", "Values": ["available"]},
 ]
 response = ec2_client.describe_images(Filters=filters)
 image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)

 if len(image_details) == 0:
 raise Exception("AMI was not found based on current search criteria: {}".format(filters))

 return image_details[0]["ImageId"]


def _connect_ec2_instance(ec2_instance):
 public_ip_address = ec2_instance.public_ip_address
 connected_instance = Connection(
Expand All@@ -118,7 +137,7 @@ def _connect_ec2_instance(ec2_instance):
 return connected_instance


def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id):
def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id, region):
 connected_instance.put(FS_MOUNT_SCRIPT, ".")
 connected_instance.run("mkdir temp_tf; mkdir temp_one_p", in_stream=False)
 for dir_name, subdir_list, file_list in os.walk(MNIST_LOCAL_DATA):
Expand All@@ -127,7 +146,7 @@ def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_syste
 connected_instance.put(local_file, "temp_tf/")
 connected_instance.put(ONE_P_LOCAL_DATA, "temp_one_p/")
 connected_instance.run(
 "sudo sh fs_mount_setup.sh {} {}".format(file_system_efs_id, file_system_fsx_id),
 "sudo sh fs_mount_setup.sh {} {} {}".format(file_system_efs_id, file_system_fsx_id, region),
 in_stream=False,
 )

Expand DownExpand Up@@ -168,7 +187,7 @@ def _check_or_create_efs(sagemaker_session):

def _create_efs_mount(sagemaker_session, file_system_id):
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 efs_client = sagemaker_session.boto_session.client("efs")
 mount_response = efs_client.create_mount_target(
Expand All@@ -188,7 +207,7 @@ def _create_efs_mount(sagemaker_session, file_system_id):
def _check_or_create_fsx(sagemaker_session):
 fsx_client = sagemaker_session.boto_session.client("fsx")
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 create_response = fsx_client.create_file_system(
 FileSystemType="LUSTRE",
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_kmeans_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,6 @@

import pytest

import tests.integ
from sagemaker import KMeans
from sagemaker.amazon.amazon_estimator import FileSystemRecordSet
from sagemaker.parameter import IntegerParameter, CategoricalParameter
Expand All@@ -25,7 +24,6 @@
from tests.integ.s3_utils import assert_s3_files_exist
from tests.integ.timeout import timeout

TRAIN_INSTANCE_TYPE = "ml.c4.xlarge"
TRAIN_INSTANCE_COUNT = 1
OBJECTIVE_METRIC_NAME = "test:msd"
EFS_DIR_PATH = "/one_p_mnist"
Expand All@@ -46,19 +44,15 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -80,19 +74,15 @@ def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -114,18 +104,14 @@ def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand DownExpand Up@@ -174,18 +160,14 @@ def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_tf_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,6 @@

import pytest

import tests.integ
from sagemaker.inputs import FileSystemInput
from sagemaker.parameter import IntegerParameter
from sagemaker.tensorflow import TensorFlow
Expand All@@ -32,7 +31,6 @@
MNIST_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tensorflow_mnist")
SCRIPT = os.path.join(MNIST_RESOURCE_PATH, "mnist.py")
TFS_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tfs", "tfs-test-entrypoint-with-handler")
INSTANCE_TYPE = "ml.c4.xlarge"
EFS_DIR_PATH = "/tensorflow"
FSX_DIR_PATH = "/fsx/tensorflow"
MAX_JOBS = 2
Expand All@@ -49,11 +47,7 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_efs(efs_fsx_setup, sagemaker_session):
def test_mnist_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -62,7 +56,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -85,11 +79,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
def test_mnist_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -98,7 +88,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -121,11 +111,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -134,7 +120,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand DownExpand Up@@ -169,11 +155,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -182,7 +164,7 @@ def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand Down

12 changes: 5 additions & 7 deletions
tests/integ/vpc_test_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -62,7 +62,7 @@ def _route_table_id(ec2_client, vpc_id):
 return desc["RouteTables"][0]["RouteTableId"]


def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NAME):
def check_or_create_vpc_resources_efs_fsx(sagemaker_session, name=VPC_NAME):
 # use lock to prevent race condition when tests are running concurrently
 with lock.lock(LOCK_PATH):
 ec2_client = sagemaker_session.boto_session.client("ec2")
Expand All@@ -74,13 +74,11 @@ def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NA
 _security_group_ids_by_vpc_id(sagemaker_session, vpc_id),
 )
 else:
 return _create_vpc_with_name_efs_fsx(ec2_client, region, name)
 return _create_vpc_with_name_efs_fsx(ec2_client, name)


def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(
 ec2_client, region, name
 )
def _create_vpc_with_name_efs_fsx(ec2_client, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(ec2_client, name)
 ec2_client.modify_vpc_attribute(EnableDnsHostnames={"Value": True}, VpcId=vpc_id)

 ig = ec2_client.create_internet_gateway()
Expand DownExpand Up@@ -121,7 +119,7 @@ def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 return [subnet_id_a], [security_group_id]


def _create_vpc_resources(ec2_client, region, name):
def _create_vpc_resources(ec2_client, name):
 vpc_id = ec2_client.create_vpc(CidrBlock="10.0.0.0/16")["Vpc"]["VpcId"]
 print("created vpc: {}".format(vpc_id))

Expand Down

7 changes: 4 additions & 3 deletions
tests/scripts/fs_mount_setup.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,18 +16,19 @@
# Mounting EFS and FSx for Lustre file systems for integration Tests
FILE_SYSTEM_EFS_ID=$1
FILE_SYSTEM_FSX_ID=$2
REGION=$3

echo "Mounting EFS File Systems"
sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
sudo yum install -y amazon-efs-utils
@laurenyulaurenyuAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do different regions have different versions released?
@caxiaohucaxiaohuAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.

The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.

But it will automatically select the correct version if i don't specify
sudo mkdir efs
sudo mount -t efs "$FILE_SYSTEM_EFS_ID":/ efs
sudo mkdir efs/tensorflow
sudo mkdir efs/one_p_mnist

echo "Mounting FSx for Lustre File System"
sudo amazon-linux-extras install -y lustre2.10
sudo yum install -y lustre-client
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx.us-west-2.amazonaws.com@tcp:/fsx /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx."$REGION".amazonaws.com@tcp:/fsx /mnt/fsx
sudo mkdir /mnt/fsx/tensorflow
sudo mkdir /mnt/fsx/one_p_mnist

Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix: change AMI ids in tests to be dynamic based on regions#1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

chuyang-deng
merged 10 commits into

from

Aug 28, 2019
 Merged


fix: change AMI ids in tests to be dynamic based on regions
#1004
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


10 commits


Select commit
Hold shift + click to select a range
a877ca9

update: change directory_path for EFS usage example in overview doc


caxiaohu Aug 23, 2019
2ca2c9e

update: changed EFS directory path instructions in doc and docstring …


caxiaohu Aug 23, 2019
a4ec855

update: changed AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
1bdf0ad

fix: change Amazon AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
c43ff38

fix: change Amazon AMI ids to be dynamic based on regions using searc…


caxiaohu Aug 27, 2019
f41d3c5

update: move region closer to its usage


caxiaohu Aug 27, 2019
0393384

update: make filter as constant variable


caxiaohu Aug 27, 2019
6b6e070

update: remove else


caxiaohu Aug 27, 2019
b80ccbc

update: add constant variable for search AMI name


caxiaohu Aug 27, 2019
6a95b89

Merge branch 'master' into fix_region_issue


chuyang-deng Aug 28, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 0 additions & 2 deletions
tests/integ/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -71,8 +71,6 @@
NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]
NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]

EFS_TEST_ENABLED_REGION = ["us-west-2"]

logging.getLogger("boto3").setLevel(logging.INFO)
logging.getLogger("botocore").setLevel(logging.INFO)

Expand Down

37 changes: 28 additions & 9 deletions
tests/integ/file_system_input_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,7 @@

import collections
import logging
from operator import itemgetter
import os
from os import path
import stat
Expand All@@ -27,13 +28,12 @@
from tests.integ.vpc_test_utils import check_or_create_vpc_resources_efs_fsx

VPC_NAME = "sagemaker-efs-fsx-vpc"
ALINUX_AMI_NAME_FILTER = "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
EFS_CREATION_TOKEN = str(uuid.uuid4())
PREFIX = "ec2_fs_key_"
KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
ROLE_NAME = "SageMakerRole"
REGION = "us-west-2"
EC2_INSTANCE_TYPE = "t2.micro"
AMI_ID = "ami-082b5a644766e0e6f"
MIN_COUNT = 1
MAX_COUNT = 1

Expand DownExpand Up@@ -69,12 +69,13 @@ def set_up_efs_fsx(sagemaker_session):
 _check_or_create_key_pair(sagemaker_session)
 _check_or_create_iam_profile_and_attach_role(sagemaker_session)
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )

 ami_id = _ami_id_for_region(sagemaker_session)
 ec2_instance = _create_ec2_instance(
 sagemaker_session,
AMI_ID,
ami_id,
 EC2_INSTANCE_TYPE,
 KEY_NAME,
 MIN_COUNT,
Expand All@@ -100,16 +101,34 @@ def set_up_efs_fsx(sagemaker_session):
 mount_efs_target_id,
 )

 region = sagemaker_session.boto_region_name
 try:
 connected_instance = _connect_ec2_instance(ec2_instance)
 _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id)
 _upload_data_and_mount_fs(
 connected_instance, file_system_efs_id, file_system_fsx_id, region
 )
 except Exception:
 tear_down(sagemaker_session, fs_resources)
 raise

 return fs_resources


def _ami_id_for_region(sagemaker_session):
 ec2_client = sagemaker_session.boto_session.client("ec2")
 filters = [
 {"Name": "name", "Values": [ALINUX_AMI_NAME_FILTER]},
 {"Name": "state", "Values": ["available"]},
 ]
 response = ec2_client.describe_images(Filters=filters)
 image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)

 if len(image_details) == 0:
 raise Exception("AMI was not found based on current search criteria: {}".format(filters))

 return image_details[0]["ImageId"]


def _connect_ec2_instance(ec2_instance):
 public_ip_address = ec2_instance.public_ip_address
 connected_instance = Connection(
Expand All@@ -118,7 +137,7 @@ def _connect_ec2_instance(ec2_instance):
 return connected_instance


def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id):
def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id, region):
 connected_instance.put(FS_MOUNT_SCRIPT, ".")
 connected_instance.run("mkdir temp_tf; mkdir temp_one_p", in_stream=False)
 for dir_name, subdir_list, file_list in os.walk(MNIST_LOCAL_DATA):
Expand All@@ -127,7 +146,7 @@ def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_syste
 connected_instance.put(local_file, "temp_tf/")
 connected_instance.put(ONE_P_LOCAL_DATA, "temp_one_p/")
 connected_instance.run(
 "sudo sh fs_mount_setup.sh {} {}".format(file_system_efs_id, file_system_fsx_id),
 "sudo sh fs_mount_setup.sh {} {} {}".format(file_system_efs_id, file_system_fsx_id, region),
 in_stream=False,
 )

Expand DownExpand Up@@ -168,7 +187,7 @@ def _check_or_create_efs(sagemaker_session):

def _create_efs_mount(sagemaker_session, file_system_id):
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 efs_client = sagemaker_session.boto_session.client("efs")
 mount_response = efs_client.create_mount_target(
Expand All@@ -188,7 +207,7 @@ def _create_efs_mount(sagemaker_session, file_system_id):
def _check_or_create_fsx(sagemaker_session):
 fsx_client = sagemaker_session.boto_session.client("fsx")
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 create_response = fsx_client.create_file_system(
 FileSystemType="LUSTRE",
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_kmeans_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,6 @@

import pytest

import tests.integ
from sagemaker import KMeans
from sagemaker.amazon.amazon_estimator import FileSystemRecordSet
from sagemaker.parameter import IntegerParameter, CategoricalParameter
Expand All@@ -25,7 +24,6 @@
from tests.integ.s3_utils import assert_s3_files_exist
from tests.integ.timeout import timeout

TRAIN_INSTANCE_TYPE = "ml.c4.xlarge"
TRAIN_INSTANCE_COUNT = 1
OBJECTIVE_METRIC_NAME = "test:msd"
EFS_DIR_PATH = "/one_p_mnist"
Expand All@@ -46,19 +44,15 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -80,19 +74,15 @@ def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -114,18 +104,14 @@ def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand DownExpand Up@@ -174,18 +160,14 @@ def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_tf_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,6 @@

import pytest

import tests.integ
from sagemaker.inputs import FileSystemInput
from sagemaker.parameter import IntegerParameter
from sagemaker.tensorflow import TensorFlow
Expand All@@ -32,7 +31,6 @@
MNIST_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tensorflow_mnist")
SCRIPT = os.path.join(MNIST_RESOURCE_PATH, "mnist.py")
TFS_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tfs", "tfs-test-entrypoint-with-handler")
INSTANCE_TYPE = "ml.c4.xlarge"
EFS_DIR_PATH = "/tensorflow"
FSX_DIR_PATH = "/fsx/tensorflow"
MAX_JOBS = 2
Expand All@@ -49,11 +47,7 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_efs(efs_fsx_setup, sagemaker_session):
def test_mnist_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -62,7 +56,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -85,11 +79,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
def test_mnist_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -98,7 +88,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -121,11 +111,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -134,7 +120,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand DownExpand Up@@ -169,11 +155,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -182,7 +164,7 @@ def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand Down

12 changes: 5 additions & 7 deletions
tests/integ/vpc_test_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -62,7 +62,7 @@ def _route_table_id(ec2_client, vpc_id):
 return desc["RouteTables"][0]["RouteTableId"]


def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NAME):
def check_or_create_vpc_resources_efs_fsx(sagemaker_session, name=VPC_NAME):
 # use lock to prevent race condition when tests are running concurrently
 with lock.lock(LOCK_PATH):
 ec2_client = sagemaker_session.boto_session.client("ec2")
Expand All@@ -74,13 +74,11 @@ def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NA
 _security_group_ids_by_vpc_id(sagemaker_session, vpc_id),
 )
 else:
 return _create_vpc_with_name_efs_fsx(ec2_client, region, name)
 return _create_vpc_with_name_efs_fsx(ec2_client, name)


def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(
 ec2_client, region, name
 )
def _create_vpc_with_name_efs_fsx(ec2_client, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(ec2_client, name)
 ec2_client.modify_vpc_attribute(EnableDnsHostnames={"Value": True}, VpcId=vpc_id)

 ig = ec2_client.create_internet_gateway()
Expand DownExpand Up@@ -121,7 +119,7 @@ def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 return [subnet_id_a], [security_group_id]


def _create_vpc_resources(ec2_client, region, name):
def _create_vpc_resources(ec2_client, name):
 vpc_id = ec2_client.create_vpc(CidrBlock="10.0.0.0/16")["Vpc"]["VpcId"]
 print("created vpc: {}".format(vpc_id))

Expand Down

7 changes: 4 additions & 3 deletions
tests/scripts/fs_mount_setup.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,18 +16,19 @@
# Mounting EFS and FSx for Lustre file systems for integration Tests
FILE_SYSTEM_EFS_ID=$1
FILE_SYSTEM_FSX_ID=$2
REGION=$3

echo "Mounting EFS File Systems"
sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
sudo yum install -y amazon-efs-utils
@laurenyulaurenyuAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do different regions have different versions released?
@caxiaohucaxiaohuAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.

The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.

But it will automatically select the correct version if i don't specify
sudo mkdir efs
sudo mount -t efs "$FILE_SYSTEM_EFS_ID":/ efs
sudo mkdir efs/tensorflow
sudo mkdir efs/one_p_mnist

echo "Mounting FSx for Lustre File System"
sudo amazon-linux-extras install -y lustre2.10
sudo yum install -y lustre-client
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx.us-west-2.amazonaws.com@tcp:/fsx /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx."$REGION".amazonaws.com@tcp:/fsx /mnt/fsx
sudo mkdir /mnt/fsx/tensorflow
sudo mkdir /mnt/fsx/one_p_mnist

Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
fix: change AMI ids in tests to be dynamic based on regions#1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

chuyang-deng
merged 10 commits into

from

Aug 28, 2019
 Merged


fix: change AMI ids in tests to be dynamic based on regions
#1004
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


10 commits


Select commit
Hold shift + click to select a range
a877ca9

update: change directory_path for EFS usage example in overview doc


caxiaohu Aug 23, 2019
2ca2c9e

update: changed EFS directory path instructions in doc and docstring …


caxiaohu Aug 23, 2019
a4ec855

update: changed AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
1bdf0ad

fix: change Amazon AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
c43ff38

fix: change Amazon AMI ids to be dynamic based on regions using searc…


caxiaohu Aug 27, 2019
f41d3c5

update: move region closer to its usage


caxiaohu Aug 27, 2019
0393384

update: make filter as constant variable


caxiaohu Aug 27, 2019
6b6e070

update: remove else


caxiaohu Aug 27, 2019
b80ccbc

update: add constant variable for search AMI name


caxiaohu Aug 27, 2019
6a95b89

Merge branch 'master' into fix_region_issue


chuyang-deng Aug 28, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 0 additions & 2 deletions
tests/integ/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -71,8 +71,6 @@
NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]
NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]

EFS_TEST_ENABLED_REGION = ["us-west-2"]

logging.getLogger("boto3").setLevel(logging.INFO)
logging.getLogger("botocore").setLevel(logging.INFO)

Expand Down

37 changes: 28 additions & 9 deletions
tests/integ/file_system_input_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,7 @@

import collections
import logging
from operator import itemgetter
import os
from os import path
import stat
Expand All@@ -27,13 +28,12 @@
from tests.integ.vpc_test_utils import check_or_create_vpc_resources_efs_fsx

VPC_NAME = "sagemaker-efs-fsx-vpc"
ALINUX_AMI_NAME_FILTER = "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
EFS_CREATION_TOKEN = str(uuid.uuid4())
PREFIX = "ec2_fs_key_"
KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
ROLE_NAME = "SageMakerRole"
REGION = "us-west-2"
EC2_INSTANCE_TYPE = "t2.micro"
AMI_ID = "ami-082b5a644766e0e6f"
MIN_COUNT = 1
MAX_COUNT = 1

Expand DownExpand Up@@ -69,12 +69,13 @@ def set_up_efs_fsx(sagemaker_session):
 _check_or_create_key_pair(sagemaker_session)
 _check_or_create_iam_profile_and_attach_role(sagemaker_session)
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )

 ami_id = _ami_id_for_region(sagemaker_session)
 ec2_instance = _create_ec2_instance(
 sagemaker_session,
AMI_ID,
ami_id,
 EC2_INSTANCE_TYPE,
 KEY_NAME,
 MIN_COUNT,
Expand All@@ -100,16 +101,34 @@ def set_up_efs_fsx(sagemaker_session):
 mount_efs_target_id,
 )

 region = sagemaker_session.boto_region_name
 try:
 connected_instance = _connect_ec2_instance(ec2_instance)
 _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id)
 _upload_data_and_mount_fs(
 connected_instance, file_system_efs_id, file_system_fsx_id, region
 )
 except Exception:
 tear_down(sagemaker_session, fs_resources)
 raise

 return fs_resources


def _ami_id_for_region(sagemaker_session):
 ec2_client = sagemaker_session.boto_session.client("ec2")
 filters = [
 {"Name": "name", "Values": [ALINUX_AMI_NAME_FILTER]},
 {"Name": "state", "Values": ["available"]},
 ]
 response = ec2_client.describe_images(Filters=filters)
 image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)

 if len(image_details) == 0:
 raise Exception("AMI was not found based on current search criteria: {}".format(filters))

 return image_details[0]["ImageId"]


def _connect_ec2_instance(ec2_instance):
 public_ip_address = ec2_instance.public_ip_address
 connected_instance = Connection(
Expand All@@ -118,7 +137,7 @@ def _connect_ec2_instance(ec2_instance):
 return connected_instance


def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id):
def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id, region):
 connected_instance.put(FS_MOUNT_SCRIPT, ".")
 connected_instance.run("mkdir temp_tf; mkdir temp_one_p", in_stream=False)
 for dir_name, subdir_list, file_list in os.walk(MNIST_LOCAL_DATA):
Expand All@@ -127,7 +146,7 @@ def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_syste
 connected_instance.put(local_file, "temp_tf/")
 connected_instance.put(ONE_P_LOCAL_DATA, "temp_one_p/")
 connected_instance.run(
 "sudo sh fs_mount_setup.sh {} {}".format(file_system_efs_id, file_system_fsx_id),
 "sudo sh fs_mount_setup.sh {} {} {}".format(file_system_efs_id, file_system_fsx_id, region),
 in_stream=False,
 )

Expand DownExpand Up@@ -168,7 +187,7 @@ def _check_or_create_efs(sagemaker_session):

def _create_efs_mount(sagemaker_session, file_system_id):
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 efs_client = sagemaker_session.boto_session.client("efs")
 mount_response = efs_client.create_mount_target(
Expand All@@ -188,7 +207,7 @@ def _create_efs_mount(sagemaker_session, file_system_id):
def _check_or_create_fsx(sagemaker_session):
 fsx_client = sagemaker_session.boto_session.client("fsx")
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 create_response = fsx_client.create_file_system(
 FileSystemType="LUSTRE",
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_kmeans_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,6 @@

import pytest

import tests.integ
from sagemaker import KMeans
from sagemaker.amazon.amazon_estimator import FileSystemRecordSet
from sagemaker.parameter import IntegerParameter, CategoricalParameter
Expand All@@ -25,7 +24,6 @@
from tests.integ.s3_utils import assert_s3_files_exist
from tests.integ.timeout import timeout

TRAIN_INSTANCE_TYPE = "ml.c4.xlarge"
TRAIN_INSTANCE_COUNT = 1
OBJECTIVE_METRIC_NAME = "test:msd"
EFS_DIR_PATH = "/one_p_mnist"
Expand All@@ -46,19 +44,15 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -80,19 +74,15 @@ def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -114,18 +104,14 @@ def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand DownExpand Up@@ -174,18 +160,14 @@ def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_tf_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,6 @@

import pytest

import tests.integ
from sagemaker.inputs import FileSystemInput
from sagemaker.parameter import IntegerParameter
from sagemaker.tensorflow import TensorFlow
Expand All@@ -32,7 +31,6 @@
MNIST_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tensorflow_mnist")
SCRIPT = os.path.join(MNIST_RESOURCE_PATH, "mnist.py")
TFS_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tfs", "tfs-test-entrypoint-with-handler")
INSTANCE_TYPE = "ml.c4.xlarge"
EFS_DIR_PATH = "/tensorflow"
FSX_DIR_PATH = "/fsx/tensorflow"
MAX_JOBS = 2
Expand All@@ -49,11 +47,7 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_efs(efs_fsx_setup, sagemaker_session):
def test_mnist_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -62,7 +56,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -85,11 +79,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
def test_mnist_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -98,7 +88,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -121,11 +111,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -134,7 +120,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand DownExpand Up@@ -169,11 +155,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -182,7 +164,7 @@ def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand Down

12 changes: 5 additions & 7 deletions
tests/integ/vpc_test_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -62,7 +62,7 @@ def _route_table_id(ec2_client, vpc_id):
 return desc["RouteTables"][0]["RouteTableId"]


def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NAME):
def check_or_create_vpc_resources_efs_fsx(sagemaker_session, name=VPC_NAME):
 # use lock to prevent race condition when tests are running concurrently
 with lock.lock(LOCK_PATH):
 ec2_client = sagemaker_session.boto_session.client("ec2")
Expand All@@ -74,13 +74,11 @@ def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NA
 _security_group_ids_by_vpc_id(sagemaker_session, vpc_id),
 )
 else:
 return _create_vpc_with_name_efs_fsx(ec2_client, region, name)
 return _create_vpc_with_name_efs_fsx(ec2_client, name)


def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(
 ec2_client, region, name
 )
def _create_vpc_with_name_efs_fsx(ec2_client, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(ec2_client, name)
 ec2_client.modify_vpc_attribute(EnableDnsHostnames={"Value": True}, VpcId=vpc_id)

 ig = ec2_client.create_internet_gateway()
Expand DownExpand Up@@ -121,7 +119,7 @@ def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 return [subnet_id_a], [security_group_id]


def _create_vpc_resources(ec2_client, region, name):
def _create_vpc_resources(ec2_client, name):
 vpc_id = ec2_client.create_vpc(CidrBlock="10.0.0.0/16")["Vpc"]["VpcId"]
 print("created vpc: {}".format(vpc_id))

Expand Down

7 changes: 4 additions & 3 deletions
tests/scripts/fs_mount_setup.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,18 +16,19 @@
# Mounting EFS and FSx for Lustre file systems for integration Tests
FILE_SYSTEM_EFS_ID=$1
FILE_SYSTEM_FSX_ID=$2
REGION=$3

echo "Mounting EFS File Systems"
sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
sudo yum install -y amazon-efs-utils
@laurenyulaurenyuAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do different regions have different versions released?
@caxiaohucaxiaohuAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.

The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.

But it will automatically select the correct version if i don't specify
sudo mkdir efs
sudo mount -t efs "$FILE_SYSTEM_EFS_ID":/ efs
sudo mkdir efs/tensorflow
sudo mkdir efs/one_p_mnist

echo "Mounting FSx for Lustre File System"
sudo amazon-linux-extras install -y lustre2.10
sudo yum install -y lustre-client
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx.us-west-2.amazonaws.com@tcp:/fsx /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx."$REGION".amazonaws.com@tcp:/fsx /mnt/fsx
sudo mkdir /mnt/fsx/tensorflow
sudo mkdir /mnt/fsx/one_p_mnist

Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix: change AMI ids in tests to be dynamic based on regions#1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

chuyang-deng
merged 10 commits into

from

Aug 28, 2019
 Merged


fix: change AMI ids in tests to be dynamic based on regions
#1004
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


10 commits


Select commit
Hold shift + click to select a range
a877ca9

update: change directory_path for EFS usage example in overview doc


caxiaohu Aug 23, 2019
2ca2c9e

update: changed EFS directory path instructions in doc and docstring …


caxiaohu Aug 23, 2019
a4ec855

update: changed AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
1bdf0ad

fix: change Amazon AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
c43ff38

fix: change Amazon AMI ids to be dynamic based on regions using searc…


caxiaohu Aug 27, 2019
f41d3c5

update: move region closer to its usage


caxiaohu Aug 27, 2019
0393384

update: make filter as constant variable


caxiaohu Aug 27, 2019
6b6e070

update: remove else


caxiaohu Aug 27, 2019
b80ccbc

update: add constant variable for search AMI name


caxiaohu Aug 27, 2019
6a95b89

Merge branch 'master' into fix_region_issue


chuyang-deng Aug 28, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 0 additions & 2 deletions
tests/integ/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -71,8 +71,6 @@
NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]
NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]

EFS_TEST_ENABLED_REGION = ["us-west-2"]

logging.getLogger("boto3").setLevel(logging.INFO)
logging.getLogger("botocore").setLevel(logging.INFO)

Expand Down

37 changes: 28 additions & 9 deletions
tests/integ/file_system_input_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,7 @@

import collections
import logging
from operator import itemgetter
import os
from os import path
import stat
Expand All@@ -27,13 +28,12 @@
from tests.integ.vpc_test_utils import check_or_create_vpc_resources_efs_fsx

VPC_NAME = "sagemaker-efs-fsx-vpc"
ALINUX_AMI_NAME_FILTER = "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
EFS_CREATION_TOKEN = str(uuid.uuid4())
PREFIX = "ec2_fs_key_"
KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
ROLE_NAME = "SageMakerRole"
REGION = "us-west-2"
EC2_INSTANCE_TYPE = "t2.micro"
AMI_ID = "ami-082b5a644766e0e6f"
MIN_COUNT = 1
MAX_COUNT = 1

Expand DownExpand Up@@ -69,12 +69,13 @@ def set_up_efs_fsx(sagemaker_session):
 _check_or_create_key_pair(sagemaker_session)
 _check_or_create_iam_profile_and_attach_role(sagemaker_session)
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )

 ami_id = _ami_id_for_region(sagemaker_session)
 ec2_instance = _create_ec2_instance(
 sagemaker_session,
AMI_ID,
ami_id,
 EC2_INSTANCE_TYPE,
 KEY_NAME,
 MIN_COUNT,
Expand All@@ -100,16 +101,34 @@ def set_up_efs_fsx(sagemaker_session):
 mount_efs_target_id,
 )

 region = sagemaker_session.boto_region_name
 try:
 connected_instance = _connect_ec2_instance(ec2_instance)
 _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id)
 _upload_data_and_mount_fs(
 connected_instance, file_system_efs_id, file_system_fsx_id, region
 )
 except Exception:
 tear_down(sagemaker_session, fs_resources)
 raise

 return fs_resources


def _ami_id_for_region(sagemaker_session):
 ec2_client = sagemaker_session.boto_session.client("ec2")
 filters = [
 {"Name": "name", "Values": [ALINUX_AMI_NAME_FILTER]},
 {"Name": "state", "Values": ["available"]},
 ]
 response = ec2_client.describe_images(Filters=filters)
 image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)

 if len(image_details) == 0:
 raise Exception("AMI was not found based on current search criteria: {}".format(filters))

 return image_details[0]["ImageId"]


def _connect_ec2_instance(ec2_instance):
 public_ip_address = ec2_instance.public_ip_address
 connected_instance = Connection(
Expand All@@ -118,7 +137,7 @@ def _connect_ec2_instance(ec2_instance):
 return connected_instance


def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id):
def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id, region):
 connected_instance.put(FS_MOUNT_SCRIPT, ".")
 connected_instance.run("mkdir temp_tf; mkdir temp_one_p", in_stream=False)
 for dir_name, subdir_list, file_list in os.walk(MNIST_LOCAL_DATA):
Expand All@@ -127,7 +146,7 @@ def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_syste
 connected_instance.put(local_file, "temp_tf/")
 connected_instance.put(ONE_P_LOCAL_DATA, "temp_one_p/")
 connected_instance.run(
 "sudo sh fs_mount_setup.sh {} {}".format(file_system_efs_id, file_system_fsx_id),
 "sudo sh fs_mount_setup.sh {} {} {}".format(file_system_efs_id, file_system_fsx_id, region),
 in_stream=False,
 )

Expand DownExpand Up@@ -168,7 +187,7 @@ def _check_or_create_efs(sagemaker_session):

def _create_efs_mount(sagemaker_session, file_system_id):
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 efs_client = sagemaker_session.boto_session.client("efs")
 mount_response = efs_client.create_mount_target(
Expand All@@ -188,7 +207,7 @@ def _create_efs_mount(sagemaker_session, file_system_id):
def _check_or_create_fsx(sagemaker_session):
 fsx_client = sagemaker_session.boto_session.client("fsx")
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 create_response = fsx_client.create_file_system(
 FileSystemType="LUSTRE",
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_kmeans_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,6 @@

import pytest

import tests.integ
from sagemaker import KMeans
from sagemaker.amazon.amazon_estimator import FileSystemRecordSet
from sagemaker.parameter import IntegerParameter, CategoricalParameter
Expand All@@ -25,7 +24,6 @@
from tests.integ.s3_utils import assert_s3_files_exist
from tests.integ.timeout import timeout

TRAIN_INSTANCE_TYPE = "ml.c4.xlarge"
TRAIN_INSTANCE_COUNT = 1
OBJECTIVE_METRIC_NAME = "test:msd"
EFS_DIR_PATH = "/one_p_mnist"
Expand All@@ -46,19 +44,15 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -80,19 +74,15 @@ def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -114,18 +104,14 @@ def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand DownExpand Up@@ -174,18 +160,14 @@ def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_tf_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,6 @@

import pytest

import tests.integ
from sagemaker.inputs import FileSystemInput
from sagemaker.parameter import IntegerParameter
from sagemaker.tensorflow import TensorFlow
Expand All@@ -32,7 +31,6 @@
MNIST_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tensorflow_mnist")
SCRIPT = os.path.join(MNIST_RESOURCE_PATH, "mnist.py")
TFS_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tfs", "tfs-test-entrypoint-with-handler")
INSTANCE_TYPE = "ml.c4.xlarge"
EFS_DIR_PATH = "/tensorflow"
FSX_DIR_PATH = "/fsx/tensorflow"
MAX_JOBS = 2
Expand All@@ -49,11 +47,7 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_efs(efs_fsx_setup, sagemaker_session):
def test_mnist_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -62,7 +56,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -85,11 +79,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
def test_mnist_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -98,7 +88,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -121,11 +111,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -134,7 +120,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand DownExpand Up@@ -169,11 +155,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -182,7 +164,7 @@ def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand Down

12 changes: 5 additions & 7 deletions
tests/integ/vpc_test_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -62,7 +62,7 @@ def _route_table_id(ec2_client, vpc_id):
 return desc["RouteTables"][0]["RouteTableId"]


def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NAME):
def check_or_create_vpc_resources_efs_fsx(sagemaker_session, name=VPC_NAME):
 # use lock to prevent race condition when tests are running concurrently
 with lock.lock(LOCK_PATH):
 ec2_client = sagemaker_session.boto_session.client("ec2")
Expand All@@ -74,13 +74,11 @@ def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NA
 _security_group_ids_by_vpc_id(sagemaker_session, vpc_id),
 )
 else:
 return _create_vpc_with_name_efs_fsx(ec2_client, region, name)
 return _create_vpc_with_name_efs_fsx(ec2_client, name)


def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(
 ec2_client, region, name
 )
def _create_vpc_with_name_efs_fsx(ec2_client, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(ec2_client, name)
 ec2_client.modify_vpc_attribute(EnableDnsHostnames={"Value": True}, VpcId=vpc_id)

 ig = ec2_client.create_internet_gateway()
Expand DownExpand Up@@ -121,7 +119,7 @@ def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 return [subnet_id_a], [security_group_id]


def _create_vpc_resources(ec2_client, region, name):
def _create_vpc_resources(ec2_client, name):
 vpc_id = ec2_client.create_vpc(CidrBlock="10.0.0.0/16")["Vpc"]["VpcId"]
 print("created vpc: {}".format(vpc_id))

Expand Down

7 changes: 4 additions & 3 deletions
tests/scripts/fs_mount_setup.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,18 +16,19 @@
# Mounting EFS and FSx for Lustre file systems for integration Tests
FILE_SYSTEM_EFS_ID=$1
FILE_SYSTEM_FSX_ID=$2
REGION=$3

echo "Mounting EFS File Systems"
sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
sudo yum install -y amazon-efs-utils
@laurenyulaurenyuAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do different regions have different versions released?
@caxiaohucaxiaohuAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.

The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.

But it will automatically select the correct version if i don't specify
sudo mkdir efs
sudo mount -t efs "$FILE_SYSTEM_EFS_ID":/ efs
sudo mkdir efs/tensorflow
sudo mkdir efs/one_p_mnist

echo "Mounting FSx for Lustre File System"
sudo amazon-linux-extras install -y lustre2.10
sudo yum install -y lustre-client
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx.us-west-2.amazonaws.com@tcp:/fsx /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx."$REGION".amazonaws.com@tcp:/fsx /mnt/fsx
sudo mkdir /mnt/fsx/tensorflow
sudo mkdir /mnt/fsx/one_p_mnist

Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix: change AMI ids in tests to be dynamic based on regions#1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

chuyang-deng
merged 10 commits into

from

Aug 28, 2019
 Merged


fix: change AMI ids in tests to be dynamic based on regions
#1004
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


10 commits


Select commit
Hold shift + click to select a range
a877ca9

update: change directory_path for EFS usage example in overview doc


caxiaohu Aug 23, 2019
2ca2c9e

update: changed EFS directory path instructions in doc and docstring …


caxiaohu Aug 23, 2019
a4ec855

update: changed AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
1bdf0ad

fix: change Amazon AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
c43ff38

fix: change Amazon AMI ids to be dynamic based on regions using searc…


caxiaohu Aug 27, 2019
f41d3c5

update: move region closer to its usage


caxiaohu Aug 27, 2019
0393384

update: make filter as constant variable


caxiaohu Aug 27, 2019
6b6e070

update: remove else


caxiaohu Aug 27, 2019
b80ccbc

update: add constant variable for search AMI name


caxiaohu Aug 27, 2019
6a95b89

Merge branch 'master' into fix_region_issue


chuyang-deng Aug 28, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 0 additions & 2 deletions
tests/integ/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -71,8 +71,6 @@
NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]
NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]

EFS_TEST_ENABLED_REGION = ["us-west-2"]

logging.getLogger("boto3").setLevel(logging.INFO)
logging.getLogger("botocore").setLevel(logging.INFO)

Expand Down

37 changes: 28 additions & 9 deletions
tests/integ/file_system_input_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,7 @@

import collections
import logging
from operator import itemgetter
import os
from os import path
import stat
Expand All@@ -27,13 +28,12 @@
from tests.integ.vpc_test_utils import check_or_create_vpc_resources_efs_fsx

VPC_NAME = "sagemaker-efs-fsx-vpc"
ALINUX_AMI_NAME_FILTER = "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
EFS_CREATION_TOKEN = str(uuid.uuid4())
PREFIX = "ec2_fs_key_"
KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
ROLE_NAME = "SageMakerRole"
REGION = "us-west-2"
EC2_INSTANCE_TYPE = "t2.micro"
AMI_ID = "ami-082b5a644766e0e6f"
MIN_COUNT = 1
MAX_COUNT = 1

Expand DownExpand Up@@ -69,12 +69,13 @@ def set_up_efs_fsx(sagemaker_session):
 _check_or_create_key_pair(sagemaker_session)
 _check_or_create_iam_profile_and_attach_role(sagemaker_session)
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )

 ami_id = _ami_id_for_region(sagemaker_session)
 ec2_instance = _create_ec2_instance(
 sagemaker_session,
AMI_ID,
ami_id,
 EC2_INSTANCE_TYPE,
 KEY_NAME,
 MIN_COUNT,
Expand All@@ -100,16 +101,34 @@ def set_up_efs_fsx(sagemaker_session):
 mount_efs_target_id,
 )

 region = sagemaker_session.boto_region_name
 try:
 connected_instance = _connect_ec2_instance(ec2_instance)
 _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id)
 _upload_data_and_mount_fs(
 connected_instance, file_system_efs_id, file_system_fsx_id, region
 )
 except Exception:
 tear_down(sagemaker_session, fs_resources)
 raise

 return fs_resources


def _ami_id_for_region(sagemaker_session):
 ec2_client = sagemaker_session.boto_session.client("ec2")
 filters = [
 {"Name": "name", "Values": [ALINUX_AMI_NAME_FILTER]},
 {"Name": "state", "Values": ["available"]},
 ]
 response = ec2_client.describe_images(Filters=filters)
 image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)

 if len(image_details) == 0:
 raise Exception("AMI was not found based on current search criteria: {}".format(filters))

 return image_details[0]["ImageId"]


def _connect_ec2_instance(ec2_instance):
 public_ip_address = ec2_instance.public_ip_address
 connected_instance = Connection(
Expand All@@ -118,7 +137,7 @@ def _connect_ec2_instance(ec2_instance):
 return connected_instance


def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id):
def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id, region):
 connected_instance.put(FS_MOUNT_SCRIPT, ".")
 connected_instance.run("mkdir temp_tf; mkdir temp_one_p", in_stream=False)
 for dir_name, subdir_list, file_list in os.walk(MNIST_LOCAL_DATA):
Expand All@@ -127,7 +146,7 @@ def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_syste
 connected_instance.put(local_file, "temp_tf/")
 connected_instance.put(ONE_P_LOCAL_DATA, "temp_one_p/")
 connected_instance.run(
 "sudo sh fs_mount_setup.sh {} {}".format(file_system_efs_id, file_system_fsx_id),
 "sudo sh fs_mount_setup.sh {} {} {}".format(file_system_efs_id, file_system_fsx_id, region),
 in_stream=False,
 )

Expand DownExpand Up@@ -168,7 +187,7 @@ def _check_or_create_efs(sagemaker_session):

def _create_efs_mount(sagemaker_session, file_system_id):
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 efs_client = sagemaker_session.boto_session.client("efs")
 mount_response = efs_client.create_mount_target(
Expand All@@ -188,7 +207,7 @@ def _create_efs_mount(sagemaker_session, file_system_id):
def _check_or_create_fsx(sagemaker_session):
 fsx_client = sagemaker_session.boto_session.client("fsx")
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 create_response = fsx_client.create_file_system(
 FileSystemType="LUSTRE",
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_kmeans_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,6 @@

import pytest

import tests.integ
from sagemaker import KMeans
from sagemaker.amazon.amazon_estimator import FileSystemRecordSet
from sagemaker.parameter import IntegerParameter, CategoricalParameter
Expand All@@ -25,7 +24,6 @@
from tests.integ.s3_utils import assert_s3_files_exist
from tests.integ.timeout import timeout

TRAIN_INSTANCE_TYPE = "ml.c4.xlarge"
TRAIN_INSTANCE_COUNT = 1
OBJECTIVE_METRIC_NAME = "test:msd"
EFS_DIR_PATH = "/one_p_mnist"
Expand All@@ -46,19 +44,15 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -80,19 +74,15 @@ def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -114,18 +104,14 @@ def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand DownExpand Up@@ -174,18 +160,14 @@ def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_tf_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,6 @@

import pytest

import tests.integ
from sagemaker.inputs import FileSystemInput
from sagemaker.parameter import IntegerParameter
from sagemaker.tensorflow import TensorFlow
Expand All@@ -32,7 +31,6 @@
MNIST_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tensorflow_mnist")
SCRIPT = os.path.join(MNIST_RESOURCE_PATH, "mnist.py")
TFS_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tfs", "tfs-test-entrypoint-with-handler")
INSTANCE_TYPE = "ml.c4.xlarge"
EFS_DIR_PATH = "/tensorflow"
FSX_DIR_PATH = "/fsx/tensorflow"
MAX_JOBS = 2
Expand All@@ -49,11 +47,7 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_efs(efs_fsx_setup, sagemaker_session):
def test_mnist_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -62,7 +56,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -85,11 +79,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
def test_mnist_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -98,7 +88,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -121,11 +111,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -134,7 +120,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand DownExpand Up@@ -169,11 +155,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -182,7 +164,7 @@ def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand Down

12 changes: 5 additions & 7 deletions
tests/integ/vpc_test_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -62,7 +62,7 @@ def _route_table_id(ec2_client, vpc_id):
 return desc["RouteTables"][0]["RouteTableId"]


def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NAME):
def check_or_create_vpc_resources_efs_fsx(sagemaker_session, name=VPC_NAME):
 # use lock to prevent race condition when tests are running concurrently
 with lock.lock(LOCK_PATH):
 ec2_client = sagemaker_session.boto_session.client("ec2")
Expand All@@ -74,13 +74,11 @@ def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NA
 _security_group_ids_by_vpc_id(sagemaker_session, vpc_id),
 )
 else:
 return _create_vpc_with_name_efs_fsx(ec2_client, region, name)
 return _create_vpc_with_name_efs_fsx(ec2_client, name)


def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(
 ec2_client, region, name
 )
def _create_vpc_with_name_efs_fsx(ec2_client, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(ec2_client, name)
 ec2_client.modify_vpc_attribute(EnableDnsHostnames={"Value": True}, VpcId=vpc_id)

 ig = ec2_client.create_internet_gateway()
Expand DownExpand Up@@ -121,7 +119,7 @@ def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 return [subnet_id_a], [security_group_id]


def _create_vpc_resources(ec2_client, region, name):
def _create_vpc_resources(ec2_client, name):
 vpc_id = ec2_client.create_vpc(CidrBlock="10.0.0.0/16")["Vpc"]["VpcId"]
 print("created vpc: {}".format(vpc_id))

Expand Down

7 changes: 4 additions & 3 deletions
tests/scripts/fs_mount_setup.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,18 +16,19 @@
# Mounting EFS and FSx for Lustre file systems for integration Tests
FILE_SYSTEM_EFS_ID=$1
FILE_SYSTEM_FSX_ID=$2
REGION=$3

echo "Mounting EFS File Systems"
sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
sudo yum install -y amazon-efs-utils
@laurenyulaurenyuAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do different regions have different versions released?
@caxiaohucaxiaohuAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.

The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.

But it will automatically select the correct version if i don't specify
sudo mkdir efs
sudo mount -t efs "$FILE_SYSTEM_EFS_ID":/ efs
sudo mkdir efs/tensorflow
sudo mkdir efs/one_p_mnist

echo "Mounting FSx for Lustre File System"
sudo amazon-linux-extras install -y lustre2.10
sudo yum install -y lustre-client
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx.us-west-2.amazonaws.com@tcp:/fsx /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx."$REGION".amazonaws.com@tcp:/fsx /mnt/fsx
sudo mkdir /mnt/fsx/tensorflow
sudo mkdir /mnt/fsx/one_p_mnist

Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
fix: change AMI ids in tests to be dynamic based on regions#1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

chuyang-deng
merged 10 commits into

from

Aug 28, 2019
 Merged


fix: change AMI ids in tests to be dynamic based on regions
#1004
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


10 commits


Select commit
Hold shift + click to select a range
a877ca9

update: change directory_path for EFS usage example in overview doc


caxiaohu Aug 23, 2019
2ca2c9e

update: changed EFS directory path instructions in doc and docstring …


caxiaohu Aug 23, 2019
a4ec855

update: changed AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
1bdf0ad

fix: change Amazon AMI ids to be dynamic based on regions


caxiaohu Aug 27, 2019
c43ff38

fix: change Amazon AMI ids to be dynamic based on regions using searc…


caxiaohu Aug 27, 2019
f41d3c5

update: move region closer to its usage


caxiaohu Aug 27, 2019
0393384

update: make filter as constant variable


caxiaohu Aug 27, 2019
6b6e070

update: remove else


caxiaohu Aug 27, 2019
b80ccbc

update: add constant variable for search AMI name


caxiaohu Aug 27, 2019
6a95b89

Merge branch 'master' into fix_region_issue


chuyang-deng Aug 28, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 0 additions & 2 deletions
tests/integ/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -71,8 +71,6 @@
NO_LDA_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]
NO_MARKET_PLACE_REGIONS = ["eu-west-3", "eu-north-1", "sa-east-1", "ap-east-1"]

EFS_TEST_ENABLED_REGION = ["us-west-2"]

logging.getLogger("boto3").setLevel(logging.INFO)
logging.getLogger("botocore").setLevel(logging.INFO)

Expand Down

37 changes: 28 additions & 9 deletions
tests/integ/file_system_input_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,7 @@

import collections
import logging
from operator import itemgetter
import os
from os import path
import stat
Expand All@@ -27,13 +28,12 @@
from tests.integ.vpc_test_utils import check_or_create_vpc_resources_efs_fsx

VPC_NAME = "sagemaker-efs-fsx-vpc"
ALINUX_AMI_NAME_FILTER = "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
EFS_CREATION_TOKEN = str(uuid.uuid4())
PREFIX = "ec2_fs_key_"
KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
ROLE_NAME = "SageMakerRole"
REGION = "us-west-2"
EC2_INSTANCE_TYPE = "t2.micro"
AMI_ID = "ami-082b5a644766e0e6f"
MIN_COUNT = 1
MAX_COUNT = 1

Expand DownExpand Up@@ -69,12 +69,13 @@ def set_up_efs_fsx(sagemaker_session):
 _check_or_create_key_pair(sagemaker_session)
 _check_or_create_iam_profile_and_attach_role(sagemaker_session)
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )

 ami_id = _ami_id_for_region(sagemaker_session)
 ec2_instance = _create_ec2_instance(
 sagemaker_session,
AMI_ID,
ami_id,
 EC2_INSTANCE_TYPE,
 KEY_NAME,
 MIN_COUNT,
Expand All@@ -100,16 +101,34 @@ def set_up_efs_fsx(sagemaker_session):
 mount_efs_target_id,
 )

 region = sagemaker_session.boto_region_name
 try:
 connected_instance = _connect_ec2_instance(ec2_instance)
 _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id)
 _upload_data_and_mount_fs(
 connected_instance, file_system_efs_id, file_system_fsx_id, region
 )
 except Exception:
 tear_down(sagemaker_session, fs_resources)
 raise

 return fs_resources


def _ami_id_for_region(sagemaker_session):
 ec2_client = sagemaker_session.boto_session.client("ec2")
 filters = [
 {"Name": "name", "Values": [ALINUX_AMI_NAME_FILTER]},
 {"Name": "state", "Values": ["available"]},
 ]
 response = ec2_client.describe_images(Filters=filters)
 image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)

 if len(image_details) == 0:
 raise Exception("AMI was not found based on current search criteria: {}".format(filters))

 return image_details[0]["ImageId"]


def _connect_ec2_instance(ec2_instance):
 public_ip_address = ec2_instance.public_ip_address
 connected_instance = Connection(
Expand All@@ -118,7 +137,7 @@ def _connect_ec2_instance(ec2_instance):
 return connected_instance


def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id):
def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_system_fsx_id, region):
 connected_instance.put(FS_MOUNT_SCRIPT, ".")
 connected_instance.run("mkdir temp_tf; mkdir temp_one_p", in_stream=False)
 for dir_name, subdir_list, file_list in os.walk(MNIST_LOCAL_DATA):
Expand All@@ -127,7 +146,7 @@ def _upload_data_and_mount_fs(connected_instance, file_system_efs_id, file_syste
 connected_instance.put(local_file, "temp_tf/")
 connected_instance.put(ONE_P_LOCAL_DATA, "temp_one_p/")
 connected_instance.run(
 "sudo sh fs_mount_setup.sh {} {}".format(file_system_efs_id, file_system_fsx_id),
 "sudo sh fs_mount_setup.sh {} {} {}".format(file_system_efs_id, file_system_fsx_id, region),
 in_stream=False,
 )

Expand DownExpand Up@@ -168,7 +187,7 @@ def _check_or_create_efs(sagemaker_session):

def _create_efs_mount(sagemaker_session, file_system_id):
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 efs_client = sagemaker_session.boto_session.client("efs")
 mount_response = efs_client.create_mount_target(
Expand All@@ -188,7 +207,7 @@ def _create_efs_mount(sagemaker_session, file_system_id):
def _check_or_create_fsx(sagemaker_session):
 fsx_client = sagemaker_session.boto_session.client("fsx")
 subnet_ids, security_group_ids = check_or_create_vpc_resources_efs_fsx(
 sagemaker_session, REGION, VPC_NAME
 sagemaker_session, VPC_NAME
 )
 create_response = fsx_client.create_file_system(
 FileSystemType="LUSTRE",
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_kmeans_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,6 @@

import pytest

import tests.integ
from sagemaker import KMeans
from sagemaker.amazon.amazon_estimator import FileSystemRecordSet
from sagemaker.parameter import IntegerParameter, CategoricalParameter
Expand All@@ -25,7 +24,6 @@
from tests.integ.s3_utils import assert_s3_files_exist
from tests.integ.timeout import timeout

TRAIN_INSTANCE_TYPE = "ml.c4.xlarge"
TRAIN_INSTANCE_COUNT = 1
OBJECTIVE_METRIC_NAME = "test:msd"
EFS_DIR_PATH = "/one_p_mnist"
Expand All@@ -46,19 +44,15 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -80,19 +74,15 @@ def test_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 with timeout(minutes=TRAINING_DEFAULT_TIMEOUT_MINUTES):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand All@@ -114,18 +104,14 @@ def test_kmeans_fsx(efs_fsx_setup, sagemaker_session):
 assert_s3_files_exist(sagemaker_session, model_path, ["model.tar.gz"])


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand DownExpand Up@@ -174,18 +160,14 @@ def test_tuning_kmeans_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session):
def test_tuning_kmeans_fsx(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
 role = efs_fsx_setup.role_name
 kmeans = KMeans(
 role=role,
 train_instance_count=TRAIN_INSTANCE_COUNT,
 train_instance_type=TRAIN_INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 k=K,
 sagemaker_session=sagemaker_session,
 subnets=subnets,
Expand Down

34 changes: 8 additions & 26 deletions
tests/integ/test_tf_efs_fsx.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,6 @@

import pytest

import tests.integ
from sagemaker.inputs import FileSystemInput
from sagemaker.parameter import IntegerParameter
from sagemaker.tensorflow import TensorFlow
Expand All@@ -32,7 +31,6 @@
MNIST_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tensorflow_mnist")
SCRIPT = os.path.join(MNIST_RESOURCE_PATH, "mnist.py")
TFS_RESOURCE_PATH = os.path.join(RESOURCE_PATH, "tfs", "tfs-test-entrypoint-with-handler")
INSTANCE_TYPE = "ml.c4.xlarge"
EFS_DIR_PATH = "/tensorflow"
FSX_DIR_PATH = "/fsx/tensorflow"
MAX_JOBS = 2
Expand All@@ -49,11 +47,7 @@ def efs_fsx_setup(sagemaker_session):
 tear_down(sagemaker_session, fs_resources)


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_efs(efs_fsx_setup, sagemaker_session):
def test_mnist_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -62,7 +56,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -85,11 +79,7 @@ def test_mnist_efs(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
def test_mnist_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -98,7 +88,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 sagemaker_session=sagemaker_session,
 script_mode=True,
 framework_version=TensorFlow.LATEST_VERSION,
Expand All@@ -121,11 +111,7 @@ def test_mnist_lustre(efs_fsx_setup, sagemaker_session):
 )


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -134,7 +120,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand DownExpand Up@@ -169,11 +155,7 @@ def test_tuning_tf_script_mode_efs(efs_fsx_setup, sagemaker_session):
 assert best_training_job


@pytest.mark.skipif(
 tests.integ.test_region() not in tests.integ.EFS_TEST_ENABLED_REGION,
 reason="EFS integration tests need to be fixed before running in all regions.",
)
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session, cpu_instance_type):
 role = efs_fsx_setup.role_name
 subnets = [efs_fsx_setup.subnet_id]
 security_group_ids = efs_fsx_setup.security_group_ids
Expand All@@ -182,7 +164,7 @@ def test_tuning_tf_script_mode_lustre(efs_fsx_setup, sagemaker_session):
 entry_point=SCRIPT,
 role=role,
 train_instance_count=1,
 train_instance_type=INSTANCE_TYPE,
 train_instance_type=cpu_instance_type,
 script_mode=True,
 sagemaker_session=sagemaker_session,
 py_version=PY_VERSION,
Expand Down

12 changes: 5 additions & 7 deletions
tests/integ/vpc_test_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -62,7 +62,7 @@ def _route_table_id(ec2_client, vpc_id):
 return desc["RouteTables"][0]["RouteTableId"]


def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NAME):
def check_or_create_vpc_resources_efs_fsx(sagemaker_session, name=VPC_NAME):
 # use lock to prevent race condition when tests are running concurrently
 with lock.lock(LOCK_PATH):
 ec2_client = sagemaker_session.boto_session.client("ec2")
Expand All@@ -74,13 +74,11 @@ def check_or_create_vpc_resources_efs_fsx(sagemaker_session, region, name=VPC_NA
 _security_group_ids_by_vpc_id(sagemaker_session, vpc_id),
 )
 else:
 return _create_vpc_with_name_efs_fsx(ec2_client, region, name)
 return _create_vpc_with_name_efs_fsx(ec2_client, name)


def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(
 ec2_client, region, name
 )
def _create_vpc_with_name_efs_fsx(ec2_client, name):
 vpc_id, [subnet_id_a, subnet_id_b], security_group_id = _create_vpc_resources(ec2_client, name)
 ec2_client.modify_vpc_attribute(EnableDnsHostnames={"Value": True}, VpcId=vpc_id)

 ig = ec2_client.create_internet_gateway()
Expand DownExpand Up@@ -121,7 +119,7 @@ def _create_vpc_with_name_efs_fsx(ec2_client, region, name):
 return [subnet_id_a], [security_group_id]


def _create_vpc_resources(ec2_client, region, name):
def _create_vpc_resources(ec2_client, name):
 vpc_id = ec2_client.create_vpc(CidrBlock="10.0.0.0/16")["Vpc"]["VpcId"]
 print("created vpc: {}".format(vpc_id))

Expand Down

7 changes: 4 additions & 3 deletions
tests/scripts/fs_mount_setup.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,18 +16,19 @@
# Mounting EFS and FSx for Lustre file systems for integration Tests
FILE_SYSTEM_EFS_ID=$1
FILE_SYSTEM_FSX_ID=$2
REGION=$3

echo "Mounting EFS File Systems"
sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
sudo yum install -y amazon-efs-utils
@laurenyulaurenyuAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do different regions have different versions released?
@caxiaohucaxiaohuAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.

The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.

But it will automatically select the correct version if i don't specify
sudo mkdir efs
sudo mount -t efs "$FILE_SYSTEM_EFS_ID":/ efs
sudo mkdir efs/tensorflow
sudo mkdir efs/one_p_mnist

echo "Mounting FSx for Lustre File System"
sudo amazon-linux-extras install -y lustre2.10
sudo yum install -y lustre-client
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx.us-west-2.amazonaws.com@tcp:/fsx /mnt/fsx
sudo mount -t lustre -o noatime,flock "$FILE_SYSTEM_FSX_ID".fsx."$REGION".amazonaws.com@tcp:/fsx /mnt/fsx
sudo mkdir /mnt/fsx/tensorflow
sudo mkdir /mnt/fsx/one_p_mnist

Expand Down