Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nadiaya merged 5 commits into from
Feb 27, 2018
Merged

Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4fac858
Update .gitignore to ignore pytest_cache.
nadiaya Feb 22, 2018
c6b444a
Support TensorFlow-1.5.0 and MXNet-1.0.0
nadiaya Feb 22, 2018
d854806
Update and refactor tests. Add tests for fw_utils.
nadiaya Feb 24, 2018
dcfe663
Fix typo.
nadiaya Feb 26, 2018
aa64686
Merge branch 'master' into release-1.1
nadiaya Feb 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -21,3 +21,4 @@ doc/_build
**/.DS_Store
venv/
*~
.pytest_cache/
18 changes: 12 additions & 6 deletions CHANGELOG.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,29 +2,35 @@
CHANGELOG
=========

1.1.0
=====

* feature: Estimators: add support for TensorFlow-1.5.0

@lukmislukmisFeb 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update after master branch merge.

@nadiayanadiayaFeb 23, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.

* feature: Estimators: add support for MXNet-1.0.0

1.0.4
=====

* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
* feature: Documentation: Fix description of an argument of sagemaker.session.train
* feature: Documentation: Add FM and LDA to the documentation
* feature: Documentation: fix description of an argument of sagemaker.session.train
* feature: Documentation: add FM and LDA to the documentation
* feature: Estimators: add support for async fit
* bug-fix: Estimators: fix estimator role expansion

1.0.3
=====

* feature: Estimators: add support for Amazon LDA algorithm
* feature: Hyperparameters: Add data_type to hyperparameters
* feature: Documentation: Update TensorFlow examples following API change
* feature: Session: Support multi-part uploads
* feature: Hyperparameters: add data_type to hyperparameters
* feature: Documentation: update TensorFlow examples following API change
* feature: Session: support multi-part uploads


1.0.2
=====

* feature: Estimators: add support for Amazon FactorizationMachines algorithm
* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
* feature: Tests: add training failure tests for TF and MXNet
* feature: Documentation: show how to make predictions against existing endpoint
* feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
Expand Down
89 changes: 60 additions & 29 deletions README.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install

git clone https://github.com/aws/sagemaker-python-sdk.git
python setup.py sdist
pip install dist/sagemaker-1.0.4.tar.gz
pip install dist/sagemaker-1.1.0.tar.gz

Supported Python versions
~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -114,6 +114,8 @@ MXNet SageMaker Estimators

With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.0.0``, ``0.12.1``.

Training with MXNet
~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
Using MXNet and numpy
^^^^^^^^^^^^^^^^^^^^^

You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.

Running an MXNet training script in SageMaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script

SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.

@winstonawswinstonawsFeb 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write this to be more future-proof? Maybe something like, "will run using the latest supported version of MXNet by default. See (something) for our currently supported versions. You can choose older versions by..."

@nadiayanadiayaFeb 23, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we still would need to make an sdk change to change the default.
I believe it adds value to say what version we support by default.


Each Docker container has the following dependencies installed:
The Docker images have the following dependencies installed:

- Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
the MXNet constructor.
- MXNet 0.12, built for either GPU or CPU, depending on the instance
type for training or deploying.
- CUDA 9.0
- numpy 1.12
+-------------------------+--------------+-------------+
| Dependencies | MXNet 0.12.1 | MXNet 1.0.0 |
+-------------------------+--------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5|
+-------------------------+--------------+-------------+
| CUDA | 9.0 | 9.0 |
+-------------------------+--------------+-------------+
| numpy | 1.13.3 | 1.13.3 |
+-------------------------+--------------+-------------+

The Docker images extend Ubuntu 16.04.

You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.

TensorFlow SageMaker Estimators
-------------------------------

TensorFlow SageMaker Estimators allow you to run your own TensorFlow
training algorithms on SageMaker Learner, and to host your own TensorFlow
models on SageMaker Hosting.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.

Training with TensorFlow
~~~~~~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
version is **1.4.0**. This training script **must contain** the following functions:
versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:

@winstonawswinstonawsFeb 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about future-proofing.


- ``model_fn``: defines the model that will be trained.
- ``train_input_fn``: preprocess and load training data.
Expand DownExpand Up@@ -1419,27 +1428,49 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
SageMaker TensorFlow Docker containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
- awscli 1.12.1
- boto3 1.4.7
- botocore 1.5.92
- futures 2.2.0
- gevent 1.2.2
- grpcio 1.7.0
- numpy 1.13.3
- pandas 0.21.0
- protobuf 3.4.0
- requests 2.14.2
- scikit-learn 0.19.1
- scipy 1.0.0
- six 1.10.0
- sklearn 0.0
- tensorflow 1.4.0
- tensorflow-serving-api 1.4.0
- tensorflow-tensorboard 0.4.0rc2
The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:

+------------------------+------------------+------------------+
| Dependencies | tensorflow 1.4.1 | tensorflow 1.5.0 |
+------------------------+------------------+------------------+
| awscli | 1.12.1 | 1.14.35 |
+------------------------+------------------+------------------+
| boto3 | 1.4.7 | 1.5.22 |
+------------------------+------------------+------------------+
| botocore | 1.5.92 | 1.8.36 |
+------------------------+------------------+------------------+
| futures | 2.2.0 | 2.2.0 |
+------------------------+------------------+------------------+
| gevent | 1.2.2 | 1.2.2 |
+------------------------+------------------+------------------+
| grpcio | 1.7.0 | 1.9.0 |
+------------------------+------------------+------------------+
| numpy | 1.13.3 | 1.14.0 |
+------------------------+------------------+------------------+
| pandas | 0.21.0 | 0.22.0 |
+------------------------+------------------+------------------+
| protobuf | 3.4.0 | 3.5.1 |
+------------------------+------------------+------------------+
| requests | 2.14.2 | 2.18.4 |
+------------------------+------------------+------------------+
| scikit-learn | 0.19.1 | 0.19.1 |
+------------------------+------------------+------------------+
| scipy | 1.0.0 | 1.0.0 |
+------------------------+------------------+------------------+
| six | 1.10.0 | 1.10.0 |
+------------------------+------------------+------------------+
| sklearn | 0.0 | 0.0 |
+------------------------+------------------+------------------+
| tensorflow | 1.4.1 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-serving-api | 1.4.0 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-tensorboard | 0.4.0 | 1.5.1 |
+------------------------+------------------+------------------+

The Docker images extend Ubuntu 16.04.

You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.

AWS SageMaker Estimators
------------------------
Expand Down
2 changes: 1 addition & 1 deletion doc/conf.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@ def __getattr__(cls, name):
'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

version = '1.0.4'
version = '1.1.0'
project = u'sagemaker'

# Add any Sphinx extension module names here, as strings. They can be extensions
Expand Down
2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@ def read(fname):


setup(name="sagemaker",
version="1.0.4",
version="1.1.0",
description="Open source library for training and deploying models on Amazon SageMaker.",
packages=find_packages('src'),
package_dir={'': 'src'},
Expand Down
46 changes: 32 additions & 14 deletions src/sagemaker/fw_utils.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand DownExpand Up@@ -28,26 +28,28 @@
"""


def create_image_uri(region, framework, instance_type, py_version='py2', tag='1.0', account='520713654638'):
def create_image_uri(region, framework, instance_type, framework_version, py_version, account='520713654638'):
"""Return the ECR URI of an image.

Args:
region (str): AWS region where the image is uploaded.
framework (str): framework used by the image.
instance_type (str): EC2 instance type. Used to determine whether to use the CPU image or GPU image.
py_version (str): Python version. (default: 'py2')
tag (str): ECR image tag, which denotes the image version. (default: '1.0')
framework_version (str): The version of the framework.
py_version (str): Python version. One of 'py2' or 'py3'.
account (str): AWS account that contains the image. (default: '520713654638')

Returns:
str: The appropriate image URI based on the given parameters.
"""
device_version = 'cpu'
# Instance types that start with G, P are GPU powered: https://aws.amazon.com/ec2/instance-types/
device_type = 'cpu'
# Instance types that start with G, P are GPU powered: https://aws.amazon.com/sagemaker/pricing/instance-types/
if instance_type[3] in ['g', 'p']:
device_version = 'gpu'
device_type = 'gpu'

tag = "{}-{}-{}".format(framework_version, device_type, py_version)
return "{}.dkr.ecr.{}.amazonaws.com/sagemaker-{}-{}-{}:{}" \
.format(account, region, framework, py_version, device_version, tag)
.format(account, region, framework, py_version, device_type, tag)


def tar_and_upload_dir(session, bucket, s3_key_prefix, script, directory):
Expand DownExpand Up@@ -112,21 +114,37 @@ def framework_name_from_image(image_name):
tuple: A tuple containing:
str: The framework name
str: The Python version
str: The image tag
"""
# image name format: <account>.dkr.ecr.<region>.amazonaws.com/sagemaker-<framework>-<py_ver>-<device>:<tag>
sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*)(:)(.*)$')
sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*:.*)$')
sagemaker_match = sagemaker_pattern.match(image_name)
if sagemaker_match is None:
return None, None
return None, None, None
else:
# extract framework and python version
name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu)$')
# extract framework, python version and image tag
name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')

name_match = name_pattern.match(sagemaker_match.group(8))

if name_match is None:
return None, None
return None, None, None
else:
return name_match.group(1), name_match.group(2)
return name_match.group(1), name_match.group(2), name_match.group(4)


def framework_version_from_tag(image_tag):
"""Extract the framework version from the image tag.

Args:
image_tag (str): Image tag, which should take the form '<framework_version>-<device>-<py_version>'

Returns:
str: The framework version.
"""
tag_pattern = re.compile('^(.*)-(cpu|gpu)-(py2|py3)$')
tag_match = tag_pattern.match(image_tag)
return None if tag_match is None else tag_match.group(1)


def parse_s3_url(url):
Expand Down
4 changes: 1 addition & 3 deletions src/sagemaker/mxnet/__init__.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand All@@ -14,5 +14,3 @@
from sagemaker.mxnet.model import MXNetModel, MXNetPredictor

__all__ = [MXNet, MXNetModel, MXNetPredictor]

DOCKER_TAG = "1.0"
13 changes: 13 additions & 0 deletions src/sagemaker/mxnet/defaults.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
MXNET_VERSION = '1.0'

@winstonawswinstonawsFeb 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "DEFAULT_MXNET_VERSION"?

@nadiayanadiayaFeb 23, 2018•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaults.DEFAULT_MXNET_VERSION ? I think it's a little bit redundant.

@winstonawswinstonawsFeb 26, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, whoops.

Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

nadiaya
merged 5 commits into

from

Feb 27, 2018
 Merged


Support TensorFlow-1.5.0 and MXNet-1.0.0 #82
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
4fac858

Update .gitignore to ignore pytest_cache.


nadiaya Feb 22, 2018
c6b444a

Support TensorFlow-1.5.0 and MXNet-1.0.0


nadiaya Feb 22, 2018
d854806

Update and refactor tests. Add tests for fw_utils.


nadiaya Feb 24, 2018
dcfe663

Fix typo.


nadiaya Feb 26, 2018
aa64686

Merge branch 'master' into release-1.1


nadiaya Feb 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
.gitignore

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,3 +21,4 @@ doc/_build
**/.DS_Store
venv/
*~
.pytest_cache/

18 changes: 12 additions & 6 deletions
CHANGELOG.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,29 +2,35 @@
CHANGELOG
=========

1.1.0
=====

* feature: Estimators: add support for TensorFlow-1.5.0
@lukmislukmisFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update after master branch merge.
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.
* feature: Estimators: add support for MXNet-1.0.0

1.0.4
=====

* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
* feature: Documentation: Fix description of an argument of sagemaker.session.train
* feature: Documentation: Add FM and LDA to the documentation
* feature: Documentation: fix description of an argument of sagemaker.session.train
* feature: Documentation: add FM and LDA to the documentation
* feature: Estimators: add support for async fit
* bug-fix: Estimators: fix estimator role expansion

1.0.3
=====

* feature: Estimators: add support for Amazon LDA algorithm
* feature: Hyperparameters: Add data_type to hyperparameters
* feature: Documentation: Update TensorFlow examples following API change
* feature: Session: Support multi-part uploads
* feature: Hyperparameters: add data_type to hyperparameters
* feature: Documentation: update TensorFlow examples following API change
* feature: Session: support multi-part uploads


1.0.2
=====

* feature: Estimators: add support for Amazon FactorizationMachines algorithm
* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
* feature: Tests: add training failure tests for TF and MXNet
* feature: Documentation: show how to make predictions against existing endpoint
* feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
Expand Down

89 changes: 60 additions & 29 deletions
README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install

 git clone https://github.com/aws/sagemaker-python-sdk.git
 python setup.py sdist
 pip install dist/sagemaker-1.0.4.tar.gz
 pip install dist/sagemaker-1.1.0.tar.gz

Supported Python versions
~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -114,6 +114,8 @@ MXNet SageMaker Estimators

With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.0.0``, ``0.12.1``.

Training with MXNet
~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
Using MXNet and numpy
^^^^^^^^^^^^^^^^^^^^^

You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.

Running an MXNet training script in SageMaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script

SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write this to be more future-proof? Maybe something like, "will run using the latest supported version of MXNet by default. See (something) for our currently supported versions. You can choose older versions by..."
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we still would need to make an sdk change to change the default.

I believe it adds value to say what version we support by default.

Each Docker container has the following dependencies installed:
The Docker images have the following dependencies installed:

- Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
 the MXNet constructor.
- MXNet 0.12, built for either GPU or CPU, depending on the instance
 type for training or deploying.
- CUDA 9.0
- numpy 1.12
+-------------------------+--------------+-------------+
| Dependencies | MXNet 0.12.1 | MXNet 1.0.0 |
+-------------------------+--------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5|
+-------------------------+--------------+-------------+
| CUDA | 9.0 | 9.0 |
+-------------------------+--------------+-------------+
| numpy | 1.13.3 | 1.13.3 |
+-------------------------+--------------+-------------+

The Docker images extend Ubuntu 16.04.

You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.

TensorFlow SageMaker Estimators
-------------------------------

TensorFlow SageMaker Estimators allow you to run your own TensorFlow
training algorithms on SageMaker Learner, and to host your own TensorFlow
models on SageMaker Hosting.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.

Training with TensorFlow
~~~~~~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
version is **1.4.0**. This training script **must contain** the following functions:
versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about future-proofing.

- ``model_fn``: defines the model that will be trained.
- ``train_input_fn``: preprocess and load training data.
Expand DownExpand Up@@ -1419,27 +1428,49 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
SageMaker TensorFlow Docker containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
- awscli 1.12.1
- boto3 1.4.7
- botocore 1.5.92
- futures 2.2.0
- gevent 1.2.2
- grpcio 1.7.0
- numpy 1.13.3
- pandas 0.21.0
- protobuf 3.4.0
- requests 2.14.2
- scikit-learn 0.19.1
- scipy 1.0.0
- six 1.10.0
- sklearn 0.0
- tensorflow 1.4.0
- tensorflow-serving-api 1.4.0
- tensorflow-tensorboard 0.4.0rc2
The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:

+------------------------+------------------+------------------+
| Dependencies | tensorflow 1.4.1 | tensorflow 1.5.0 |
+------------------------+------------------+------------------+
| awscli | 1.12.1 | 1.14.35 |
+------------------------+------------------+------------------+
| boto3 | 1.4.7 | 1.5.22 |
+------------------------+------------------+------------------+
| botocore | 1.5.92 | 1.8.36 |
+------------------------+------------------+------------------+
| futures | 2.2.0 | 2.2.0 |
+------------------------+------------------+------------------+
| gevent | 1.2.2 | 1.2.2 |
+------------------------+------------------+------------------+
| grpcio | 1.7.0 | 1.9.0 |
+------------------------+------------------+------------------+
| numpy | 1.13.3 | 1.14.0 |
+------------------------+------------------+------------------+
| pandas | 0.21.0 | 0.22.0 |
+------------------------+------------------+------------------+
| protobuf | 3.4.0 | 3.5.1 |
+------------------------+------------------+------------------+
| requests | 2.14.2 | 2.18.4 |
+------------------------+------------------+------------------+
| scikit-learn | 0.19.1 | 0.19.1 |
+------------------------+------------------+------------------+
| scipy | 1.0.0 | 1.0.0 |
+------------------------+------------------+------------------+
| six | 1.10.0 | 1.10.0 |
+------------------------+------------------+------------------+
| sklearn | 0.0 | 0.0 |
+------------------------+------------------+------------------+
| tensorflow | 1.4.1 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-serving-api | 1.4.0 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-tensorboard | 0.4.0 | 1.5.1 |
+------------------------+------------------+------------------+

The Docker images extend Ubuntu 16.04.

You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.

AWS SageMaker Estimators
------------------------
Expand Down

2 changes: 1 addition & 1 deletion
doc/conf.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@ def __getattr__(cls, name):
 'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

version = '1.0.4'
version = '1.1.0'
project = u'sagemaker'

# Add any Sphinx extension module names here, as strings. They can be extensions
Expand Down

2 changes: 1 addition & 1 deletion
setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@ def read(fname):


setup(name="sagemaker",
 version="1.0.4",
 version="1.1.0",
 description="Open source library for training and deploying models on Amazon SageMaker.",
 packages=find_packages('src'),
 package_dir={'': 'src'},
Expand Down

46 changes: 32 additions & 14 deletions
src/sagemaker/fw_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand DownExpand Up@@ -28,26 +28,28 @@
"""


def create_image_uri(region, framework, instance_type, py_version='py2', tag='1.0', account='520713654638'):
def create_image_uri(region, framework, instance_type, framework_version, py_version, account='520713654638'):
 """Return the ECR URI of an image.

 Args:
 region (str): AWS region where the image is uploaded.
 framework (str): framework used by the image.
 instance_type (str): EC2 instance type. Used to determine whether to use the CPU image or GPU image.
py_version (str): Python version. (default: 'py2')
tag (str): ECR image tag, which denotes the image version. (default: '1.0')
framework_version (str): The version of the framework.
py_version (str): Python version. One of 'py2' or 'py3'.
 account (str): AWS account that contains the image. (default: '520713654638')

 Returns:
 str: The appropriate image URI based on the given parameters.
 """
device_version = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/ec2/instance-types/
device_type = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/sagemaker/pricing/instance-types/
 if instance_type[3] in ['g', 'p']:
 device_version = 'gpu'
 device_type = 'gpu'

 tag = "{}-{}-{}".format(framework_version, device_type, py_version)
 return "{}.dkr.ecr.{}.amazonaws.com/sagemaker-{}-{}-{}:{}" \
 .format(account, region, framework, py_version, device_version, tag)
 .format(account, region, framework, py_version, device_type, tag)


def tar_and_upload_dir(session, bucket, s3_key_prefix, script, directory):
Expand DownExpand Up@@ -112,21 +114,37 @@ def framework_name_from_image(image_name):
 tuple: A tuple containing:
 str: The framework name
 str: The Python version
 str: The image tag
 """
 # image name format: <account>.dkr.ecr.<region>.amazonaws.com/sagemaker-<framework>-<py_ver>-<device>:<tag>
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*)(:)(.*)$')
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*:.*)$')
 sagemaker_match = sagemaker_pattern.match(image_name)
 if sagemaker_match is None:
 return None, None
 return None, None, None
 else:
 # extract framework and python version
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu)$')
 # extract framework, python version and image tag
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')

 name_match = name_pattern.match(sagemaker_match.group(8))

 if name_match is None:
 return None, None
 return None, None, None
 else:
 return name_match.group(1), name_match.group(2)
 return name_match.group(1), name_match.group(2), name_match.group(4)


def framework_version_from_tag(image_tag):
 """Extract the framework version from the image tag.

 Args:
 image_tag (str): Image tag, which should take the form '<framework_version>-<device>-<py_version>'

 Returns:
 str: The framework version.
 """
 tag_pattern = re.compile('^(.*)-(cpu|gpu)-(py2|py3)$')
 tag_match = tag_pattern.match(image_tag)
 return None if tag_match is None else tag_match.group(1)


def parse_s3_url(url):
Expand Down

4 changes: 1 addition & 3 deletions
src/sagemaker/mxnet/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand All@@ -14,5 +14,3 @@
from sagemaker.mxnet.model import MXNetModel, MXNetPredictor

__all__ = [MXNet, MXNetModel, MXNetPredictor]

DOCKER_TAG = "1.0"

13 changes: 13 additions & 0 deletions
src/sagemaker/mxnet/defaults.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
MXNET_VERSION = '1.0'
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "DEFAULT_MXNET_VERSION"?
@nadiayanadiayaFeb 23, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaults.DEFAULT_MXNET_VERSION ? I think it's a little bit redundant.
@winstonawswinstonawsFeb 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, whoops.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

nadiaya
merged 5 commits into

from

Feb 27, 2018
 Merged


Support TensorFlow-1.5.0 and MXNet-1.0.0 #82
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
4fac858

Update .gitignore to ignore pytest_cache.


nadiaya Feb 22, 2018
c6b444a

Support TensorFlow-1.5.0 and MXNet-1.0.0


nadiaya Feb 22, 2018
d854806

Update and refactor tests. Add tests for fw_utils.


nadiaya Feb 24, 2018
dcfe663

Fix typo.


nadiaya Feb 26, 2018
aa64686

Merge branch 'master' into release-1.1


nadiaya Feb 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
.gitignore

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,3 +21,4 @@ doc/_build
**/.DS_Store
venv/
*~
.pytest_cache/

18 changes: 12 additions & 6 deletions
CHANGELOG.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,29 +2,35 @@
CHANGELOG
=========

1.1.0
=====

* feature: Estimators: add support for TensorFlow-1.5.0
@lukmislukmisFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update after master branch merge.
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.
* feature: Estimators: add support for MXNet-1.0.0

1.0.4
=====

* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
* feature: Documentation: Fix description of an argument of sagemaker.session.train
* feature: Documentation: Add FM and LDA to the documentation
* feature: Documentation: fix description of an argument of sagemaker.session.train
* feature: Documentation: add FM and LDA to the documentation
* feature: Estimators: add support for async fit
* bug-fix: Estimators: fix estimator role expansion

1.0.3
=====

* feature: Estimators: add support for Amazon LDA algorithm
* feature: Hyperparameters: Add data_type to hyperparameters
* feature: Documentation: Update TensorFlow examples following API change
* feature: Session: Support multi-part uploads
* feature: Hyperparameters: add data_type to hyperparameters
* feature: Documentation: update TensorFlow examples following API change
* feature: Session: support multi-part uploads


1.0.2
=====

* feature: Estimators: add support for Amazon FactorizationMachines algorithm
* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
* feature: Tests: add training failure tests for TF and MXNet
* feature: Documentation: show how to make predictions against existing endpoint
* feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
Expand Down

89 changes: 60 additions & 29 deletions
README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install

 git clone https://github.com/aws/sagemaker-python-sdk.git
 python setup.py sdist
 pip install dist/sagemaker-1.0.4.tar.gz
 pip install dist/sagemaker-1.1.0.tar.gz

Supported Python versions
~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -114,6 +114,8 @@ MXNet SageMaker Estimators

With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.0.0``, ``0.12.1``.

Training with MXNet
~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
Using MXNet and numpy
^^^^^^^^^^^^^^^^^^^^^

You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.

Running an MXNet training script in SageMaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script

SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write this to be more future-proof? Maybe something like, "will run using the latest supported version of MXNet by default. See (something) for our currently supported versions. You can choose older versions by..."
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we still would need to make an sdk change to change the default.

I believe it adds value to say what version we support by default.

Each Docker container has the following dependencies installed:
The Docker images have the following dependencies installed:

- Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
 the MXNet constructor.
- MXNet 0.12, built for either GPU or CPU, depending on the instance
 type for training or deploying.
- CUDA 9.0
- numpy 1.12
+-------------------------+--------------+-------------+
| Dependencies | MXNet 0.12.1 | MXNet 1.0.0 |
+-------------------------+--------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5|
+-------------------------+--------------+-------------+
| CUDA | 9.0 | 9.0 |
+-------------------------+--------------+-------------+
| numpy | 1.13.3 | 1.13.3 |
+-------------------------+--------------+-------------+

The Docker images extend Ubuntu 16.04.

You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.

TensorFlow SageMaker Estimators
-------------------------------

TensorFlow SageMaker Estimators allow you to run your own TensorFlow
training algorithms on SageMaker Learner, and to host your own TensorFlow
models on SageMaker Hosting.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.

Training with TensorFlow
~~~~~~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
version is **1.4.0**. This training script **must contain** the following functions:
versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about future-proofing.

- ``model_fn``: defines the model that will be trained.
- ``train_input_fn``: preprocess and load training data.
Expand DownExpand Up@@ -1419,27 +1428,49 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
SageMaker TensorFlow Docker containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
- awscli 1.12.1
- boto3 1.4.7
- botocore 1.5.92
- futures 2.2.0
- gevent 1.2.2
- grpcio 1.7.0
- numpy 1.13.3
- pandas 0.21.0
- protobuf 3.4.0
- requests 2.14.2
- scikit-learn 0.19.1
- scipy 1.0.0
- six 1.10.0
- sklearn 0.0
- tensorflow 1.4.0
- tensorflow-serving-api 1.4.0
- tensorflow-tensorboard 0.4.0rc2
The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:

+------------------------+------------------+------------------+
| Dependencies | tensorflow 1.4.1 | tensorflow 1.5.0 |
+------------------------+------------------+------------------+
| awscli | 1.12.1 | 1.14.35 |
+------------------------+------------------+------------------+
| boto3 | 1.4.7 | 1.5.22 |
+------------------------+------------------+------------------+
| botocore | 1.5.92 | 1.8.36 |
+------------------------+------------------+------------------+
| futures | 2.2.0 | 2.2.0 |
+------------------------+------------------+------------------+
| gevent | 1.2.2 | 1.2.2 |
+------------------------+------------------+------------------+
| grpcio | 1.7.0 | 1.9.0 |
+------------------------+------------------+------------------+
| numpy | 1.13.3 | 1.14.0 |
+------------------------+------------------+------------------+
| pandas | 0.21.0 | 0.22.0 |
+------------------------+------------------+------------------+
| protobuf | 3.4.0 | 3.5.1 |
+------------------------+------------------+------------------+
| requests | 2.14.2 | 2.18.4 |
+------------------------+------------------+------------------+
| scikit-learn | 0.19.1 | 0.19.1 |
+------------------------+------------------+------------------+
| scipy | 1.0.0 | 1.0.0 |
+------------------------+------------------+------------------+
| six | 1.10.0 | 1.10.0 |
+------------------------+------------------+------------------+
| sklearn | 0.0 | 0.0 |
+------------------------+------------------+------------------+
| tensorflow | 1.4.1 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-serving-api | 1.4.0 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-tensorboard | 0.4.0 | 1.5.1 |
+------------------------+------------------+------------------+

The Docker images extend Ubuntu 16.04.

You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.

AWS SageMaker Estimators
------------------------
Expand Down

2 changes: 1 addition & 1 deletion
doc/conf.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@ def __getattr__(cls, name):
 'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

version = '1.0.4'
version = '1.1.0'
project = u'sagemaker'

# Add any Sphinx extension module names here, as strings. They can be extensions
Expand Down

2 changes: 1 addition & 1 deletion
setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@ def read(fname):


setup(name="sagemaker",
 version="1.0.4",
 version="1.1.0",
 description="Open source library for training and deploying models on Amazon SageMaker.",
 packages=find_packages('src'),
 package_dir={'': 'src'},
Expand Down

46 changes: 32 additions & 14 deletions
src/sagemaker/fw_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand DownExpand Up@@ -28,26 +28,28 @@
"""


def create_image_uri(region, framework, instance_type, py_version='py2', tag='1.0', account='520713654638'):
def create_image_uri(region, framework, instance_type, framework_version, py_version, account='520713654638'):
 """Return the ECR URI of an image.

 Args:
 region (str): AWS region where the image is uploaded.
 framework (str): framework used by the image.
 instance_type (str): EC2 instance type. Used to determine whether to use the CPU image or GPU image.
py_version (str): Python version. (default: 'py2')
tag (str): ECR image tag, which denotes the image version. (default: '1.0')
framework_version (str): The version of the framework.
py_version (str): Python version. One of 'py2' or 'py3'.
 account (str): AWS account that contains the image. (default: '520713654638')

 Returns:
 str: The appropriate image URI based on the given parameters.
 """
device_version = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/ec2/instance-types/
device_type = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/sagemaker/pricing/instance-types/
 if instance_type[3] in ['g', 'p']:
 device_version = 'gpu'
 device_type = 'gpu'

 tag = "{}-{}-{}".format(framework_version, device_type, py_version)
 return "{}.dkr.ecr.{}.amazonaws.com/sagemaker-{}-{}-{}:{}" \
 .format(account, region, framework, py_version, device_version, tag)
 .format(account, region, framework, py_version, device_type, tag)


def tar_and_upload_dir(session, bucket, s3_key_prefix, script, directory):
Expand DownExpand Up@@ -112,21 +114,37 @@ def framework_name_from_image(image_name):
 tuple: A tuple containing:
 str: The framework name
 str: The Python version
 str: The image tag
 """
 # image name format: <account>.dkr.ecr.<region>.amazonaws.com/sagemaker-<framework>-<py_ver>-<device>:<tag>
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*)(:)(.*)$')
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*:.*)$')
 sagemaker_match = sagemaker_pattern.match(image_name)
 if sagemaker_match is None:
 return None, None
 return None, None, None
 else:
 # extract framework and python version
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu)$')
 # extract framework, python version and image tag
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')

 name_match = name_pattern.match(sagemaker_match.group(8))

 if name_match is None:
 return None, None
 return None, None, None
 else:
 return name_match.group(1), name_match.group(2)
 return name_match.group(1), name_match.group(2), name_match.group(4)


def framework_version_from_tag(image_tag):
 """Extract the framework version from the image tag.

 Args:
 image_tag (str): Image tag, which should take the form '<framework_version>-<device>-<py_version>'

 Returns:
 str: The framework version.
 """
 tag_pattern = re.compile('^(.*)-(cpu|gpu)-(py2|py3)$')
 tag_match = tag_pattern.match(image_tag)
 return None if tag_match is None else tag_match.group(1)


def parse_s3_url(url):
Expand Down

4 changes: 1 addition & 3 deletions
src/sagemaker/mxnet/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand All@@ -14,5 +14,3 @@
from sagemaker.mxnet.model import MXNetModel, MXNetPredictor

__all__ = [MXNet, MXNetModel, MXNetPredictor]

DOCKER_TAG = "1.0"

13 changes: 13 additions & 0 deletions
src/sagemaker/mxnet/defaults.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
MXNET_VERSION = '1.0'
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "DEFAULT_MXNET_VERSION"?
@nadiayanadiayaFeb 23, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaults.DEFAULT_MXNET_VERSION ? I think it's a little bit redundant.
@winstonawswinstonawsFeb 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, whoops.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

nadiaya
merged 5 commits into

from

Feb 27, 2018
 Merged


Support TensorFlow-1.5.0 and MXNet-1.0.0 #82
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
4fac858

Update .gitignore to ignore pytest_cache.


nadiaya Feb 22, 2018
c6b444a

Support TensorFlow-1.5.0 and MXNet-1.0.0


nadiaya Feb 22, 2018
d854806

Update and refactor tests. Add tests for fw_utils.


nadiaya Feb 24, 2018
dcfe663

Fix typo.


nadiaya Feb 26, 2018
aa64686

Merge branch 'master' into release-1.1


nadiaya Feb 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
.gitignore

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,3 +21,4 @@ doc/_build
**/.DS_Store
venv/
*~
.pytest_cache/

18 changes: 12 additions & 6 deletions
CHANGELOG.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,29 +2,35 @@
CHANGELOG
=========

1.1.0
=====

* feature: Estimators: add support for TensorFlow-1.5.0
@lukmislukmisFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update after master branch merge.
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.
* feature: Estimators: add support for MXNet-1.0.0

1.0.4
=====

* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
* feature: Documentation: Fix description of an argument of sagemaker.session.train
* feature: Documentation: Add FM and LDA to the documentation
* feature: Documentation: fix description of an argument of sagemaker.session.train
* feature: Documentation: add FM and LDA to the documentation
* feature: Estimators: add support for async fit
* bug-fix: Estimators: fix estimator role expansion

1.0.3
=====

* feature: Estimators: add support for Amazon LDA algorithm
* feature: Hyperparameters: Add data_type to hyperparameters
* feature: Documentation: Update TensorFlow examples following API change
* feature: Session: Support multi-part uploads
* feature: Hyperparameters: add data_type to hyperparameters
* feature: Documentation: update TensorFlow examples following API change
* feature: Session: support multi-part uploads


1.0.2
=====

* feature: Estimators: add support for Amazon FactorizationMachines algorithm
* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
* feature: Tests: add training failure tests for TF and MXNet
* feature: Documentation: show how to make predictions against existing endpoint
* feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
Expand Down

89 changes: 60 additions & 29 deletions
README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install

 git clone https://github.com/aws/sagemaker-python-sdk.git
 python setup.py sdist
 pip install dist/sagemaker-1.0.4.tar.gz
 pip install dist/sagemaker-1.1.0.tar.gz

Supported Python versions
~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -114,6 +114,8 @@ MXNet SageMaker Estimators

With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.0.0``, ``0.12.1``.

Training with MXNet
~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
Using MXNet and numpy
^^^^^^^^^^^^^^^^^^^^^

You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.

Running an MXNet training script in SageMaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script

SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write this to be more future-proof? Maybe something like, "will run using the latest supported version of MXNet by default. See (something) for our currently supported versions. You can choose older versions by..."
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we still would need to make an sdk change to change the default.

I believe it adds value to say what version we support by default.

Each Docker container has the following dependencies installed:
The Docker images have the following dependencies installed:

- Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
 the MXNet constructor.
- MXNet 0.12, built for either GPU or CPU, depending on the instance
 type for training or deploying.
- CUDA 9.0
- numpy 1.12
+-------------------------+--------------+-------------+
| Dependencies | MXNet 0.12.1 | MXNet 1.0.0 |
+-------------------------+--------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5|
+-------------------------+--------------+-------------+
| CUDA | 9.0 | 9.0 |
+-------------------------+--------------+-------------+
| numpy | 1.13.3 | 1.13.3 |
+-------------------------+--------------+-------------+

The Docker images extend Ubuntu 16.04.

You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.

TensorFlow SageMaker Estimators
-------------------------------

TensorFlow SageMaker Estimators allow you to run your own TensorFlow
training algorithms on SageMaker Learner, and to host your own TensorFlow
models on SageMaker Hosting.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.

Training with TensorFlow
~~~~~~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
version is **1.4.0**. This training script **must contain** the following functions:
versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about future-proofing.

- ``model_fn``: defines the model that will be trained.
- ``train_input_fn``: preprocess and load training data.
Expand DownExpand Up@@ -1419,27 +1428,49 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
SageMaker TensorFlow Docker containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
- awscli 1.12.1
- boto3 1.4.7
- botocore 1.5.92
- futures 2.2.0
- gevent 1.2.2
- grpcio 1.7.0
- numpy 1.13.3
- pandas 0.21.0
- protobuf 3.4.0
- requests 2.14.2
- scikit-learn 0.19.1
- scipy 1.0.0
- six 1.10.0
- sklearn 0.0
- tensorflow 1.4.0
- tensorflow-serving-api 1.4.0
- tensorflow-tensorboard 0.4.0rc2
The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:

+------------------------+------------------+------------------+
| Dependencies | tensorflow 1.4.1 | tensorflow 1.5.0 |
+------------------------+------------------+------------------+
| awscli | 1.12.1 | 1.14.35 |
+------------------------+------------------+------------------+
| boto3 | 1.4.7 | 1.5.22 |
+------------------------+------------------+------------------+
| botocore | 1.5.92 | 1.8.36 |
+------------------------+------------------+------------------+
| futures | 2.2.0 | 2.2.0 |
+------------------------+------------------+------------------+
| gevent | 1.2.2 | 1.2.2 |
+------------------------+------------------+------------------+
| grpcio | 1.7.0 | 1.9.0 |
+------------------------+------------------+------------------+
| numpy | 1.13.3 | 1.14.0 |
+------------------------+------------------+------------------+
| pandas | 0.21.0 | 0.22.0 |
+------------------------+------------------+------------------+
| protobuf | 3.4.0 | 3.5.1 |
+------------------------+------------------+------------------+
| requests | 2.14.2 | 2.18.4 |
+------------------------+------------------+------------------+
| scikit-learn | 0.19.1 | 0.19.1 |
+------------------------+------------------+------------------+
| scipy | 1.0.0 | 1.0.0 |
+------------------------+------------------+------------------+
| six | 1.10.0 | 1.10.0 |
+------------------------+------------------+------------------+
| sklearn | 0.0 | 0.0 |
+------------------------+------------------+------------------+
| tensorflow | 1.4.1 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-serving-api | 1.4.0 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-tensorboard | 0.4.0 | 1.5.1 |
+------------------------+------------------+------------------+

The Docker images extend Ubuntu 16.04.

You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.

AWS SageMaker Estimators
------------------------
Expand Down

2 changes: 1 addition & 1 deletion
doc/conf.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@ def __getattr__(cls, name):
 'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

version = '1.0.4'
version = '1.1.0'
project = u'sagemaker'

# Add any Sphinx extension module names here, as strings. They can be extensions
Expand Down

2 changes: 1 addition & 1 deletion
setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@ def read(fname):


setup(name="sagemaker",
 version="1.0.4",
 version="1.1.0",
 description="Open source library for training and deploying models on Amazon SageMaker.",
 packages=find_packages('src'),
 package_dir={'': 'src'},
Expand Down

46 changes: 32 additions & 14 deletions
src/sagemaker/fw_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand DownExpand Up@@ -28,26 +28,28 @@
"""


def create_image_uri(region, framework, instance_type, py_version='py2', tag='1.0', account='520713654638'):
def create_image_uri(region, framework, instance_type, framework_version, py_version, account='520713654638'):
 """Return the ECR URI of an image.

 Args:
 region (str): AWS region where the image is uploaded.
 framework (str): framework used by the image.
 instance_type (str): EC2 instance type. Used to determine whether to use the CPU image or GPU image.
py_version (str): Python version. (default: 'py2')
tag (str): ECR image tag, which denotes the image version. (default: '1.0')
framework_version (str): The version of the framework.
py_version (str): Python version. One of 'py2' or 'py3'.
 account (str): AWS account that contains the image. (default: '520713654638')

 Returns:
 str: The appropriate image URI based on the given parameters.
 """
device_version = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/ec2/instance-types/
device_type = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/sagemaker/pricing/instance-types/
 if instance_type[3] in ['g', 'p']:
 device_version = 'gpu'
 device_type = 'gpu'

 tag = "{}-{}-{}".format(framework_version, device_type, py_version)
 return "{}.dkr.ecr.{}.amazonaws.com/sagemaker-{}-{}-{}:{}" \
 .format(account, region, framework, py_version, device_version, tag)
 .format(account, region, framework, py_version, device_type, tag)


def tar_and_upload_dir(session, bucket, s3_key_prefix, script, directory):
Expand DownExpand Up@@ -112,21 +114,37 @@ def framework_name_from_image(image_name):
 tuple: A tuple containing:
 str: The framework name
 str: The Python version
 str: The image tag
 """
 # image name format: <account>.dkr.ecr.<region>.amazonaws.com/sagemaker-<framework>-<py_ver>-<device>:<tag>
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*)(:)(.*)$')
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*:.*)$')
 sagemaker_match = sagemaker_pattern.match(image_name)
 if sagemaker_match is None:
 return None, None
 return None, None, None
 else:
 # extract framework and python version
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu)$')
 # extract framework, python version and image tag
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')

 name_match = name_pattern.match(sagemaker_match.group(8))

 if name_match is None:
 return None, None
 return None, None, None
 else:
 return name_match.group(1), name_match.group(2)
 return name_match.group(1), name_match.group(2), name_match.group(4)


def framework_version_from_tag(image_tag):
 """Extract the framework version from the image tag.

 Args:
 image_tag (str): Image tag, which should take the form '<framework_version>-<device>-<py_version>'

 Returns:
 str: The framework version.
 """
 tag_pattern = re.compile('^(.*)-(cpu|gpu)-(py2|py3)$')
 tag_match = tag_pattern.match(image_tag)
 return None if tag_match is None else tag_match.group(1)


def parse_s3_url(url):
Expand Down

4 changes: 1 addition & 3 deletions
src/sagemaker/mxnet/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand All@@ -14,5 +14,3 @@
from sagemaker.mxnet.model import MXNetModel, MXNetPredictor

__all__ = [MXNet, MXNetModel, MXNetPredictor]

DOCKER_TAG = "1.0"

13 changes: 13 additions & 0 deletions
src/sagemaker/mxnet/defaults.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
MXNET_VERSION = '1.0'
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "DEFAULT_MXNET_VERSION"?
@nadiayanadiayaFeb 23, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaults.DEFAULT_MXNET_VERSION ? I think it's a little bit redundant.
@winstonawswinstonawsFeb 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, whoops.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

nadiaya
merged 5 commits into

from

Feb 27, 2018
 Merged


Support TensorFlow-1.5.0 and MXNet-1.0.0 #82
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
4fac858

Update .gitignore to ignore pytest_cache.


nadiaya Feb 22, 2018
c6b444a

Support TensorFlow-1.5.0 and MXNet-1.0.0


nadiaya Feb 22, 2018
d854806

Update and refactor tests. Add tests for fw_utils.


nadiaya Feb 24, 2018
dcfe663

Fix typo.


nadiaya Feb 26, 2018
aa64686

Merge branch 'master' into release-1.1


nadiaya Feb 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
.gitignore

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,3 +21,4 @@ doc/_build
**/.DS_Store
venv/
*~
.pytest_cache/

18 changes: 12 additions & 6 deletions
CHANGELOG.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,29 +2,35 @@
CHANGELOG
=========

1.1.0
=====

* feature: Estimators: add support for TensorFlow-1.5.0
@lukmislukmisFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update after master branch merge.
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.
* feature: Estimators: add support for MXNet-1.0.0

1.0.4
=====

* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
* feature: Documentation: Fix description of an argument of sagemaker.session.train
* feature: Documentation: Add FM and LDA to the documentation
* feature: Documentation: fix description of an argument of sagemaker.session.train
* feature: Documentation: add FM and LDA to the documentation
* feature: Estimators: add support for async fit
* bug-fix: Estimators: fix estimator role expansion

1.0.3
=====

* feature: Estimators: add support for Amazon LDA algorithm
* feature: Hyperparameters: Add data_type to hyperparameters
* feature: Documentation: Update TensorFlow examples following API change
* feature: Session: Support multi-part uploads
* feature: Hyperparameters: add data_type to hyperparameters
* feature: Documentation: update TensorFlow examples following API change
* feature: Session: support multi-part uploads


1.0.2
=====

* feature: Estimators: add support for Amazon FactorizationMachines algorithm
* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
* feature: Tests: add training failure tests for TF and MXNet
* feature: Documentation: show how to make predictions against existing endpoint
* feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
Expand Down

89 changes: 60 additions & 29 deletions
README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install

 git clone https://github.com/aws/sagemaker-python-sdk.git
 python setup.py sdist
 pip install dist/sagemaker-1.0.4.tar.gz
 pip install dist/sagemaker-1.1.0.tar.gz

Supported Python versions
~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -114,6 +114,8 @@ MXNet SageMaker Estimators

With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.0.0``, ``0.12.1``.

Training with MXNet
~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
Using MXNet and numpy
^^^^^^^^^^^^^^^^^^^^^

You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.

Running an MXNet training script in SageMaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script

SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write this to be more future-proof? Maybe something like, "will run using the latest supported version of MXNet by default. See (something) for our currently supported versions. You can choose older versions by..."
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we still would need to make an sdk change to change the default.

I believe it adds value to say what version we support by default.

Each Docker container has the following dependencies installed:
The Docker images have the following dependencies installed:

- Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
 the MXNet constructor.
- MXNet 0.12, built for either GPU or CPU, depending on the instance
 type for training or deploying.
- CUDA 9.0
- numpy 1.12
+-------------------------+--------------+-------------+
| Dependencies | MXNet 0.12.1 | MXNet 1.0.0 |
+-------------------------+--------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5|
+-------------------------+--------------+-------------+
| CUDA | 9.0 | 9.0 |
+-------------------------+--------------+-------------+
| numpy | 1.13.3 | 1.13.3 |
+-------------------------+--------------+-------------+

The Docker images extend Ubuntu 16.04.

You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.

TensorFlow SageMaker Estimators
-------------------------------

TensorFlow SageMaker Estimators allow you to run your own TensorFlow
training algorithms on SageMaker Learner, and to host your own TensorFlow
models on SageMaker Hosting.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.

Training with TensorFlow
~~~~~~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
version is **1.4.0**. This training script **must contain** the following functions:
versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about future-proofing.

- ``model_fn``: defines the model that will be trained.
- ``train_input_fn``: preprocess and load training data.
Expand DownExpand Up@@ -1419,27 +1428,49 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
SageMaker TensorFlow Docker containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
- awscli 1.12.1
- boto3 1.4.7
- botocore 1.5.92
- futures 2.2.0
- gevent 1.2.2
- grpcio 1.7.0
- numpy 1.13.3
- pandas 0.21.0
- protobuf 3.4.0
- requests 2.14.2
- scikit-learn 0.19.1
- scipy 1.0.0
- six 1.10.0
- sklearn 0.0
- tensorflow 1.4.0
- tensorflow-serving-api 1.4.0
- tensorflow-tensorboard 0.4.0rc2
The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:

+------------------------+------------------+------------------+
| Dependencies | tensorflow 1.4.1 | tensorflow 1.5.0 |
+------------------------+------------------+------------------+
| awscli | 1.12.1 | 1.14.35 |
+------------------------+------------------+------------------+
| boto3 | 1.4.7 | 1.5.22 |
+------------------------+------------------+------------------+
| botocore | 1.5.92 | 1.8.36 |
+------------------------+------------------+------------------+
| futures | 2.2.0 | 2.2.0 |
+------------------------+------------------+------------------+
| gevent | 1.2.2 | 1.2.2 |
+------------------------+------------------+------------------+
| grpcio | 1.7.0 | 1.9.0 |
+------------------------+------------------+------------------+
| numpy | 1.13.3 | 1.14.0 |
+------------------------+------------------+------------------+
| pandas | 0.21.0 | 0.22.0 |
+------------------------+------------------+------------------+
| protobuf | 3.4.0 | 3.5.1 |
+------------------------+------------------+------------------+
| requests | 2.14.2 | 2.18.4 |
+------------------------+------------------+------------------+
| scikit-learn | 0.19.1 | 0.19.1 |
+------------------------+------------------+------------------+
| scipy | 1.0.0 | 1.0.0 |
+------------------------+------------------+------------------+
| six | 1.10.0 | 1.10.0 |
+------------------------+------------------+------------------+
| sklearn | 0.0 | 0.0 |
+------------------------+------------------+------------------+
| tensorflow | 1.4.1 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-serving-api | 1.4.0 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-tensorboard | 0.4.0 | 1.5.1 |
+------------------------+------------------+------------------+

The Docker images extend Ubuntu 16.04.

You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.

AWS SageMaker Estimators
------------------------
Expand Down

2 changes: 1 addition & 1 deletion
doc/conf.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@ def __getattr__(cls, name):
 'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

version = '1.0.4'
version = '1.1.0'
project = u'sagemaker'

# Add any Sphinx extension module names here, as strings. They can be extensions
Expand Down

2 changes: 1 addition & 1 deletion
setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@ def read(fname):


setup(name="sagemaker",
 version="1.0.4",
 version="1.1.0",
 description="Open source library for training and deploying models on Amazon SageMaker.",
 packages=find_packages('src'),
 package_dir={'': 'src'},
Expand Down

46 changes: 32 additions & 14 deletions
src/sagemaker/fw_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand DownExpand Up@@ -28,26 +28,28 @@
"""


def create_image_uri(region, framework, instance_type, py_version='py2', tag='1.0', account='520713654638'):
def create_image_uri(region, framework, instance_type, framework_version, py_version, account='520713654638'):
 """Return the ECR URI of an image.

 Args:
 region (str): AWS region where the image is uploaded.
 framework (str): framework used by the image.
 instance_type (str): EC2 instance type. Used to determine whether to use the CPU image or GPU image.
py_version (str): Python version. (default: 'py2')
tag (str): ECR image tag, which denotes the image version. (default: '1.0')
framework_version (str): The version of the framework.
py_version (str): Python version. One of 'py2' or 'py3'.
 account (str): AWS account that contains the image. (default: '520713654638')

 Returns:
 str: The appropriate image URI based on the given parameters.
 """
device_version = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/ec2/instance-types/
device_type = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/sagemaker/pricing/instance-types/
 if instance_type[3] in ['g', 'p']:
 device_version = 'gpu'
 device_type = 'gpu'

 tag = "{}-{}-{}".format(framework_version, device_type, py_version)
 return "{}.dkr.ecr.{}.amazonaws.com/sagemaker-{}-{}-{}:{}" \
 .format(account, region, framework, py_version, device_version, tag)
 .format(account, region, framework, py_version, device_type, tag)


def tar_and_upload_dir(session, bucket, s3_key_prefix, script, directory):
Expand DownExpand Up@@ -112,21 +114,37 @@ def framework_name_from_image(image_name):
 tuple: A tuple containing:
 str: The framework name
 str: The Python version
 str: The image tag
 """
 # image name format: <account>.dkr.ecr.<region>.amazonaws.com/sagemaker-<framework>-<py_ver>-<device>:<tag>
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*)(:)(.*)$')
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*:.*)$')
 sagemaker_match = sagemaker_pattern.match(image_name)
 if sagemaker_match is None:
 return None, None
 return None, None, None
 else:
 # extract framework and python version
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu)$')
 # extract framework, python version and image tag
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')

 name_match = name_pattern.match(sagemaker_match.group(8))

 if name_match is None:
 return None, None
 return None, None, None
 else:
 return name_match.group(1), name_match.group(2)
 return name_match.group(1), name_match.group(2), name_match.group(4)


def framework_version_from_tag(image_tag):
 """Extract the framework version from the image tag.

 Args:
 image_tag (str): Image tag, which should take the form '<framework_version>-<device>-<py_version>'

 Returns:
 str: The framework version.
 """
 tag_pattern = re.compile('^(.*)-(cpu|gpu)-(py2|py3)$')
 tag_match = tag_pattern.match(image_tag)
 return None if tag_match is None else tag_match.group(1)


def parse_s3_url(url):
Expand Down

4 changes: 1 addition & 3 deletions
src/sagemaker/mxnet/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand All@@ -14,5 +14,3 @@
from sagemaker.mxnet.model import MXNetModel, MXNetPredictor

__all__ = [MXNet, MXNetModel, MXNetPredictor]

DOCKER_TAG = "1.0"

13 changes: 13 additions & 0 deletions
src/sagemaker/mxnet/defaults.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
MXNET_VERSION = '1.0'
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "DEFAULT_MXNET_VERSION"?
@nadiayanadiayaFeb 23, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaults.DEFAULT_MXNET_VERSION ? I think it's a little bit redundant.
@winstonawswinstonawsFeb 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, whoops.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

nadiaya
merged 5 commits into

from

Feb 27, 2018
 Merged


Support TensorFlow-1.5.0 and MXNet-1.0.0 #82
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
4fac858

Update .gitignore to ignore pytest_cache.


nadiaya Feb 22, 2018
c6b444a

Support TensorFlow-1.5.0 and MXNet-1.0.0


nadiaya Feb 22, 2018
d854806

Update and refactor tests. Add tests for fw_utils.


nadiaya Feb 24, 2018
dcfe663

Fix typo.


nadiaya Feb 26, 2018
aa64686

Merge branch 'master' into release-1.1


nadiaya Feb 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
.gitignore

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,3 +21,4 @@ doc/_build
**/.DS_Store
venv/
*~
.pytest_cache/

18 changes: 12 additions & 6 deletions
CHANGELOG.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,29 +2,35 @@
CHANGELOG
=========

1.1.0
=====

* feature: Estimators: add support for TensorFlow-1.5.0
@lukmislukmisFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update after master branch merge.
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.
* feature: Estimators: add support for MXNet-1.0.0

1.0.4
=====

* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
* feature: Documentation: Fix description of an argument of sagemaker.session.train
* feature: Documentation: Add FM and LDA to the documentation
* feature: Documentation: fix description of an argument of sagemaker.session.train
* feature: Documentation: add FM and LDA to the documentation
* feature: Estimators: add support for async fit
* bug-fix: Estimators: fix estimator role expansion

1.0.3
=====

* feature: Estimators: add support for Amazon LDA algorithm
* feature: Hyperparameters: Add data_type to hyperparameters
* feature: Documentation: Update TensorFlow examples following API change
* feature: Session: Support multi-part uploads
* feature: Hyperparameters: add data_type to hyperparameters
* feature: Documentation: update TensorFlow examples following API change
* feature: Session: support multi-part uploads


1.0.2
=====

* feature: Estimators: add support for Amazon FactorizationMachines algorithm
* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
* feature: Tests: add training failure tests for TF and MXNet
* feature: Documentation: show how to make predictions against existing endpoint
* feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
Expand Down

89 changes: 60 additions & 29 deletions
README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install

 git clone https://github.com/aws/sagemaker-python-sdk.git
 python setup.py sdist
 pip install dist/sagemaker-1.0.4.tar.gz
 pip install dist/sagemaker-1.1.0.tar.gz

Supported Python versions
~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -114,6 +114,8 @@ MXNet SageMaker Estimators

With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.0.0``, ``0.12.1``.

Training with MXNet
~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
Using MXNet and numpy
^^^^^^^^^^^^^^^^^^^^^

You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.

Running an MXNet training script in SageMaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script

SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write this to be more future-proof? Maybe something like, "will run using the latest supported version of MXNet by default. See (something) for our currently supported versions. You can choose older versions by..."
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we still would need to make an sdk change to change the default.

I believe it adds value to say what version we support by default.

Each Docker container has the following dependencies installed:
The Docker images have the following dependencies installed:

- Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
 the MXNet constructor.
- MXNet 0.12, built for either GPU or CPU, depending on the instance
 type for training or deploying.
- CUDA 9.0
- numpy 1.12
+-------------------------+--------------+-------------+
| Dependencies | MXNet 0.12.1 | MXNet 1.0.0 |
+-------------------------+--------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5|
+-------------------------+--------------+-------------+
| CUDA | 9.0 | 9.0 |
+-------------------------+--------------+-------------+
| numpy | 1.13.3 | 1.13.3 |
+-------------------------+--------------+-------------+

The Docker images extend Ubuntu 16.04.

You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.

TensorFlow SageMaker Estimators
-------------------------------

TensorFlow SageMaker Estimators allow you to run your own TensorFlow
training algorithms on SageMaker Learner, and to host your own TensorFlow
models on SageMaker Hosting.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.

Training with TensorFlow
~~~~~~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
version is **1.4.0**. This training script **must contain** the following functions:
versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about future-proofing.

- ``model_fn``: defines the model that will be trained.
- ``train_input_fn``: preprocess and load training data.
Expand DownExpand Up@@ -1419,27 +1428,49 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
SageMaker TensorFlow Docker containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
- awscli 1.12.1
- boto3 1.4.7
- botocore 1.5.92
- futures 2.2.0
- gevent 1.2.2
- grpcio 1.7.0
- numpy 1.13.3
- pandas 0.21.0
- protobuf 3.4.0
- requests 2.14.2
- scikit-learn 0.19.1
- scipy 1.0.0
- six 1.10.0
- sklearn 0.0
- tensorflow 1.4.0
- tensorflow-serving-api 1.4.0
- tensorflow-tensorboard 0.4.0rc2
The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:

+------------------------+------------------+------------------+
| Dependencies | tensorflow 1.4.1 | tensorflow 1.5.0 |
+------------------------+------------------+------------------+
| awscli | 1.12.1 | 1.14.35 |
+------------------------+------------------+------------------+
| boto3 | 1.4.7 | 1.5.22 |
+------------------------+------------------+------------------+
| botocore | 1.5.92 | 1.8.36 |
+------------------------+------------------+------------------+
| futures | 2.2.0 | 2.2.0 |
+------------------------+------------------+------------------+
| gevent | 1.2.2 | 1.2.2 |
+------------------------+------------------+------------------+
| grpcio | 1.7.0 | 1.9.0 |
+------------------------+------------------+------------------+
| numpy | 1.13.3 | 1.14.0 |
+------------------------+------------------+------------------+
| pandas | 0.21.0 | 0.22.0 |
+------------------------+------------------+------------------+
| protobuf | 3.4.0 | 3.5.1 |
+------------------------+------------------+------------------+
| requests | 2.14.2 | 2.18.4 |
+------------------------+------------------+------------------+
| scikit-learn | 0.19.1 | 0.19.1 |
+------------------------+------------------+------------------+
| scipy | 1.0.0 | 1.0.0 |
+------------------------+------------------+------------------+
| six | 1.10.0 | 1.10.0 |
+------------------------+------------------+------------------+
| sklearn | 0.0 | 0.0 |
+------------------------+------------------+------------------+
| tensorflow | 1.4.1 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-serving-api | 1.4.0 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-tensorboard | 0.4.0 | 1.5.1 |
+------------------------+------------------+------------------+

The Docker images extend Ubuntu 16.04.

You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.

AWS SageMaker Estimators
------------------------
Expand Down

2 changes: 1 addition & 1 deletion
doc/conf.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@ def __getattr__(cls, name):
 'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

version = '1.0.4'
version = '1.1.0'
project = u'sagemaker'

# Add any Sphinx extension module names here, as strings. They can be extensions
Expand Down

2 changes: 1 addition & 1 deletion
setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@ def read(fname):


setup(name="sagemaker",
 version="1.0.4",
 version="1.1.0",
 description="Open source library for training and deploying models on Amazon SageMaker.",
 packages=find_packages('src'),
 package_dir={'': 'src'},
Expand Down

46 changes: 32 additions & 14 deletions
src/sagemaker/fw_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand DownExpand Up@@ -28,26 +28,28 @@
"""


def create_image_uri(region, framework, instance_type, py_version='py2', tag='1.0', account='520713654638'):
def create_image_uri(region, framework, instance_type, framework_version, py_version, account='520713654638'):
 """Return the ECR URI of an image.

 Args:
 region (str): AWS region where the image is uploaded.
 framework (str): framework used by the image.
 instance_type (str): EC2 instance type. Used to determine whether to use the CPU image or GPU image.
py_version (str): Python version. (default: 'py2')
tag (str): ECR image tag, which denotes the image version. (default: '1.0')
framework_version (str): The version of the framework.
py_version (str): Python version. One of 'py2' or 'py3'.
 account (str): AWS account that contains the image. (default: '520713654638')

 Returns:
 str: The appropriate image URI based on the given parameters.
 """
device_version = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/ec2/instance-types/
device_type = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/sagemaker/pricing/instance-types/
 if instance_type[3] in ['g', 'p']:
 device_version = 'gpu'
 device_type = 'gpu'

 tag = "{}-{}-{}".format(framework_version, device_type, py_version)
 return "{}.dkr.ecr.{}.amazonaws.com/sagemaker-{}-{}-{}:{}" \
 .format(account, region, framework, py_version, device_version, tag)
 .format(account, region, framework, py_version, device_type, tag)


def tar_and_upload_dir(session, bucket, s3_key_prefix, script, directory):
Expand DownExpand Up@@ -112,21 +114,37 @@ def framework_name_from_image(image_name):
 tuple: A tuple containing:
 str: The framework name
 str: The Python version
 str: The image tag
 """
 # image name format: <account>.dkr.ecr.<region>.amazonaws.com/sagemaker-<framework>-<py_ver>-<device>:<tag>
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*)(:)(.*)$')
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*:.*)$')
 sagemaker_match = sagemaker_pattern.match(image_name)
 if sagemaker_match is None:
 return None, None
 return None, None, None
 else:
 # extract framework and python version
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu)$')
 # extract framework, python version and image tag
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')

 name_match = name_pattern.match(sagemaker_match.group(8))

 if name_match is None:
 return None, None
 return None, None, None
 else:
 return name_match.group(1), name_match.group(2)
 return name_match.group(1), name_match.group(2), name_match.group(4)


def framework_version_from_tag(image_tag):
 """Extract the framework version from the image tag.

 Args:
 image_tag (str): Image tag, which should take the form '<framework_version>-<device>-<py_version>'

 Returns:
 str: The framework version.
 """
 tag_pattern = re.compile('^(.*)-(cpu|gpu)-(py2|py3)$')
 tag_match = tag_pattern.match(image_tag)
 return None if tag_match is None else tag_match.group(1)


def parse_s3_url(url):
Expand Down

4 changes: 1 addition & 3 deletions
src/sagemaker/mxnet/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand All@@ -14,5 +14,3 @@
from sagemaker.mxnet.model import MXNetModel, MXNetPredictor

__all__ = [MXNet, MXNetModel, MXNetPredictor]

DOCKER_TAG = "1.0"

13 changes: 13 additions & 0 deletions
src/sagemaker/mxnet/defaults.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
MXNET_VERSION = '1.0'
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "DEFAULT_MXNET_VERSION"?
@nadiayanadiayaFeb 23, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaults.DEFAULT_MXNET_VERSION ? I think it's a little bit redundant.
@winstonawswinstonawsFeb 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, whoops.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

nadiaya
merged 5 commits into

from

Feb 27, 2018
 Merged


Support TensorFlow-1.5.0 and MXNet-1.0.0 #82
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
4fac858

Update .gitignore to ignore pytest_cache.


nadiaya Feb 22, 2018
c6b444a

Support TensorFlow-1.5.0 and MXNet-1.0.0


nadiaya Feb 22, 2018
d854806

Update and refactor tests. Add tests for fw_utils.


nadiaya Feb 24, 2018
dcfe663

Fix typo.


nadiaya Feb 26, 2018
aa64686

Merge branch 'master' into release-1.1


nadiaya Feb 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
.gitignore

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,3 +21,4 @@ doc/_build
**/.DS_Store
venv/
*~
.pytest_cache/

18 changes: 12 additions & 6 deletions
CHANGELOG.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,29 +2,35 @@
CHANGELOG
=========

1.1.0
=====

* feature: Estimators: add support for TensorFlow-1.5.0
@lukmislukmisFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update after master branch merge.
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.
* feature: Estimators: add support for MXNet-1.0.0

1.0.4
=====

* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
* feature: Documentation: Fix description of an argument of sagemaker.session.train
* feature: Documentation: Add FM and LDA to the documentation
* feature: Documentation: fix description of an argument of sagemaker.session.train
* feature: Documentation: add FM and LDA to the documentation
* feature: Estimators: add support for async fit
* bug-fix: Estimators: fix estimator role expansion

1.0.3
=====

* feature: Estimators: add support for Amazon LDA algorithm
* feature: Hyperparameters: Add data_type to hyperparameters
* feature: Documentation: Update TensorFlow examples following API change
* feature: Session: Support multi-part uploads
* feature: Hyperparameters: add data_type to hyperparameters
* feature: Documentation: update TensorFlow examples following API change
* feature: Session: support multi-part uploads


1.0.2
=====

* feature: Estimators: add support for Amazon FactorizationMachines algorithm
* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
* feature: Tests: add training failure tests for TF and MXNet
* feature: Documentation: show how to make predictions against existing endpoint
* feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
Expand Down

89 changes: 60 additions & 29 deletions
README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install

 git clone https://github.com/aws/sagemaker-python-sdk.git
 python setup.py sdist
 pip install dist/sagemaker-1.0.4.tar.gz
 pip install dist/sagemaker-1.1.0.tar.gz

Supported Python versions
~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -114,6 +114,8 @@ MXNet SageMaker Estimators

With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.0.0``, ``0.12.1``.

Training with MXNet
~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
Using MXNet and numpy
^^^^^^^^^^^^^^^^^^^^^

You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.

Running an MXNet training script in SageMaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script

SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write this to be more future-proof? Maybe something like, "will run using the latest supported version of MXNet by default. See (something) for our currently supported versions. You can choose older versions by..."
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we still would need to make an sdk change to change the default.

I believe it adds value to say what version we support by default.

Each Docker container has the following dependencies installed:
The Docker images have the following dependencies installed:

- Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
 the MXNet constructor.
- MXNet 0.12, built for either GPU or CPU, depending on the instance
 type for training or deploying.
- CUDA 9.0
- numpy 1.12
+-------------------------+--------------+-------------+
| Dependencies | MXNet 0.12.1 | MXNet 1.0.0 |
+-------------------------+--------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5|
+-------------------------+--------------+-------------+
| CUDA | 9.0 | 9.0 |
+-------------------------+--------------+-------------+
| numpy | 1.13.3 | 1.13.3 |
+-------------------------+--------------+-------------+

The Docker images extend Ubuntu 16.04.

You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.

TensorFlow SageMaker Estimators
-------------------------------

TensorFlow SageMaker Estimators allow you to run your own TensorFlow
training algorithms on SageMaker Learner, and to host your own TensorFlow
models on SageMaker Hosting.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.

Training with TensorFlow
~~~~~~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
version is **1.4.0**. This training script **must contain** the following functions:
versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about future-proofing.

- ``model_fn``: defines the model that will be trained.
- ``train_input_fn``: preprocess and load training data.
Expand DownExpand Up@@ -1419,27 +1428,49 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
SageMaker TensorFlow Docker containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
- awscli 1.12.1
- boto3 1.4.7
- botocore 1.5.92
- futures 2.2.0
- gevent 1.2.2
- grpcio 1.7.0
- numpy 1.13.3
- pandas 0.21.0
- protobuf 3.4.0
- requests 2.14.2
- scikit-learn 0.19.1
- scipy 1.0.0
- six 1.10.0
- sklearn 0.0
- tensorflow 1.4.0
- tensorflow-serving-api 1.4.0
- tensorflow-tensorboard 0.4.0rc2
The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:

+------------------------+------------------+------------------+
| Dependencies | tensorflow 1.4.1 | tensorflow 1.5.0 |
+------------------------+------------------+------------------+
| awscli | 1.12.1 | 1.14.35 |
+------------------------+------------------+------------------+
| boto3 | 1.4.7 | 1.5.22 |
+------------------------+------------------+------------------+
| botocore | 1.5.92 | 1.8.36 |
+------------------------+------------------+------------------+
| futures | 2.2.0 | 2.2.0 |
+------------------------+------------------+------------------+
| gevent | 1.2.2 | 1.2.2 |
+------------------------+------------------+------------------+
| grpcio | 1.7.0 | 1.9.0 |
+------------------------+------------------+------------------+
| numpy | 1.13.3 | 1.14.0 |
+------------------------+------------------+------------------+
| pandas | 0.21.0 | 0.22.0 |
+------------------------+------------------+------------------+
| protobuf | 3.4.0 | 3.5.1 |
+------------------------+------------------+------------------+
| requests | 2.14.2 | 2.18.4 |
+------------------------+------------------+------------------+
| scikit-learn | 0.19.1 | 0.19.1 |
+------------------------+------------------+------------------+
| scipy | 1.0.0 | 1.0.0 |
+------------------------+------------------+------------------+
| six | 1.10.0 | 1.10.0 |
+------------------------+------------------+------------------+
| sklearn | 0.0 | 0.0 |
+------------------------+------------------+------------------+
| tensorflow | 1.4.1 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-serving-api | 1.4.0 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-tensorboard | 0.4.0 | 1.5.1 |
+------------------------+------------------+------------------+

The Docker images extend Ubuntu 16.04.

You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.

AWS SageMaker Estimators
------------------------
Expand Down

2 changes: 1 addition & 1 deletion
doc/conf.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@ def __getattr__(cls, name):
 'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

version = '1.0.4'
version = '1.1.0'
project = u'sagemaker'

# Add any Sphinx extension module names here, as strings. They can be extensions
Expand Down

2 changes: 1 addition & 1 deletion
setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@ def read(fname):


setup(name="sagemaker",
 version="1.0.4",
 version="1.1.0",
 description="Open source library for training and deploying models on Amazon SageMaker.",
 packages=find_packages('src'),
 package_dir={'': 'src'},
Expand Down

46 changes: 32 additions & 14 deletions
src/sagemaker/fw_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand DownExpand Up@@ -28,26 +28,28 @@
"""


def create_image_uri(region, framework, instance_type, py_version='py2', tag='1.0', account='520713654638'):
def create_image_uri(region, framework, instance_type, framework_version, py_version, account='520713654638'):
 """Return the ECR URI of an image.

 Args:
 region (str): AWS region where the image is uploaded.
 framework (str): framework used by the image.
 instance_type (str): EC2 instance type. Used to determine whether to use the CPU image or GPU image.
py_version (str): Python version. (default: 'py2')
tag (str): ECR image tag, which denotes the image version. (default: '1.0')
framework_version (str): The version of the framework.
py_version (str): Python version. One of 'py2' or 'py3'.
 account (str): AWS account that contains the image. (default: '520713654638')

 Returns:
 str: The appropriate image URI based on the given parameters.
 """
device_version = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/ec2/instance-types/
device_type = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/sagemaker/pricing/instance-types/
 if instance_type[3] in ['g', 'p']:
 device_version = 'gpu'
 device_type = 'gpu'

 tag = "{}-{}-{}".format(framework_version, device_type, py_version)
 return "{}.dkr.ecr.{}.amazonaws.com/sagemaker-{}-{}-{}:{}" \
 .format(account, region, framework, py_version, device_version, tag)
 .format(account, region, framework, py_version, device_type, tag)


def tar_and_upload_dir(session, bucket, s3_key_prefix, script, directory):
Expand DownExpand Up@@ -112,21 +114,37 @@ def framework_name_from_image(image_name):
 tuple: A tuple containing:
 str: The framework name
 str: The Python version
 str: The image tag
 """
 # image name format: <account>.dkr.ecr.<region>.amazonaws.com/sagemaker-<framework>-<py_ver>-<device>:<tag>
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*)(:)(.*)$')
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*:.*)$')
 sagemaker_match = sagemaker_pattern.match(image_name)
 if sagemaker_match is None:
 return None, None
 return None, None, None
 else:
 # extract framework and python version
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu)$')
 # extract framework, python version and image tag
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')

 name_match = name_pattern.match(sagemaker_match.group(8))

 if name_match is None:
 return None, None
 return None, None, None
 else:
 return name_match.group(1), name_match.group(2)
 return name_match.group(1), name_match.group(2), name_match.group(4)


def framework_version_from_tag(image_tag):
 """Extract the framework version from the image tag.

 Args:
 image_tag (str): Image tag, which should take the form '<framework_version>-<device>-<py_version>'

 Returns:
 str: The framework version.
 """
 tag_pattern = re.compile('^(.*)-(cpu|gpu)-(py2|py3)$')
 tag_match = tag_pattern.match(image_tag)
 return None if tag_match is None else tag_match.group(1)


def parse_s3_url(url):
Expand Down

4 changes: 1 addition & 3 deletions
src/sagemaker/mxnet/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand All@@ -14,5 +14,3 @@
from sagemaker.mxnet.model import MXNetModel, MXNetPredictor

__all__ = [MXNet, MXNetModel, MXNetPredictor]

DOCKER_TAG = "1.0"

13 changes: 13 additions & 0 deletions
src/sagemaker/mxnet/defaults.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
MXNET_VERSION = '1.0'
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "DEFAULT_MXNET_VERSION"?
@nadiayanadiayaFeb 23, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaults.DEFAULT_MXNET_VERSION ? I think it's a little bit redundant.
@winstonawswinstonawsFeb 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, whoops.
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Support TensorFlow-1.5.0 and MXNet-1.0.0 #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

nadiaya
merged 5 commits into

from

Feb 27, 2018
 Merged


Support TensorFlow-1.5.0 and MXNet-1.0.0 #82
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
4fac858

Update .gitignore to ignore pytest_cache.


nadiaya Feb 22, 2018
c6b444a

Support TensorFlow-1.5.0 and MXNet-1.0.0


nadiaya Feb 22, 2018
d854806

Update and refactor tests. Add tests for fw_utils.


nadiaya Feb 24, 2018
dcfe663

Fix typo.


nadiaya Feb 26, 2018
aa64686

Merge branch 'master' into release-1.1


nadiaya Feb 26, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
.gitignore

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,3 +21,4 @@ doc/_build
**/.DS_Store
venv/
*~
.pytest_cache/

18 changes: 12 additions & 6 deletions
CHANGELOG.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,29 +2,35 @@
CHANGELOG
=========

1.1.0
=====

* feature: Estimators: add support for TensorFlow-1.5.0
@lukmislukmisFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update after master branch merge.
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.
* feature: Estimators: add support for MXNet-1.0.0

1.0.4
=====

* feature: Estimators: add support for Amazon Neural Topic Model(NTM) algorithm
* feature: Documentation: Fix description of an argument of sagemaker.session.train
* feature: Documentation: Add FM and LDA to the documentation
* feature: Documentation: fix description of an argument of sagemaker.session.train
* feature: Documentation: add FM and LDA to the documentation
* feature: Estimators: add support for async fit
* bug-fix: Estimators: fix estimator role expansion

1.0.3
=====

* feature: Estimators: add support for Amazon LDA algorithm
* feature: Hyperparameters: Add data_type to hyperparameters
* feature: Documentation: Update TensorFlow examples following API change
* feature: Session: Support multi-part uploads
* feature: Hyperparameters: add data_type to hyperparameters
* feature: Documentation: update TensorFlow examples following API change
* feature: Session: support multi-part uploads


1.0.2
=====

* feature: Estimators: add support for Amazon FactorizationMachines algorithm
* feature: Session: Correctly handle TooManyBuckets error_code in default_bucket method
* feature: Session: correctly handle TooManyBuckets error_code in default_bucket method
* feature: Tests: add training failure tests for TF and MXNet
* feature: Documentation: show how to make predictions against existing endpoint
* feature: Estimators: implement write_spmatrix_to_sparse_tensor to support any scipy.sparse matrix
Expand Down

89 changes: 60 additions & 29 deletions
README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,7 +39,7 @@ You can install from source by cloning this repository and issuing a pip install

 git clone https://github.com/aws/sagemaker-python-sdk.git
 python setup.py sdist
 pip install dist/sagemaker-1.0.4.tar.gz
 pip install dist/sagemaker-1.1.0.tar.gz

Supported Python versions
~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -114,6 +114,8 @@ MXNet SageMaker Estimators

With MXNet Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``1.0.0``, ``0.12.1``.

Training with MXNet
~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -185,7 +187,7 @@ If you want to run your training script locally via the Python interpreter, look
Using MXNet and numpy
^^^^^^^^^^^^^^^^^^^^^

You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 0.12 and numpy version 1.12.0. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.
You can import both ``mxnet`` and ``numpy`` in your training script. When your script runs in SageMaker, it will run with access to MXNet version 1.0.0 and numpy version 1.13.3 by default. For more information on the environment your script runs in, please see `SageMaker MXNet Containers <#sagemaker-mxnet-containers>`__.

Running an MXNet training script in SageMaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand DownExpand Up@@ -682,26 +684,33 @@ When training and deploying training scripts, SageMaker runs your Python script

SageMaker runs MXNet Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by passing a ``py_version`` keyword arg to the MXNet Estimator constructor. Setting this to ``py2`` (the default) will cause your training script to be run on Python 2.7. Setting this to ``py3`` will cause your training script to be run on Python 3.5. This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

Your MXNet training script will be run on version 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the train_instance_type, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
Your MXNet training script will be run on version 1.0.0 (by default) or 0.12 of MXNet, built for either GPU or CPU use. The decision to use the GPU or CPU version of MXNet is made by the ``train_instance_type``, set on the MXNet constructor. If you choose a GPU instance type, your training job will be run on a GPU version of MXNet. If you choose a CPU instance type, your training job will be run on a CPU version of MXNet. Similarly, when you call deploy, specifying a GPU or CPU deploy_instance_type, will control which MXNet build your Endpoint runs.
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write this to be more future-proof? Maybe something like, "will run using the latest supported version of MXNet by default. See (something) for our currently supported versions. You can choose older versions by..."
@nadiayanadiayaFeb 23, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we still would need to make an sdk change to change the default.

I believe it adds value to say what version we support by default.

Each Docker container has the following dependencies installed:
The Docker images have the following dependencies installed:

- Python 2.7 or Python 3.5, depending on the ``py_version`` argument on
 the MXNet constructor.
- MXNet 0.12, built for either GPU or CPU, depending on the instance
 type for training or deploying.
- CUDA 9.0
- numpy 1.12
+-------------------------+--------------+-------------+
| Dependencies | MXNet 0.12.1 | MXNet 1.0.0 |
+-------------------------+--------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5|
+-------------------------+--------------+-------------+
| CUDA | 9.0 | 9.0 |
+-------------------------+--------------+-------------+
| numpy | 1.13.3 | 1.13.3 |
+-------------------------+--------------+-------------+

The Docker images extend Ubuntu 16.04.

You can select version of MXNet by passing a ``framework_version`` keyword arg to the MXNet Estimator constructor. Currently supported versions are ``1.0.0`` and ``0.12.1``. You can also set ``framework_version`` to ``1.0 (default)`` or ``0.12`` which will cause your training script to be run on the latest supported MXNet 1.0 or 0.12 versions respectively.

TensorFlow SageMaker Estimators
-------------------------------

TensorFlow SageMaker Estimators allow you to run your own TensorFlow
training algorithms on SageMaker Learner, and to host your own TensorFlow
models on SageMaker Hosting.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``.

Training with TensorFlow
~~~~~~~~~~~~~~~~~~~~~~~~

Expand DownExpand Up@@ -735,7 +744,7 @@ Preparing the TensorFlow training script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your TensorFlow training script must be a **Python 2.7** source file. The current supported TensorFlow
version is **1.4.0**. This training script **must contain** the following functions:
versions are **1.5.0 (default)** and **1.4.1**. This training script **must contain** the following functions:
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about future-proofing.

- ``model_fn``: defines the model that will be trained.
- ``train_input_fn``: preprocess and load training data.
Expand DownExpand Up@@ -1419,27 +1428,49 @@ A example with ``input_fn`` and ``output_fn`` above can be found in
SageMaker TensorFlow Docker containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The TensorFlow Docker container supports Python 2.7. The Docker container has the following Python modules installed:
- awscli 1.12.1
- boto3 1.4.7
- botocore 1.5.92
- futures 2.2.0
- gevent 1.2.2
- grpcio 1.7.0
- numpy 1.13.3
- pandas 0.21.0
- protobuf 3.4.0
- requests 2.14.2
- scikit-learn 0.19.1
- scipy 1.0.0
- six 1.10.0
- sklearn 0.0
- tensorflow 1.4.0
- tensorflow-serving-api 1.4.0
- tensorflow-tensorboard 0.4.0rc2
The TensorFlow Docker images support Python 2.7 and have the following Python modules installed:

+------------------------+------------------+------------------+
| Dependencies | tensorflow 1.4.1 | tensorflow 1.5.0 |
+------------------------+------------------+------------------+
| awscli | 1.12.1 | 1.14.35 |
+------------------------+------------------+------------------+
| boto3 | 1.4.7 | 1.5.22 |
+------------------------+------------------+------------------+
| botocore | 1.5.92 | 1.8.36 |
+------------------------+------------------+------------------+
| futures | 2.2.0 | 2.2.0 |
+------------------------+------------------+------------------+
| gevent | 1.2.2 | 1.2.2 |
+------------------------+------------------+------------------+
| grpcio | 1.7.0 | 1.9.0 |
+------------------------+------------------+------------------+
| numpy | 1.13.3 | 1.14.0 |
+------------------------+------------------+------------------+
| pandas | 0.21.0 | 0.22.0 |
+------------------------+------------------+------------------+
| protobuf | 3.4.0 | 3.5.1 |
+------------------------+------------------+------------------+
| requests | 2.14.2 | 2.18.4 |
+------------------------+------------------+------------------+
| scikit-learn | 0.19.1 | 0.19.1 |
+------------------------+------------------+------------------+
| scipy | 1.0.0 | 1.0.0 |
+------------------------+------------------+------------------+
| six | 1.10.0 | 1.10.0 |
+------------------------+------------------+------------------+
| sklearn | 0.0 | 0.0 |
+------------------------+------------------+------------------+
| tensorflow | 1.4.1 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-serving-api | 1.4.0 | 1.5.0 |
+------------------------+------------------+------------------+
| tensorflow-tensorboard | 0.4.0 | 1.5.1 |
+------------------------+------------------+------------------+

The Docker images extend Ubuntu 16.04.

You can select version of TensorFlow by passing a ``framework_version`` keyword arg to the TensorFlow Estimator constructor. Currently supported versions are ``1.5.0`` and ``1.4.1``. You can also set ``framework_version`` to ``1.5 (default)`` or ``1.4`` which will cause your training script to be run on the latest supported TensorFlow 1.5 or 1.4 versions respectively.

AWS SageMaker Estimators
------------------------
Expand Down

2 changes: 1 addition & 1 deletion
doc/conf.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@ def __getattr__(cls, name):
 'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)

version = '1.0.4'
version = '1.1.0'
project = u'sagemaker'

# Add any Sphinx extension module names here, as strings. They can be extensions
Expand Down

2 changes: 1 addition & 1 deletion
setup.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@ def read(fname):


setup(name="sagemaker",
 version="1.0.4",
 version="1.1.0",
 description="Open source library for training and deploying models on Amazon SageMaker.",
 packages=find_packages('src'),
 package_dir={'': 'src'},
Expand Down

46 changes: 32 additions & 14 deletions
src/sagemaker/fw_utils.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand DownExpand Up@@ -28,26 +28,28 @@
"""


def create_image_uri(region, framework, instance_type, py_version='py2', tag='1.0', account='520713654638'):
def create_image_uri(region, framework, instance_type, framework_version, py_version, account='520713654638'):
 """Return the ECR URI of an image.

 Args:
 region (str): AWS region where the image is uploaded.
 framework (str): framework used by the image.
 instance_type (str): EC2 instance type. Used to determine whether to use the CPU image or GPU image.
py_version (str): Python version. (default: 'py2')
tag (str): ECR image tag, which denotes the image version. (default: '1.0')
framework_version (str): The version of the framework.
py_version (str): Python version. One of 'py2' or 'py3'.
 account (str): AWS account that contains the image. (default: '520713654638')

 Returns:
 str: The appropriate image URI based on the given parameters.
 """
device_version = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/ec2/instance-types/
device_type = 'cpu'
 # Instance types that start with G, P are GPU powered: https://aws.amazon.com/sagemaker/pricing/instance-types/
 if instance_type[3] in ['g', 'p']:
 device_version = 'gpu'
 device_type = 'gpu'

 tag = "{}-{}-{}".format(framework_version, device_type, py_version)
 return "{}.dkr.ecr.{}.amazonaws.com/sagemaker-{}-{}-{}:{}" \
 .format(account, region, framework, py_version, device_version, tag)
 .format(account, region, framework, py_version, device_type, tag)


def tar_and_upload_dir(session, bucket, s3_key_prefix, script, directory):
Expand DownExpand Up@@ -112,21 +114,37 @@ def framework_name_from_image(image_name):
 tuple: A tuple containing:
 str: The framework name
 str: The Python version
 str: The image tag
 """
 # image name format: <account>.dkr.ecr.<region>.amazonaws.com/sagemaker-<framework>-<py_ver>-<device>:<tag>
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*)(:)(.*)$')
 sagemaker_pattern = re.compile('^(\d+)(\.)dkr(\.)ecr(\.)(.+)(\.)amazonaws.com(/)(.*:.*)$')
 sagemaker_match = sagemaker_pattern.match(image_name)
 if sagemaker_match is None:
 return None, None
 return None, None, None
 else:
 # extract framework and python version
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu)$')
 # extract framework, python version and image tag
 name_pattern = re.compile('^sagemaker-(tensorflow|mxnet)-(py2|py3)-(cpu|gpu):(.*)$')

 name_match = name_pattern.match(sagemaker_match.group(8))

 if name_match is None:
 return None, None
 return None, None, None
 else:
 return name_match.group(1), name_match.group(2)
 return name_match.group(1), name_match.group(2), name_match.group(4)


def framework_version_from_tag(image_tag):
 """Extract the framework version from the image tag.

 Args:
 image_tag (str): Image tag, which should take the form '<framework_version>-<device>-<py_version>'

 Returns:
 str: The framework version.
 """
 tag_pattern = re.compile('^(.*)-(cpu|gpu)-(py2|py3)$')
 tag_match = tag_pattern.match(image_tag)
 return None if tag_match is None else tag_match.group(1)


def parse_s3_url(url):
Expand Down

4 changes: 1 addition & 3 deletions
src/sagemaker/mxnet/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
Expand All@@ -14,5 +14,3 @@
from sagemaker.mxnet.model import MXNetModel, MXNetPredictor

__all__ = [MXNet, MXNetModel, MXNetPredictor]

DOCKER_TAG = "1.0"

13 changes: 13 additions & 0 deletions
src/sagemaker/mxnet/defaults.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
MXNET_VERSION = '1.0'
@winstonawswinstonawsFeb 23, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "DEFAULT_MXNET_VERSION"?
@nadiayanadiayaFeb 23, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defaults.DEFAULT_MXNET_VERSION ? I think it's a little bit redundant.
@winstonawswinstonawsFeb 26, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, whoops.
Loading