Bug fix for getting dataframes in TrainingJobAnalytics. - #441

Merged
laurenyu merged 30 commits into
aws:masterfrom
piyushadlakha:TrainingJobAnalytics
Nov 14, 2018
Merged

Bug fix for getting dataframes in TrainingJobAnalytics.#441
laurenyu merged 30 commits into
aws:masterfrom
piyushadlakha:TrainingJobAnalytics

Conversation

@piyushadlakha

@piyushadlakhapiyushadlakha commented Oct 25, 2018•
edited
Loading

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:
Bug fix for getting dataframes in TrainingJobAnalytics.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have updated the changelog with a description of my changes (if appropriate)
  • I have updated any necessary documentation (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

laurenyu
laurenyu reviewed Oct 25, 2018
View reviewed changes

@laurenyulaurenyu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a unit test?

Comment threadsrc/sagemaker/analytics.py Outdated
description = self._sage_client.describe_training_job(TrainingJobName=self.name)
start_time = description[u'TrainingStartTime'] # datetime object
end_time = description.get(u'TrainingEndTime', datetime.datetime.utcnow())
# Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.

@laurenyulaurenyuOct 25, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cloud watch/CloudWatch

@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment threadsrc/sagemaker/analytics.py Outdated
# Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
# This results in logs being searched in the time range in which the correct log line was not present.
# Example - Log time - 2018-10-22 08:25:55
# Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)

@laurenyulaurenyuOct 25, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: there's an extra space after "be"

@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@codecov-io

codecov-io commented Oct 25, 2018•
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #441 into master will increase coverage by 0.27%.
The diff coverage is 94.18%.

Impacted file tree graph

@@ Coverage Diff @@## master #441 +/- ##
==========================================
+ Coverage 93.75% 94.02% +0.27% 
==========================================
Files 55 57 +2 Lines 4034 4269 +235 ==========================================
+ Hits 3782 4014 +232 - Misses 252 255 +3
Impacted FilesCoverage Δ
src/sagemaker/cli/tensorflow.py50% <ø> (ø)⬆️
src/sagemaker/transformer.py100% <ø> (ø)⬆️
src/sagemaker/cli/mxnet.py100% <ø> (ø)⬆️
src/sagemaker/fw_utils.py100% <100%> (ø)⬆️
src/sagemaker/predictor.py96.77% <100%> (+0.17%)⬆️
src/sagemaker/analytics.py91.94% <100%> (ø)⬆️
src/sagemaker/pytorch/estimator.py100% <100%> (ø)⬆️
src/sagemaker/tensorflow/predictor.py95.83% <100%> (ø)⬆️
src/sagemaker/chainer/estimator.py100% <100%> (ø)⬆️
src/sagemaker/amazon/kmeans.py100% <100%> (ø)⬆️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5595af...25038d1. Read the comment docs.

@piyushadlakha

piyushadlakha commented Oct 25, 2018

Copy link
Copy Markdown
ContributorAuthor

Unit test for this are already in place. We are in all cases mocking the calls to CW client and that mock is independent of the namespace.

@laurenyu

laurenyu commented Oct 25, 2018

Copy link
Copy Markdown
Contributor

given the change in logic for calculating end time, it would be good to have a unit test for that

@piyushadlakha

piyushadlakha commented Oct 25, 2018

Copy link
Copy Markdown
ContributorAuthor

UT's added.

laurenyu
laurenyu previously approved these changes Oct 26, 2018
View reviewed changes
mvsusp
mvsusp previously requested changes Oct 26, 2018
View reviewed changes

@mvsuspmvsusp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check our PR guidelines in description of the PR. Make sure that you provided unit tests and updated the Changelog.

Thanks

Comment threadsrc/sagemaker/analytics.py
@laurenyulaurenyu mentioned this pull request Nov 13, 2018
Merged
4 tasks
RodrigoAtAWSand others added 16 commits November 13, 2018 17:20
@RodrigoAtAWSPiyush Adlakha
Add model parameters to Estimator, and bump library version to 1.13.0 (…
c55e7ca
…#450)
* Add incremental training model parameters to Estimator, and bump library version to 1.13.0
* Update README.rst
Removed unnecessary comma from sentence.
* Update estimator.py
Removed duplicated line
@iquinteroPiyush Adlakha
local mode: improve training input/output (#449)
5bdb50a
output_path is now honored:
Can be either file:// or s3:// - This also changes the default
behavior of local mode to use the SDK provided default S3 bucket
if nothing is passed. This makes it easier for customers to create
models in SageMaker too since their Model Artifacts will already be
a tarfile in S3.
input_channel content_type is now honored in the same way as SageMaker. If it is not provided it is not passed to the container. Before we were always passing 'application/octet-stream'
@pdasamznPiyush Adlakha
Add image URIs for built-in Algorithms for SIN/LHR/BOM/SFO/YUL (#456)
3d103b4
@laurenyuPiyush Adlakha
Support MXNet 1.3 with its training script format changes (#446)
23851b1
@nadiayaPiyush Adlakha
Make InputDataConfig optional for training. (#459)
bd1f43b
* Make InputDataConfig optional for training.
* Update boto3 dependency to make sure boto support no InputDataConfig.
* Update changelog.
* Add missing assertion for chainer failure script test.
@jesterhazyPiyush Adlakha
add tfs container support (#460)
2edcc3a
* add tensorflow serving container support
@jesterhazyPiyush Adlakha
simplify create_image_uri function (#462)
10b7e42
* add failure test case
* fix flaky assert
@jesterhazyPiyush Adlakha
bump version to 1.14.0 (#463)
e8bf717
@yangawsPiyush Adlakha
Skip gpu tests in regions without ml.p2.xlarge (#461)
eea2ad5
@pnpnpnPiyush Adlakha
Adding Object2Vec support to SageMaker Python SDK (#467)
c8147a9
@jesterhazyPiyush Adlakha
fix readme rendering (#464)
52d1ec4
@iquinteroPiyush Adlakha
Add Pylint (#465)
3a69cf6
Add Pylint checking. Fixed all the current errors and warnings,
any future PRs will fail if they introduce any pylint error/warnings.
@nadiayaPiyush Adlakha
Support optional input channels in local mode. (#466)
9a997a5
@iquinteroPiyush Adlakha
build: upgrade docker-compose to 1.23 (#470)
6a1d93c
docker-compose 1.23 has moved to a newer version of requests
which allows us to remove the upper bound on urllib3 that
was causing a lot of problems for everyone.
@jesterhazyPiyush Adlakha
add tensorflow serving docs (#468)
876287e
* add tensorflow serving docs
* add content_type to tensorflow.serving.Predictor
* support CustomAttributes in local mode
@leopdPiyush Adlakha
Very minor: Better documentation comment on DeferredError. (#469)
0c4ad9c
* Better documentation comment on DeferredError.
* Using Napolean-style docstring formatting for example code.
* Fixup flake8 trailing whitespace.
@laurenyu

laurenyu commented Nov 14, 2018

Copy link
Copy Markdown
Contributor

can you check the unit tests? four are failing with:

E RemovedInPytest4Warning: Fixture "sagemaker_session" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

@laurenyu
laurenyu merged commit 5db2603 into aws:masterNov 14, 2018
ChoiByungWook pushed a commit that referenced this pull request Dec 8, 2020
@akhilmehra@ChoiByungWook
feature: add model parallelism support (#441)
bbb3f16
@icywang86ruiicywang86rui mentioned this pull request Jan 7, 2021
Closed
Evan-W-ang added a commit to Evan-W-ang/sagemaker-python-sdk that referenced this pull request Jun 8, 2026
@Evan-W-ang
Bug fix for getting dataframes in TrainingJobAnalytics. (aws#441)
957026c
Evan-W-ang added a commit to Evan-W-ang/sagemaker-python-sdk that referenced this pull request Jun 8, 2026
@Evan-W-ang
feature: add model parallelism support (aws#441)
5bf53f0
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@laurenyulaurenyulaurenyu approved these changes
@mvsuspmvsuspmvsusp left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

12 participants

@piyushadlakha@codecov-io@laurenyu@mvsusp@RodrigoAtAWS@iquintero@pdasamzn@nadiaya@jesterhazy@yangaws@pnpnpn@leopd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Bug fix for getting dataframes in TrainingJobAnalytics. - #441
Merged
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Nov 14, 2018
Merged
Bug fix for getting dataframes in TrainingJobAnalytics.#441
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Conversation
@piyushadlakha
@piyushadlakhapiyushadlakha
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Contributor
Issue #, if available:
Description of changes:

Bug fix for getting dataframes in TrainingJobAnalytics.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated the changelog with a description of my changes (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
laurenyu
laurenyu
reviewed
Oct 25, 2018
View reviewed changes
@laurenyulaurenyu
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a unit test?
Comment threadsrc/sagemaker/analytics.py
Outdated

 description = self._sage_client.describe_training_job(TrainingJobName=self.name)
 start_time = description[u'TrainingStartTime'] # datetime object
 end_time = description.get(u'TrainingEndTime', datetime.datetime.utcnow())
 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cloud watch/CloudWatch
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
Comment threadsrc/sagemaker/analytics.py
Outdated

 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
 # This results in logs being searched in the time range in which the correct log line was not present.
 # Example - Log time - 2018-10-22 08:25:55
 # Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: there's an extra space after "be"
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@codecov-io
codecov-io
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #441 into master will increase coverage by 0.27%.

The diff coverage is 94.18%.
Impacted file tree graph
@@ Coverage Diff @@## master #441 +/- ##
==========================================
+ Coverage 93.75% 94.02% +0.27% 
==========================================
Files 55 57 +2 Lines 4034 4269 +235 ==========================================
+ Hits 3782 4014 +232 - Misses 252 255 +3
Impacted FilesCoverage Δ
src/sagemaker/cli/tensorflow.py50% <ø> (ø)⬆️
src/sagemaker/transformer.py100% <ø> (ø)⬆️
src/sagemaker/cli/mxnet.py100% <ø> (ø)⬆️
src/sagemaker/fw_utils.py100% <100%> (ø)⬆️
src/sagemaker/predictor.py96.77% <100%> (+0.17%)⬆️
src/sagemaker/analytics.py91.94% <100%> (ø)⬆️
src/sagemaker/pytorch/estimator.py100% <100%> (ø)⬆️
src/sagemaker/tensorflow/predictor.py95.83% <100%> (ø)⬆️
src/sagemaker/chainer/estimator.py100% <100%> (ø)⬆️
src/sagemaker/amazon/kmeans.py100% <100%> (ø)⬆️
... and 20 more
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update a5595af...25038d1. Read the comment docs.
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Unit test for this are already in place. We are in all cases mocking the calls to CW client and that mock is independent of the namespace.
@laurenyu
laurenyu
commented
Oct 25, 2018

Copy link


Copy Markdown

Contributor
given the change in logic for calculating end time, it would be good to have a unit test for that
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
UT's added.
laurenyu
laurenyu
previously approved these changes
Oct 26, 2018
View reviewed changes
mvsusp
mvsusp
previously requested changes
Oct 26, 2018
View reviewed changes
@mvsuspmvsusp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check our PR guidelines in description of the PR. Make sure that you provided unit tests and updated the Changelog.
Thanks
Comment threadsrc/sagemaker/analytics.py
@laurenyulaurenyu
mentioned this pull request
Nov 13, 2018
 Merged

4 tasks
RodrigoAtAWSand others
added 16 commits
November 13, 2018 17:20
@RodrigoAtAWSPiyush Adlakha
Add model parameters to Estimator, and bump library version to 1.13.0 (…
c55e7ca
…#450)
* Add incremental training model parameters to Estimator, and bump library version to 1.13.0
* Update README.rst
Removed unnecessary comma from sentence.
* Update estimator.py
Removed duplicated line
@iquinteroPiyush Adlakha
local mode: improve training input/output (#449)
5bdb50a
output_path is now honored:
Can be either file:// or s3:// - This also changes the default
behavior of local mode to use the SDK provided default S3 bucket
if nothing is passed. This makes it easier for customers to create
models in SageMaker too since their Model Artifacts will already be
a tarfile in S3.
input_channel content_type is now honored in the same way as SageMaker. If it is not provided it is not passed to the container. Before we were always passing 'application/octet-stream'
@pdasamznPiyush Adlakha
Add image URIs for built-in Algorithms for SIN/LHR/BOM/SFO/YUL (#456)
3d103b4
@laurenyuPiyush Adlakha
Support MXNet 1.3 with its training script format changes (#446)
23851b1
@nadiayaPiyush Adlakha
Make InputDataConfig optional for training. (#459)
bd1f43b
* Make InputDataConfig optional for training.
* Update boto3 dependency to make sure boto support no InputDataConfig.
* Update changelog.
* Add missing assertion for chainer failure script test.
@jesterhazyPiyush Adlakha
add tfs container support (#460)
2edcc3a
* add tensorflow serving container support
@jesterhazyPiyush Adlakha
simplify create_image_uri function (#462)
10b7e42
* add failure test case
* fix flaky assert
@jesterhazyPiyush Adlakha
bump version to 1.14.0 (#463)
e8bf717
@yangawsPiyush Adlakha
Skip gpu tests in regions without ml.p2.xlarge (#461)
eea2ad5
@pnpnpnPiyush Adlakha
Adding Object2Vec support to SageMaker Python SDK (#467)
c8147a9
@jesterhazyPiyush Adlakha
fix readme rendering (#464)
52d1ec4
@iquinteroPiyush Adlakha
Add Pylint (#465)
3a69cf6
Add Pylint checking. Fixed all the current errors and warnings,
any future PRs will fail if they introduce any pylint error/warnings.
@nadiayaPiyush Adlakha
Support optional input channels in local mode. (#466)
9a997a5
@iquinteroPiyush Adlakha
build: upgrade docker-compose to 1.23 (#470)
6a1d93c
docker-compose 1.23 has moved to a newer version of requests
which allows us to remove the upper bound on urllib3 that
was causing a lot of problems for everyone.
@jesterhazyPiyush Adlakha
add tensorflow serving docs (#468)
876287e
* add tensorflow serving docs
* add content_type to tensorflow.serving.Predictor
* support CustomAttributes in local mode
@leopdPiyush Adlakha
Very minor: Better documentation comment on DeferredError. (#469)
0c4ad9c
* Better documentation comment on DeferredError.
* Using Napolean-style docstring formatting for example code.
* Fixup flake8 trailing whitespace.
@laurenyu
laurenyu
commented
Nov 14, 2018

Copy link


Copy Markdown

Contributor
can you check the unit tests? four are failing with:
E RemovedInPytest4Warning: Fixture "sagemaker_session" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

@laurenyu
laurenyu
merged commit 5db2603
into
aws:masterNov 14, 2018
ChoiByungWook pushed a commit
that referenced
this pull request
Dec 8, 2020
@akhilmehra@ChoiByungWook
feature: add model parallelism support (#441)
bbb3f16
@icywang86ruiicywang86rui
mentioned this pull request
Jan 7, 2021

Closed

Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Bug fix for getting dataframes in TrainingJobAnalytics. (aws#441)
957026c
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
feature: add model parallelism support (aws#441)
5bf53f0
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


2 more reviewers

@laurenyulaurenyulaurenyu approved these changes
@mvsuspmvsuspmvsusp left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

12 participants

@piyushadlakha@codecov-io@laurenyu@mvsusp@RodrigoAtAWS@iquintero@pdasamzn@nadiaya@jesterhazy@yangaws@pnpnpn@leopd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Bug fix for getting dataframes in TrainingJobAnalytics. - #441
Merged
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Nov 14, 2018
Merged
Bug fix for getting dataframes in TrainingJobAnalytics.#441
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Conversation
@piyushadlakha
@piyushadlakhapiyushadlakha
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Contributor
Issue #, if available:
Description of changes:

Bug fix for getting dataframes in TrainingJobAnalytics.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated the changelog with a description of my changes (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
laurenyu
laurenyu
reviewed
Oct 25, 2018
View reviewed changes
@laurenyulaurenyu
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a unit test?
Comment threadsrc/sagemaker/analytics.py
Outdated

 description = self._sage_client.describe_training_job(TrainingJobName=self.name)
 start_time = description[u'TrainingStartTime'] # datetime object
 end_time = description.get(u'TrainingEndTime', datetime.datetime.utcnow())
 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cloud watch/CloudWatch
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
Comment threadsrc/sagemaker/analytics.py
Outdated

 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
 # This results in logs being searched in the time range in which the correct log line was not present.
 # Example - Log time - 2018-10-22 08:25:55
 # Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: there's an extra space after "be"
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@codecov-io
codecov-io
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #441 into master will increase coverage by 0.27%.

The diff coverage is 94.18%.
Impacted file tree graph
@@ Coverage Diff @@## master #441 +/- ##
==========================================
+ Coverage 93.75% 94.02% +0.27% 
==========================================
Files 55 57 +2 Lines 4034 4269 +235 ==========================================
+ Hits 3782 4014 +232 - Misses 252 255 +3
Impacted FilesCoverage Δ
src/sagemaker/cli/tensorflow.py50% <ø> (ø)⬆️
src/sagemaker/transformer.py100% <ø> (ø)⬆️
src/sagemaker/cli/mxnet.py100% <ø> (ø)⬆️
src/sagemaker/fw_utils.py100% <100%> (ø)⬆️
src/sagemaker/predictor.py96.77% <100%> (+0.17%)⬆️
src/sagemaker/analytics.py91.94% <100%> (ø)⬆️
src/sagemaker/pytorch/estimator.py100% <100%> (ø)⬆️
src/sagemaker/tensorflow/predictor.py95.83% <100%> (ø)⬆️
src/sagemaker/chainer/estimator.py100% <100%> (ø)⬆️
src/sagemaker/amazon/kmeans.py100% <100%> (ø)⬆️
... and 20 more
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update a5595af...25038d1. Read the comment docs.
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Unit test for this are already in place. We are in all cases mocking the calls to CW client and that mock is independent of the namespace.
@laurenyu
laurenyu
commented
Oct 25, 2018

Copy link


Copy Markdown

Contributor
given the change in logic for calculating end time, it would be good to have a unit test for that
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
UT's added.
laurenyu
laurenyu
previously approved these changes
Oct 26, 2018
View reviewed changes
mvsusp
mvsusp
previously requested changes
Oct 26, 2018
View reviewed changes
@mvsuspmvsusp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check our PR guidelines in description of the PR. Make sure that you provided unit tests and updated the Changelog.
Thanks
Comment threadsrc/sagemaker/analytics.py
@laurenyulaurenyu
mentioned this pull request
Nov 13, 2018
 Merged

4 tasks
RodrigoAtAWSand others
added 16 commits
November 13, 2018 17:20
@RodrigoAtAWSPiyush Adlakha
Add model parameters to Estimator, and bump library version to 1.13.0 (…
c55e7ca
…#450)
* Add incremental training model parameters to Estimator, and bump library version to 1.13.0
* Update README.rst
Removed unnecessary comma from sentence.
* Update estimator.py
Removed duplicated line
@iquinteroPiyush Adlakha
local mode: improve training input/output (#449)
5bdb50a
output_path is now honored:
Can be either file:// or s3:// - This also changes the default
behavior of local mode to use the SDK provided default S3 bucket
if nothing is passed. This makes it easier for customers to create
models in SageMaker too since their Model Artifacts will already be
a tarfile in S3.
input_channel content_type is now honored in the same way as SageMaker. If it is not provided it is not passed to the container. Before we were always passing 'application/octet-stream'
@pdasamznPiyush Adlakha
Add image URIs for built-in Algorithms for SIN/LHR/BOM/SFO/YUL (#456)
3d103b4
@laurenyuPiyush Adlakha
Support MXNet 1.3 with its training script format changes (#446)
23851b1
@nadiayaPiyush Adlakha
Make InputDataConfig optional for training. (#459)
bd1f43b
* Make InputDataConfig optional for training.
* Update boto3 dependency to make sure boto support no InputDataConfig.
* Update changelog.
* Add missing assertion for chainer failure script test.
@jesterhazyPiyush Adlakha
add tfs container support (#460)
2edcc3a
* add tensorflow serving container support
@jesterhazyPiyush Adlakha
simplify create_image_uri function (#462)
10b7e42
* add failure test case
* fix flaky assert
@jesterhazyPiyush Adlakha
bump version to 1.14.0 (#463)
e8bf717
@yangawsPiyush Adlakha
Skip gpu tests in regions without ml.p2.xlarge (#461)
eea2ad5
@pnpnpnPiyush Adlakha
Adding Object2Vec support to SageMaker Python SDK (#467)
c8147a9
@jesterhazyPiyush Adlakha
fix readme rendering (#464)
52d1ec4
@iquinteroPiyush Adlakha
Add Pylint (#465)
3a69cf6
Add Pylint checking. Fixed all the current errors and warnings,
any future PRs will fail if they introduce any pylint error/warnings.
@nadiayaPiyush Adlakha
Support optional input channels in local mode. (#466)
9a997a5
@iquinteroPiyush Adlakha
build: upgrade docker-compose to 1.23 (#470)
6a1d93c
docker-compose 1.23 has moved to a newer version of requests
which allows us to remove the upper bound on urllib3 that
was causing a lot of problems for everyone.
@jesterhazyPiyush Adlakha
add tensorflow serving docs (#468)
876287e
* add tensorflow serving docs
* add content_type to tensorflow.serving.Predictor
* support CustomAttributes in local mode
@leopdPiyush Adlakha
Very minor: Better documentation comment on DeferredError. (#469)
0c4ad9c
* Better documentation comment on DeferredError.
* Using Napolean-style docstring formatting for example code.
* Fixup flake8 trailing whitespace.
@laurenyu
laurenyu
commented
Nov 14, 2018

Copy link


Copy Markdown

Contributor
can you check the unit tests? four are failing with:
E RemovedInPytest4Warning: Fixture "sagemaker_session" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

@laurenyu
laurenyu
merged commit 5db2603
into
aws:masterNov 14, 2018
ChoiByungWook pushed a commit
that referenced
this pull request
Dec 8, 2020
@akhilmehra@ChoiByungWook
feature: add model parallelism support (#441)
bbb3f16
@icywang86ruiicywang86rui
mentioned this pull request
Jan 7, 2021

Closed

Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Bug fix for getting dataframes in TrainingJobAnalytics. (aws#441)
957026c
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
feature: add model parallelism support (aws#441)
5bf53f0
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


2 more reviewers

@laurenyulaurenyulaurenyu approved these changes
@mvsuspmvsuspmvsusp left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

12 participants

@piyushadlakha@codecov-io@laurenyu@mvsusp@RodrigoAtAWS@iquintero@pdasamzn@nadiaya@jesterhazy@yangaws@pnpnpn@leopd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Bug fix for getting dataframes in TrainingJobAnalytics. - #441
Merged
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Nov 14, 2018
Merged
Bug fix for getting dataframes in TrainingJobAnalytics.#441
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Conversation
@piyushadlakha
@piyushadlakhapiyushadlakha
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Contributor
Issue #, if available:
Description of changes:

Bug fix for getting dataframes in TrainingJobAnalytics.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated the changelog with a description of my changes (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
laurenyu
laurenyu
reviewed
Oct 25, 2018
View reviewed changes
@laurenyulaurenyu
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a unit test?
Comment threadsrc/sagemaker/analytics.py
Outdated

 description = self._sage_client.describe_training_job(TrainingJobName=self.name)
 start_time = description[u'TrainingStartTime'] # datetime object
 end_time = description.get(u'TrainingEndTime', datetime.datetime.utcnow())
 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cloud watch/CloudWatch
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
Comment threadsrc/sagemaker/analytics.py
Outdated

 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
 # This results in logs being searched in the time range in which the correct log line was not present.
 # Example - Log time - 2018-10-22 08:25:55
 # Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: there's an extra space after "be"
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@codecov-io
codecov-io
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #441 into master will increase coverage by 0.27%.

The diff coverage is 94.18%.
Impacted file tree graph
@@ Coverage Diff @@## master #441 +/- ##
==========================================
+ Coverage 93.75% 94.02% +0.27% 
==========================================
Files 55 57 +2 Lines 4034 4269 +235 ==========================================
+ Hits 3782 4014 +232 - Misses 252 255 +3
Impacted FilesCoverage Δ
src/sagemaker/cli/tensorflow.py50% <ø> (ø)⬆️
src/sagemaker/transformer.py100% <ø> (ø)⬆️
src/sagemaker/cli/mxnet.py100% <ø> (ø)⬆️
src/sagemaker/fw_utils.py100% <100%> (ø)⬆️
src/sagemaker/predictor.py96.77% <100%> (+0.17%)⬆️
src/sagemaker/analytics.py91.94% <100%> (ø)⬆️
src/sagemaker/pytorch/estimator.py100% <100%> (ø)⬆️
src/sagemaker/tensorflow/predictor.py95.83% <100%> (ø)⬆️
src/sagemaker/chainer/estimator.py100% <100%> (ø)⬆️
src/sagemaker/amazon/kmeans.py100% <100%> (ø)⬆️
... and 20 more
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update a5595af...25038d1. Read the comment docs.
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Unit test for this are already in place. We are in all cases mocking the calls to CW client and that mock is independent of the namespace.
@laurenyu
laurenyu
commented
Oct 25, 2018

Copy link


Copy Markdown

Contributor
given the change in logic for calculating end time, it would be good to have a unit test for that
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
UT's added.
laurenyu
laurenyu
previously approved these changes
Oct 26, 2018
View reviewed changes
mvsusp
mvsusp
previously requested changes
Oct 26, 2018
View reviewed changes
@mvsuspmvsusp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check our PR guidelines in description of the PR. Make sure that you provided unit tests and updated the Changelog.
Thanks
Comment threadsrc/sagemaker/analytics.py
@laurenyulaurenyu
mentioned this pull request
Nov 13, 2018
 Merged

4 tasks
RodrigoAtAWSand others
added 16 commits
November 13, 2018 17:20
@RodrigoAtAWSPiyush Adlakha
Add model parameters to Estimator, and bump library version to 1.13.0 (…
c55e7ca
…#450)
* Add incremental training model parameters to Estimator, and bump library version to 1.13.0
* Update README.rst
Removed unnecessary comma from sentence.
* Update estimator.py
Removed duplicated line
@iquinteroPiyush Adlakha
local mode: improve training input/output (#449)
5bdb50a
output_path is now honored:
Can be either file:// or s3:// - This also changes the default
behavior of local mode to use the SDK provided default S3 bucket
if nothing is passed. This makes it easier for customers to create
models in SageMaker too since their Model Artifacts will already be
a tarfile in S3.
input_channel content_type is now honored in the same way as SageMaker. If it is not provided it is not passed to the container. Before we were always passing 'application/octet-stream'
@pdasamznPiyush Adlakha
Add image URIs for built-in Algorithms for SIN/LHR/BOM/SFO/YUL (#456)
3d103b4
@laurenyuPiyush Adlakha
Support MXNet 1.3 with its training script format changes (#446)
23851b1
@nadiayaPiyush Adlakha
Make InputDataConfig optional for training. (#459)
bd1f43b
* Make InputDataConfig optional for training.
* Update boto3 dependency to make sure boto support no InputDataConfig.
* Update changelog.
* Add missing assertion for chainer failure script test.
@jesterhazyPiyush Adlakha
add tfs container support (#460)
2edcc3a
* add tensorflow serving container support
@jesterhazyPiyush Adlakha
simplify create_image_uri function (#462)
10b7e42
* add failure test case
* fix flaky assert
@jesterhazyPiyush Adlakha
bump version to 1.14.0 (#463)
e8bf717
@yangawsPiyush Adlakha
Skip gpu tests in regions without ml.p2.xlarge (#461)
eea2ad5
@pnpnpnPiyush Adlakha
Adding Object2Vec support to SageMaker Python SDK (#467)
c8147a9
@jesterhazyPiyush Adlakha
fix readme rendering (#464)
52d1ec4
@iquinteroPiyush Adlakha
Add Pylint (#465)
3a69cf6
Add Pylint checking. Fixed all the current errors and warnings,
any future PRs will fail if they introduce any pylint error/warnings.
@nadiayaPiyush Adlakha
Support optional input channels in local mode. (#466)
9a997a5
@iquinteroPiyush Adlakha
build: upgrade docker-compose to 1.23 (#470)
6a1d93c
docker-compose 1.23 has moved to a newer version of requests
which allows us to remove the upper bound on urllib3 that
was causing a lot of problems for everyone.
@jesterhazyPiyush Adlakha
add tensorflow serving docs (#468)
876287e
* add tensorflow serving docs
* add content_type to tensorflow.serving.Predictor
* support CustomAttributes in local mode
@leopdPiyush Adlakha
Very minor: Better documentation comment on DeferredError. (#469)
0c4ad9c
* Better documentation comment on DeferredError.
* Using Napolean-style docstring formatting for example code.
* Fixup flake8 trailing whitespace.
@laurenyu
laurenyu
commented
Nov 14, 2018

Copy link


Copy Markdown

Contributor
can you check the unit tests? four are failing with:
E RemovedInPytest4Warning: Fixture "sagemaker_session" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

@laurenyu
laurenyu
merged commit 5db2603
into
aws:masterNov 14, 2018
ChoiByungWook pushed a commit
that referenced
this pull request
Dec 8, 2020
@akhilmehra@ChoiByungWook
feature: add model parallelism support (#441)
bbb3f16
@icywang86ruiicywang86rui
mentioned this pull request
Jan 7, 2021

Closed

Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Bug fix for getting dataframes in TrainingJobAnalytics. (aws#441)
957026c
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
feature: add model parallelism support (aws#441)
5bf53f0
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


2 more reviewers

@laurenyulaurenyulaurenyu approved these changes
@mvsuspmvsuspmvsusp left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

12 participants

@piyushadlakha@codecov-io@laurenyu@mvsusp@RodrigoAtAWS@iquintero@pdasamzn@nadiaya@jesterhazy@yangaws@pnpnpn@leopd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Bug fix for getting dataframes in TrainingJobAnalytics. - #441
Merged
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Nov 14, 2018
Merged
Bug fix for getting dataframes in TrainingJobAnalytics.#441
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Conversation
@piyushadlakha
@piyushadlakhapiyushadlakha
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Contributor
Issue #, if available:
Description of changes:

Bug fix for getting dataframes in TrainingJobAnalytics.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated the changelog with a description of my changes (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
laurenyu
laurenyu
reviewed
Oct 25, 2018
View reviewed changes
@laurenyulaurenyu
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a unit test?
Comment threadsrc/sagemaker/analytics.py
Outdated

 description = self._sage_client.describe_training_job(TrainingJobName=self.name)
 start_time = description[u'TrainingStartTime'] # datetime object
 end_time = description.get(u'TrainingEndTime', datetime.datetime.utcnow())
 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cloud watch/CloudWatch
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
Comment threadsrc/sagemaker/analytics.py
Outdated

 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
 # This results in logs being searched in the time range in which the correct log line was not present.
 # Example - Log time - 2018-10-22 08:25:55
 # Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: there's an extra space after "be"
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@codecov-io
codecov-io
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #441 into master will increase coverage by 0.27%.

The diff coverage is 94.18%.
Impacted file tree graph
@@ Coverage Diff @@## master #441 +/- ##
==========================================
+ Coverage 93.75% 94.02% +0.27% 
==========================================
Files 55 57 +2 Lines 4034 4269 +235 ==========================================
+ Hits 3782 4014 +232 - Misses 252 255 +3
Impacted FilesCoverage Δ
src/sagemaker/cli/tensorflow.py50% <ø> (ø)⬆️
src/sagemaker/transformer.py100% <ø> (ø)⬆️
src/sagemaker/cli/mxnet.py100% <ø> (ø)⬆️
src/sagemaker/fw_utils.py100% <100%> (ø)⬆️
src/sagemaker/predictor.py96.77% <100%> (+0.17%)⬆️
src/sagemaker/analytics.py91.94% <100%> (ø)⬆️
src/sagemaker/pytorch/estimator.py100% <100%> (ø)⬆️
src/sagemaker/tensorflow/predictor.py95.83% <100%> (ø)⬆️
src/sagemaker/chainer/estimator.py100% <100%> (ø)⬆️
src/sagemaker/amazon/kmeans.py100% <100%> (ø)⬆️
... and 20 more
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update a5595af...25038d1. Read the comment docs.
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Unit test for this are already in place. We are in all cases mocking the calls to CW client and that mock is independent of the namespace.
@laurenyu
laurenyu
commented
Oct 25, 2018

Copy link


Copy Markdown

Contributor
given the change in logic for calculating end time, it would be good to have a unit test for that
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
UT's added.
laurenyu
laurenyu
previously approved these changes
Oct 26, 2018
View reviewed changes
mvsusp
mvsusp
previously requested changes
Oct 26, 2018
View reviewed changes
@mvsuspmvsusp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check our PR guidelines in description of the PR. Make sure that you provided unit tests and updated the Changelog.
Thanks
Comment threadsrc/sagemaker/analytics.py
@laurenyulaurenyu
mentioned this pull request
Nov 13, 2018
 Merged

4 tasks
RodrigoAtAWSand others
added 16 commits
November 13, 2018 17:20
@RodrigoAtAWSPiyush Adlakha
Add model parameters to Estimator, and bump library version to 1.13.0 (…
c55e7ca
…#450)
* Add incremental training model parameters to Estimator, and bump library version to 1.13.0
* Update README.rst
Removed unnecessary comma from sentence.
* Update estimator.py
Removed duplicated line
@iquinteroPiyush Adlakha
local mode: improve training input/output (#449)
5bdb50a
output_path is now honored:
Can be either file:// or s3:// - This also changes the default
behavior of local mode to use the SDK provided default S3 bucket
if nothing is passed. This makes it easier for customers to create
models in SageMaker too since their Model Artifacts will already be
a tarfile in S3.
input_channel content_type is now honored in the same way as SageMaker. If it is not provided it is not passed to the container. Before we were always passing 'application/octet-stream'
@pdasamznPiyush Adlakha
Add image URIs for built-in Algorithms for SIN/LHR/BOM/SFO/YUL (#456)
3d103b4
@laurenyuPiyush Adlakha
Support MXNet 1.3 with its training script format changes (#446)
23851b1
@nadiayaPiyush Adlakha
Make InputDataConfig optional for training. (#459)
bd1f43b
* Make InputDataConfig optional for training.
* Update boto3 dependency to make sure boto support no InputDataConfig.
* Update changelog.
* Add missing assertion for chainer failure script test.
@jesterhazyPiyush Adlakha
add tfs container support (#460)
2edcc3a
* add tensorflow serving container support
@jesterhazyPiyush Adlakha
simplify create_image_uri function (#462)
10b7e42
* add failure test case
* fix flaky assert
@jesterhazyPiyush Adlakha
bump version to 1.14.0 (#463)
e8bf717
@yangawsPiyush Adlakha
Skip gpu tests in regions without ml.p2.xlarge (#461)
eea2ad5
@pnpnpnPiyush Adlakha
Adding Object2Vec support to SageMaker Python SDK (#467)
c8147a9
@jesterhazyPiyush Adlakha
fix readme rendering (#464)
52d1ec4
@iquinteroPiyush Adlakha
Add Pylint (#465)
3a69cf6
Add Pylint checking. Fixed all the current errors and warnings,
any future PRs will fail if they introduce any pylint error/warnings.
@nadiayaPiyush Adlakha
Support optional input channels in local mode. (#466)
9a997a5
@iquinteroPiyush Adlakha
build: upgrade docker-compose to 1.23 (#470)
6a1d93c
docker-compose 1.23 has moved to a newer version of requests
which allows us to remove the upper bound on urllib3 that
was causing a lot of problems for everyone.
@jesterhazyPiyush Adlakha
add tensorflow serving docs (#468)
876287e
* add tensorflow serving docs
* add content_type to tensorflow.serving.Predictor
* support CustomAttributes in local mode
@leopdPiyush Adlakha
Very minor: Better documentation comment on DeferredError. (#469)
0c4ad9c
* Better documentation comment on DeferredError.
* Using Napolean-style docstring formatting for example code.
* Fixup flake8 trailing whitespace.
@laurenyu
laurenyu
commented
Nov 14, 2018

Copy link


Copy Markdown

Contributor
can you check the unit tests? four are failing with:
E RemovedInPytest4Warning: Fixture "sagemaker_session" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

@laurenyu
laurenyu
merged commit 5db2603
into
aws:masterNov 14, 2018
ChoiByungWook pushed a commit
that referenced
this pull request
Dec 8, 2020
@akhilmehra@ChoiByungWook
feature: add model parallelism support (#441)
bbb3f16
@icywang86ruiicywang86rui
mentioned this pull request
Jan 7, 2021

Closed

Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Bug fix for getting dataframes in TrainingJobAnalytics. (aws#441)
957026c
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
feature: add model parallelism support (aws#441)
5bf53f0
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


2 more reviewers

@laurenyulaurenyulaurenyu approved these changes
@mvsuspmvsuspmvsusp left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

12 participants

@piyushadlakha@codecov-io@laurenyu@mvsusp@RodrigoAtAWS@iquintero@pdasamzn@nadiaya@jesterhazy@yangaws@pnpnpn@leopd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Bug fix for getting dataframes in TrainingJobAnalytics. - #441
Merged
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Nov 14, 2018
Merged
Bug fix for getting dataframes in TrainingJobAnalytics.#441
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Conversation
@piyushadlakha
@piyushadlakhapiyushadlakha
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Contributor
Issue #, if available:
Description of changes:

Bug fix for getting dataframes in TrainingJobAnalytics.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated the changelog with a description of my changes (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
laurenyu
laurenyu
reviewed
Oct 25, 2018
View reviewed changes
@laurenyulaurenyu
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a unit test?
Comment threadsrc/sagemaker/analytics.py
Outdated

 description = self._sage_client.describe_training_job(TrainingJobName=self.name)
 start_time = description[u'TrainingStartTime'] # datetime object
 end_time = description.get(u'TrainingEndTime', datetime.datetime.utcnow())
 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cloud watch/CloudWatch
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
Comment threadsrc/sagemaker/analytics.py
Outdated

 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
 # This results in logs being searched in the time range in which the correct log line was not present.
 # Example - Log time - 2018-10-22 08:25:55
 # Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: there's an extra space after "be"
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@codecov-io
codecov-io
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #441 into master will increase coverage by 0.27%.

The diff coverage is 94.18%.
Impacted file tree graph
@@ Coverage Diff @@## master #441 +/- ##
==========================================
+ Coverage 93.75% 94.02% +0.27% 
==========================================
Files 55 57 +2 Lines 4034 4269 +235 ==========================================
+ Hits 3782 4014 +232 - Misses 252 255 +3
Impacted FilesCoverage Δ
src/sagemaker/cli/tensorflow.py50% <ø> (ø)⬆️
src/sagemaker/transformer.py100% <ø> (ø)⬆️
src/sagemaker/cli/mxnet.py100% <ø> (ø)⬆️
src/sagemaker/fw_utils.py100% <100%> (ø)⬆️
src/sagemaker/predictor.py96.77% <100%> (+0.17%)⬆️
src/sagemaker/analytics.py91.94% <100%> (ø)⬆️
src/sagemaker/pytorch/estimator.py100% <100%> (ø)⬆️
src/sagemaker/tensorflow/predictor.py95.83% <100%> (ø)⬆️
src/sagemaker/chainer/estimator.py100% <100%> (ø)⬆️
src/sagemaker/amazon/kmeans.py100% <100%> (ø)⬆️
... and 20 more
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update a5595af...25038d1. Read the comment docs.
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Unit test for this are already in place. We are in all cases mocking the calls to CW client and that mock is independent of the namespace.
@laurenyu
laurenyu
commented
Oct 25, 2018

Copy link


Copy Markdown

Contributor
given the change in logic for calculating end time, it would be good to have a unit test for that
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
UT's added.
laurenyu
laurenyu
previously approved these changes
Oct 26, 2018
View reviewed changes
mvsusp
mvsusp
previously requested changes
Oct 26, 2018
View reviewed changes
@mvsuspmvsusp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check our PR guidelines in description of the PR. Make sure that you provided unit tests and updated the Changelog.
Thanks
Comment threadsrc/sagemaker/analytics.py
@laurenyulaurenyu
mentioned this pull request
Nov 13, 2018
 Merged

4 tasks
RodrigoAtAWSand others
added 16 commits
November 13, 2018 17:20
@RodrigoAtAWSPiyush Adlakha
Add model parameters to Estimator, and bump library version to 1.13.0 (…
c55e7ca
…#450)
* Add incremental training model parameters to Estimator, and bump library version to 1.13.0
* Update README.rst
Removed unnecessary comma from sentence.
* Update estimator.py
Removed duplicated line
@iquinteroPiyush Adlakha
local mode: improve training input/output (#449)
5bdb50a
output_path is now honored:
Can be either file:// or s3:// - This also changes the default
behavior of local mode to use the SDK provided default S3 bucket
if nothing is passed. This makes it easier for customers to create
models in SageMaker too since their Model Artifacts will already be
a tarfile in S3.
input_channel content_type is now honored in the same way as SageMaker. If it is not provided it is not passed to the container. Before we were always passing 'application/octet-stream'
@pdasamznPiyush Adlakha
Add image URIs for built-in Algorithms for SIN/LHR/BOM/SFO/YUL (#456)
3d103b4
@laurenyuPiyush Adlakha
Support MXNet 1.3 with its training script format changes (#446)
23851b1
@nadiayaPiyush Adlakha
Make InputDataConfig optional for training. (#459)
bd1f43b
* Make InputDataConfig optional for training.
* Update boto3 dependency to make sure boto support no InputDataConfig.
* Update changelog.
* Add missing assertion for chainer failure script test.
@jesterhazyPiyush Adlakha
add tfs container support (#460)
2edcc3a
* add tensorflow serving container support
@jesterhazyPiyush Adlakha
simplify create_image_uri function (#462)
10b7e42
* add failure test case
* fix flaky assert
@jesterhazyPiyush Adlakha
bump version to 1.14.0 (#463)
e8bf717
@yangawsPiyush Adlakha
Skip gpu tests in regions without ml.p2.xlarge (#461)
eea2ad5
@pnpnpnPiyush Adlakha
Adding Object2Vec support to SageMaker Python SDK (#467)
c8147a9
@jesterhazyPiyush Adlakha
fix readme rendering (#464)
52d1ec4
@iquinteroPiyush Adlakha
Add Pylint (#465)
3a69cf6
Add Pylint checking. Fixed all the current errors and warnings,
any future PRs will fail if they introduce any pylint error/warnings.
@nadiayaPiyush Adlakha
Support optional input channels in local mode. (#466)
9a997a5
@iquinteroPiyush Adlakha
build: upgrade docker-compose to 1.23 (#470)
6a1d93c
docker-compose 1.23 has moved to a newer version of requests
which allows us to remove the upper bound on urllib3 that
was causing a lot of problems for everyone.
@jesterhazyPiyush Adlakha
add tensorflow serving docs (#468)
876287e
* add tensorflow serving docs
* add content_type to tensorflow.serving.Predictor
* support CustomAttributes in local mode
@leopdPiyush Adlakha
Very minor: Better documentation comment on DeferredError. (#469)
0c4ad9c
* Better documentation comment on DeferredError.
* Using Napolean-style docstring formatting for example code.
* Fixup flake8 trailing whitespace.
@laurenyu
laurenyu
commented
Nov 14, 2018

Copy link


Copy Markdown

Contributor
can you check the unit tests? four are failing with:
E RemovedInPytest4Warning: Fixture "sagemaker_session" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

@laurenyu
laurenyu
merged commit 5db2603
into
aws:masterNov 14, 2018
ChoiByungWook pushed a commit
that referenced
this pull request
Dec 8, 2020
@akhilmehra@ChoiByungWook
feature: add model parallelism support (#441)
bbb3f16
@icywang86ruiicywang86rui
mentioned this pull request
Jan 7, 2021

Closed

Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Bug fix for getting dataframes in TrainingJobAnalytics. (aws#441)
957026c
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
feature: add model parallelism support (aws#441)
5bf53f0
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


2 more reviewers

@laurenyulaurenyulaurenyu approved these changes
@mvsuspmvsuspmvsusp left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

12 participants

@piyushadlakha@codecov-io@laurenyu@mvsusp@RodrigoAtAWS@iquintero@pdasamzn@nadiaya@jesterhazy@yangaws@pnpnpn@leopd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Bug fix for getting dataframes in TrainingJobAnalytics. - #441
Merged
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Nov 14, 2018
Merged
Bug fix for getting dataframes in TrainingJobAnalytics.#441
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Conversation
@piyushadlakha
@piyushadlakhapiyushadlakha
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Contributor
Issue #, if available:
Description of changes:

Bug fix for getting dataframes in TrainingJobAnalytics.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated the changelog with a description of my changes (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
laurenyu
laurenyu
reviewed
Oct 25, 2018
View reviewed changes
@laurenyulaurenyu
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a unit test?
Comment threadsrc/sagemaker/analytics.py
Outdated

 description = self._sage_client.describe_training_job(TrainingJobName=self.name)
 start_time = description[u'TrainingStartTime'] # datetime object
 end_time = description.get(u'TrainingEndTime', datetime.datetime.utcnow())
 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cloud watch/CloudWatch
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
Comment threadsrc/sagemaker/analytics.py
Outdated

 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
 # This results in logs being searched in the time range in which the correct log line was not present.
 # Example - Log time - 2018-10-22 08:25:55
 # Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: there's an extra space after "be"
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@codecov-io
codecov-io
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #441 into master will increase coverage by 0.27%.

The diff coverage is 94.18%.
Impacted file tree graph
@@ Coverage Diff @@## master #441 +/- ##
==========================================
+ Coverage 93.75% 94.02% +0.27% 
==========================================
Files 55 57 +2 Lines 4034 4269 +235 ==========================================
+ Hits 3782 4014 +232 - Misses 252 255 +3
Impacted FilesCoverage Δ
src/sagemaker/cli/tensorflow.py50% <ø> (ø)⬆️
src/sagemaker/transformer.py100% <ø> (ø)⬆️
src/sagemaker/cli/mxnet.py100% <ø> (ø)⬆️
src/sagemaker/fw_utils.py100% <100%> (ø)⬆️
src/sagemaker/predictor.py96.77% <100%> (+0.17%)⬆️
src/sagemaker/analytics.py91.94% <100%> (ø)⬆️
src/sagemaker/pytorch/estimator.py100% <100%> (ø)⬆️
src/sagemaker/tensorflow/predictor.py95.83% <100%> (ø)⬆️
src/sagemaker/chainer/estimator.py100% <100%> (ø)⬆️
src/sagemaker/amazon/kmeans.py100% <100%> (ø)⬆️
... and 20 more
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update a5595af...25038d1. Read the comment docs.
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Unit test for this are already in place. We are in all cases mocking the calls to CW client and that mock is independent of the namespace.
@laurenyu
laurenyu
commented
Oct 25, 2018

Copy link


Copy Markdown

Contributor
given the change in logic for calculating end time, it would be good to have a unit test for that
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
UT's added.
laurenyu
laurenyu
previously approved these changes
Oct 26, 2018
View reviewed changes
mvsusp
mvsusp
previously requested changes
Oct 26, 2018
View reviewed changes
@mvsuspmvsusp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check our PR guidelines in description of the PR. Make sure that you provided unit tests and updated the Changelog.
Thanks
Comment threadsrc/sagemaker/analytics.py
@laurenyulaurenyu
mentioned this pull request
Nov 13, 2018
 Merged

4 tasks
RodrigoAtAWSand others
added 16 commits
November 13, 2018 17:20
@RodrigoAtAWSPiyush Adlakha
Add model parameters to Estimator, and bump library version to 1.13.0 (…
c55e7ca
…#450)
* Add incremental training model parameters to Estimator, and bump library version to 1.13.0
* Update README.rst
Removed unnecessary comma from sentence.
* Update estimator.py
Removed duplicated line
@iquinteroPiyush Adlakha
local mode: improve training input/output (#449)
5bdb50a
output_path is now honored:
Can be either file:// or s3:// - This also changes the default
behavior of local mode to use the SDK provided default S3 bucket
if nothing is passed. This makes it easier for customers to create
models in SageMaker too since their Model Artifacts will already be
a tarfile in S3.
input_channel content_type is now honored in the same way as SageMaker. If it is not provided it is not passed to the container. Before we were always passing 'application/octet-stream'
@pdasamznPiyush Adlakha
Add image URIs for built-in Algorithms for SIN/LHR/BOM/SFO/YUL (#456)
3d103b4
@laurenyuPiyush Adlakha
Support MXNet 1.3 with its training script format changes (#446)
23851b1
@nadiayaPiyush Adlakha
Make InputDataConfig optional for training. (#459)
bd1f43b
* Make InputDataConfig optional for training.
* Update boto3 dependency to make sure boto support no InputDataConfig.
* Update changelog.
* Add missing assertion for chainer failure script test.
@jesterhazyPiyush Adlakha
add tfs container support (#460)
2edcc3a
* add tensorflow serving container support
@jesterhazyPiyush Adlakha
simplify create_image_uri function (#462)
10b7e42
* add failure test case
* fix flaky assert
@jesterhazyPiyush Adlakha
bump version to 1.14.0 (#463)
e8bf717
@yangawsPiyush Adlakha
Skip gpu tests in regions without ml.p2.xlarge (#461)
eea2ad5
@pnpnpnPiyush Adlakha
Adding Object2Vec support to SageMaker Python SDK (#467)
c8147a9
@jesterhazyPiyush Adlakha
fix readme rendering (#464)
52d1ec4
@iquinteroPiyush Adlakha
Add Pylint (#465)
3a69cf6
Add Pylint checking. Fixed all the current errors and warnings,
any future PRs will fail if they introduce any pylint error/warnings.
@nadiayaPiyush Adlakha
Support optional input channels in local mode. (#466)
9a997a5
@iquinteroPiyush Adlakha
build: upgrade docker-compose to 1.23 (#470)
6a1d93c
docker-compose 1.23 has moved to a newer version of requests
which allows us to remove the upper bound on urllib3 that
was causing a lot of problems for everyone.
@jesterhazyPiyush Adlakha
add tensorflow serving docs (#468)
876287e
* add tensorflow serving docs
* add content_type to tensorflow.serving.Predictor
* support CustomAttributes in local mode
@leopdPiyush Adlakha
Very minor: Better documentation comment on DeferredError. (#469)
0c4ad9c
* Better documentation comment on DeferredError.
* Using Napolean-style docstring formatting for example code.
* Fixup flake8 trailing whitespace.
@laurenyu
laurenyu
commented
Nov 14, 2018

Copy link


Copy Markdown

Contributor
can you check the unit tests? four are failing with:
E RemovedInPytest4Warning: Fixture "sagemaker_session" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

@laurenyu
laurenyu
merged commit 5db2603
into
aws:masterNov 14, 2018
ChoiByungWook pushed a commit
that referenced
this pull request
Dec 8, 2020
@akhilmehra@ChoiByungWook
feature: add model parallelism support (#441)
bbb3f16
@icywang86ruiicywang86rui
mentioned this pull request
Jan 7, 2021

Closed

Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Bug fix for getting dataframes in TrainingJobAnalytics. (aws#441)
957026c
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
feature: add model parallelism support (aws#441)
5bf53f0
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


2 more reviewers

@laurenyulaurenyulaurenyu approved these changes
@mvsuspmvsuspmvsusp left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

12 participants

@piyushadlakha@codecov-io@laurenyu@mvsusp@RodrigoAtAWS@iquintero@pdasamzn@nadiaya@jesterhazy@yangaws@pnpnpn@leopd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Bug fix for getting dataframes in TrainingJobAnalytics. - #441
Merged
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Nov 14, 2018
Merged
Bug fix for getting dataframes in TrainingJobAnalytics.#441
laurenyu merged 30 commits into
aws:masterfrom 
piyushadlakha:TrainingJobAnalytics
Conversation
@piyushadlakha
@piyushadlakhapiyushadlakha
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Contributor
Issue #, if available:
Description of changes:

Bug fix for getting dataframes in TrainingJobAnalytics.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated the changelog with a description of my changes (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
laurenyu
laurenyu
reviewed
Oct 25, 2018
View reviewed changes
@laurenyulaurenyu
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a unit test?
Comment threadsrc/sagemaker/analytics.py
Outdated

 description = self._sage_client.describe_training_job(TrainingJobName=self.name)
 start_time = description[u'TrainingStartTime'] # datetime object
 end_time = description.get(u'TrainingEndTime', datetime.datetime.utcnow())
 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cloud watch/CloudWatch
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
Comment threadsrc/sagemaker/analytics.py
Outdated

 # Incrementing end time by 1 min since cloud watch drops seconds before finding the logs.
 # This results in logs being searched in the time range in which the correct log line was not present.
 # Example - Log time - 2018-10-22 08:25:55
 # Here calculated end time would also be 2018-10-22 08:25:55 (without 1 min addition)
@laurenyulaurenyuOct 25, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: there's an extra space after "be"
@piyushadlakhapiyushadlakhaOct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
@codecov-io
codecov-io
commented
Oct 25, 2018•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #441 into master will increase coverage by 0.27%.

The diff coverage is 94.18%.
Impacted file tree graph
@@ Coverage Diff @@## master #441 +/- ##
==========================================
+ Coverage 93.75% 94.02% +0.27% 
==========================================
Files 55 57 +2 Lines 4034 4269 +235 ==========================================
+ Hits 3782 4014 +232 - Misses 252 255 +3
Impacted FilesCoverage Δ
src/sagemaker/cli/tensorflow.py50% <ø> (ø)⬆️
src/sagemaker/transformer.py100% <ø> (ø)⬆️
src/sagemaker/cli/mxnet.py100% <ø> (ø)⬆️
src/sagemaker/fw_utils.py100% <100%> (ø)⬆️
src/sagemaker/predictor.py96.77% <100%> (+0.17%)⬆️
src/sagemaker/analytics.py91.94% <100%> (ø)⬆️
src/sagemaker/pytorch/estimator.py100% <100%> (ø)⬆️
src/sagemaker/tensorflow/predictor.py95.83% <100%> (ø)⬆️
src/sagemaker/chainer/estimator.py100% <100%> (ø)⬆️
src/sagemaker/amazon/kmeans.py100% <100%> (ø)⬆️
... and 20 more
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data

Powered by Codecov. Last update a5595af...25038d1. Read the comment docs.
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
Unit test for this are already in place. We are in all cases mocking the calls to CW client and that mock is independent of the namespace.
@laurenyu
laurenyu
commented
Oct 25, 2018

Copy link


Copy Markdown

Contributor
given the change in logic for calculating end time, it would be good to have a unit test for that
@piyushadlakha
piyushadlakha
commented
Oct 25, 2018

Copy link


Copy Markdown

ContributorAuthor
UT's added.
laurenyu
laurenyu
previously approved these changes
Oct 26, 2018
View reviewed changes
mvsusp
mvsusp
previously requested changes
Oct 26, 2018
View reviewed changes
@mvsuspmvsusp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check our PR guidelines in description of the PR. Make sure that you provided unit tests and updated the Changelog.
Thanks
Comment threadsrc/sagemaker/analytics.py
@laurenyulaurenyu
mentioned this pull request
Nov 13, 2018
 Merged

4 tasks
RodrigoAtAWSand others
added 16 commits
November 13, 2018 17:20
@RodrigoAtAWSPiyush Adlakha
Add model parameters to Estimator, and bump library version to 1.13.0 (…
c55e7ca
…#450)
* Add incremental training model parameters to Estimator, and bump library version to 1.13.0
* Update README.rst
Removed unnecessary comma from sentence.
* Update estimator.py
Removed duplicated line
@iquinteroPiyush Adlakha
local mode: improve training input/output (#449)
5bdb50a
output_path is now honored:
Can be either file:// or s3:// - This also changes the default
behavior of local mode to use the SDK provided default S3 bucket
if nothing is passed. This makes it easier for customers to create
models in SageMaker too since their Model Artifacts will already be
a tarfile in S3.
input_channel content_type is now honored in the same way as SageMaker. If it is not provided it is not passed to the container. Before we were always passing 'application/octet-stream'
@pdasamznPiyush Adlakha
Add image URIs for built-in Algorithms for SIN/LHR/BOM/SFO/YUL (#456)
3d103b4
@laurenyuPiyush Adlakha
Support MXNet 1.3 with its training script format changes (#446)
23851b1
@nadiayaPiyush Adlakha
Make InputDataConfig optional for training. (#459)
bd1f43b
* Make InputDataConfig optional for training.
* Update boto3 dependency to make sure boto support no InputDataConfig.
* Update changelog.
* Add missing assertion for chainer failure script test.
@jesterhazyPiyush Adlakha
add tfs container support (#460)
2edcc3a
* add tensorflow serving container support
@jesterhazyPiyush Adlakha
simplify create_image_uri function (#462)
10b7e42
* add failure test case
* fix flaky assert
@jesterhazyPiyush Adlakha
bump version to 1.14.0 (#463)
e8bf717
@yangawsPiyush Adlakha
Skip gpu tests in regions without ml.p2.xlarge (#461)
eea2ad5
@pnpnpnPiyush Adlakha
Adding Object2Vec support to SageMaker Python SDK (#467)
c8147a9
@jesterhazyPiyush Adlakha
fix readme rendering (#464)
52d1ec4
@iquinteroPiyush Adlakha
Add Pylint (#465)
3a69cf6
Add Pylint checking. Fixed all the current errors and warnings,
any future PRs will fail if they introduce any pylint error/warnings.
@nadiayaPiyush Adlakha
Support optional input channels in local mode. (#466)
9a997a5
@iquinteroPiyush Adlakha
build: upgrade docker-compose to 1.23 (#470)
6a1d93c
docker-compose 1.23 has moved to a newer version of requests
which allows us to remove the upper bound on urllib3 that
was causing a lot of problems for everyone.
@jesterhazyPiyush Adlakha
add tensorflow serving docs (#468)
876287e
* add tensorflow serving docs
* add content_type to tensorflow.serving.Predictor
* support CustomAttributes in local mode
@leopdPiyush Adlakha
Very minor: Better documentation comment on DeferredError. (#469)
0c4ad9c
* Better documentation comment on DeferredError.
* Using Napolean-style docstring formatting for example code.
* Fixup flake8 trailing whitespace.
@laurenyu
laurenyu
commented
Nov 14, 2018

Copy link


Copy Markdown

Contributor
can you check the unit tests? four are failing with:
E RemovedInPytest4Warning: Fixture "sagemaker_session" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.

@laurenyu
laurenyu
merged commit 5db2603
into
aws:masterNov 14, 2018
ChoiByungWook pushed a commit
that referenced
this pull request
Dec 8, 2020
@akhilmehra@ChoiByungWook
feature: add model parallelism support (#441)
bbb3f16
@icywang86ruiicywang86rui
mentioned this pull request
Jan 7, 2021

Closed

Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Bug fix for getting dataframes in TrainingJobAnalytics. (aws#441)
957026c
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
feature: add model parallelism support (aws#441)
5bf53f0
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


2 more reviewers

@laurenyulaurenyulaurenyu approved these changes
@mvsuspmvsuspmvsusp left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

12 participants

@piyushadlakha@codecov-io@laurenyu@mvsusp@RodrigoAtAWS@iquintero@pdasamzn@nadiaya@jesterhazy@yangaws@pnpnpn@leopd