feature: sts endpoint support - #784

Closed
jmgray24 wants to merge 10 commits into
aws:masterfrom
jmgray24:master
Closed

feature: sts endpoint support#784
jmgray24 wants to merge 10 commits into
aws:masterfrom
jmgray24:master

Conversation

@jmgray24

@jmgray24jmgray24 commented May 8, 2019

Copy link
Copy Markdown

Issue #, if available: N/A

Description of changes:

Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

  • I have read the CONTRIBUTING doc
  • I used the commit message format described in CONTRIBUTING
  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have updated any necessary documentation (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@jesterhazy

jesterhazy commented May 8, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jesterhazy

jesterhazy commented May 8, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jesterhazy

jesterhazy commented May 8, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jesterhazy

jesterhazy commented May 8, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jesterhazy

jesterhazy commented May 8, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jmgray24jmgray24 changed the title STS Endpoint Supportfeature: sts endpoint supportMay 9, 2019
@jmgray24

jmgray24 commented May 9, 2019

Copy link
Copy Markdown
Author

This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.

Thus ready for review.

@jesterhazy

jesterhazy commented May 9, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@imujjwal96

imujjwal96 commented May 10, 2019

Copy link
Copy Markdown
Contributor

The issue has been fixed in the PR #776
you may run the following commands to pull the latest commits:

git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

@jesterhazy

jesterhazy commented May 10, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jmgray24

jmgray24 commented May 10, 2019

Copy link
Copy Markdown
Author

Failure of test here

_________________________ test_horovod_local_mode[2-2] _________________________
[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python
sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>
instances = 2, processes = 2
tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')

@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
[1, 2],
(2, 1),
(2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
output_path = 'file://%s' % tmpdir
job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
role='SageMakerRole',
train_instance_count=2,
train_instance_type='local',
sagemaker_session=sagemaker_local_session,
py_version=integ.PYTHON_VERSION,
script_mode=True,
output_path=output_path,
framework_version='1.12',
distributions={'mpi': {'enabled': True,
'processes_per_host': processes}})
with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
estimator.fit(job_name=job_name)
tmp = str(tmpdir)
extract_files(output_path.replace('file://', ''), tmp)
size = instances * processes
for rank in range(size):

 assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:

file = 'rank-0'
tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'

def read_json(file, tmp):

 with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'

tests/integ/test_horovod.py:102: FileNotFoundError
----------------------------- Captured stdout call -----------------------------

@jesterhazy

jesterhazy commented May 13, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jesterhazy

jesterhazy commented May 13, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jesterhazy

jesterhazy commented May 14, 2019

Copy link
Copy Markdown
Contributor

AWS CodeBuild CI Report

  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jmgray24

jmgray24 commented May 14, 2019

Copy link
Copy Markdown
Author

Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jmgray24@jesterhazy@imujjwal96
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
feature: sts endpoint support - #784
Closed
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Closed
feature: sts endpoint support#784
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Conversation
@jmgray24
@jmgray24jmgray24
commented
May 8, 2019

Copy link


Copy Markdown

Issue #, if available: N/A
Description of changes:
Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I used the commit message format described in CONTRIBUTING
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24jmgray24
changed the title
STS Endpoint Supportfeature: sts endpoint supportMay 9, 2019
@jmgray24
jmgray24
commented
May 9, 2019

Copy link


Copy Markdown

Author
This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.
Thus ready for review.
@jesterhazy
jesterhazy
commented
May 9, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@imujjwal96
imujjwal96
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
The issue has been fixed in the PR #776

you may run the following commands to pull the latest commits:
git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

@jesterhazy
jesterhazy
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 10, 2019

Copy link


Copy Markdown

Author
Failure of test here
_________________________ test_horovod_local_mode[2-2] _________________________

[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python

sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>

instances = 2, processes = 2

tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')
@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
[1, 2],
(2, 1),
(2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
output_path = 'file://%s' % tmpdir
job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
role='SageMakerRole',
train_instance_count=2,
train_instance_type='local',
sagemaker_session=sagemaker_local_session,
py_version=integ.PYTHON_VERSION,
script_mode=True,
output_path=output_path,
framework_version='1.12',
distributions={'mpi': {'enabled': True,
'processes_per_host': processes}})
with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
estimator.fit(job_name=job_name)
tmp = str(tmpdir)
extract_files(output_path.replace('file://', ''), tmp)
size = instances * processes
for rank in range(size):

 assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:
file = 'rank-0'

tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'
def read_json(file, tmp):

 with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'
tests/integ/test_horovod.py:102: FileNotFoundError

----------------------------- Captured stdout call -----------------------------
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 14, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 14, 2019

Copy link


Copy Markdown

Author
Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jmgray24@jesterhazy@imujjwal96

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feature: sts endpoint support - #784
Closed
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Closed
feature: sts endpoint support#784
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Conversation
@jmgray24
@jmgray24jmgray24
commented
May 8, 2019

Copy link


Copy Markdown

Issue #, if available: N/A
Description of changes:
Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I used the commit message format described in CONTRIBUTING
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24jmgray24
changed the title
STS Endpoint Supportfeature: sts endpoint supportMay 9, 2019
@jmgray24
jmgray24
commented
May 9, 2019

Copy link


Copy Markdown

Author
This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.
Thus ready for review.
@jesterhazy
jesterhazy
commented
May 9, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@imujjwal96
imujjwal96
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
The issue has been fixed in the PR #776

you may run the following commands to pull the latest commits:
git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

@jesterhazy
jesterhazy
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 10, 2019

Copy link


Copy Markdown

Author
Failure of test here
_________________________ test_horovod_local_mode[2-2] _________________________

[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python

sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>

instances = 2, processes = 2

tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')
@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
[1, 2],
(2, 1),
(2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
output_path = 'file://%s' % tmpdir
job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
role='SageMakerRole',
train_instance_count=2,
train_instance_type='local',
sagemaker_session=sagemaker_local_session,
py_version=integ.PYTHON_VERSION,
script_mode=True,
output_path=output_path,
framework_version='1.12',
distributions={'mpi': {'enabled': True,
'processes_per_host': processes}})
with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
estimator.fit(job_name=job_name)
tmp = str(tmpdir)
extract_files(output_path.replace('file://', ''), tmp)
size = instances * processes
for rank in range(size):

 assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:
file = 'rank-0'

tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'
def read_json(file, tmp):

 with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'
tests/integ/test_horovod.py:102: FileNotFoundError

----------------------------- Captured stdout call -----------------------------
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 14, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 14, 2019

Copy link


Copy Markdown

Author
Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jmgray24@jesterhazy@imujjwal96

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feature: sts endpoint support - #784
Closed
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Closed
feature: sts endpoint support#784
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Conversation
@jmgray24
@jmgray24jmgray24
commented
May 8, 2019

Copy link


Copy Markdown

Issue #, if available: N/A
Description of changes:
Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I used the commit message format described in CONTRIBUTING
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24jmgray24
changed the title
STS Endpoint Supportfeature: sts endpoint supportMay 9, 2019
@jmgray24
jmgray24
commented
May 9, 2019

Copy link


Copy Markdown

Author
This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.
Thus ready for review.
@jesterhazy
jesterhazy
commented
May 9, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@imujjwal96
imujjwal96
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
The issue has been fixed in the PR #776

you may run the following commands to pull the latest commits:
git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

@jesterhazy
jesterhazy
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 10, 2019

Copy link


Copy Markdown

Author
Failure of test here
_________________________ test_horovod_local_mode[2-2] _________________________

[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python

sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>

instances = 2, processes = 2

tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')
@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
[1, 2],
(2, 1),
(2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
output_path = 'file://%s' % tmpdir
job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
role='SageMakerRole',
train_instance_count=2,
train_instance_type='local',
sagemaker_session=sagemaker_local_session,
py_version=integ.PYTHON_VERSION,
script_mode=True,
output_path=output_path,
framework_version='1.12',
distributions={'mpi': {'enabled': True,
'processes_per_host': processes}})
with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
estimator.fit(job_name=job_name)
tmp = str(tmpdir)
extract_files(output_path.replace('file://', ''), tmp)
size = instances * processes
for rank in range(size):

 assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:
file = 'rank-0'

tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'
def read_json(file, tmp):

 with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'
tests/integ/test_horovod.py:102: FileNotFoundError

----------------------------- Captured stdout call -----------------------------
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 14, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 14, 2019

Copy link


Copy Markdown

Author
Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jmgray24@jesterhazy@imujjwal96

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
feature: sts endpoint support - #784
Closed
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Closed
feature: sts endpoint support#784
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Conversation
@jmgray24
@jmgray24jmgray24
commented
May 8, 2019

Copy link


Copy Markdown

Issue #, if available: N/A
Description of changes:
Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I used the commit message format described in CONTRIBUTING
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24jmgray24
changed the title
STS Endpoint Supportfeature: sts endpoint supportMay 9, 2019
@jmgray24
jmgray24
commented
May 9, 2019

Copy link


Copy Markdown

Author
This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.
Thus ready for review.
@jesterhazy
jesterhazy
commented
May 9, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@imujjwal96
imujjwal96
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
The issue has been fixed in the PR #776

you may run the following commands to pull the latest commits:
git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

@jesterhazy
jesterhazy
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 10, 2019

Copy link


Copy Markdown

Author
Failure of test here
_________________________ test_horovod_local_mode[2-2] _________________________

[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python

sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>

instances = 2, processes = 2

tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')
@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
[1, 2],
(2, 1),
(2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
output_path = 'file://%s' % tmpdir
job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
role='SageMakerRole',
train_instance_count=2,
train_instance_type='local',
sagemaker_session=sagemaker_local_session,
py_version=integ.PYTHON_VERSION,
script_mode=True,
output_path=output_path,
framework_version='1.12',
distributions={'mpi': {'enabled': True,
'processes_per_host': processes}})
with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
estimator.fit(job_name=job_name)
tmp = str(tmpdir)
extract_files(output_path.replace('file://', ''), tmp)
size = instances * processes
for rank in range(size):

 assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:
file = 'rank-0'

tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'
def read_json(file, tmp):

 with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'
tests/integ/test_horovod.py:102: FileNotFoundError

----------------------------- Captured stdout call -----------------------------
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 14, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 14, 2019

Copy link


Copy Markdown

Author
Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jmgray24@jesterhazy@imujjwal96

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feature: sts endpoint support - #784
Closed
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Closed
feature: sts endpoint support#784
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Conversation
@jmgray24
@jmgray24jmgray24
commented
May 8, 2019

Copy link


Copy Markdown

Issue #, if available: N/A
Description of changes:
Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I used the commit message format described in CONTRIBUTING
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24jmgray24
changed the title
STS Endpoint Supportfeature: sts endpoint supportMay 9, 2019
@jmgray24
jmgray24
commented
May 9, 2019

Copy link


Copy Markdown

Author
This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.
Thus ready for review.
@jesterhazy
jesterhazy
commented
May 9, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@imujjwal96
imujjwal96
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
The issue has been fixed in the PR #776

you may run the following commands to pull the latest commits:
git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

@jesterhazy
jesterhazy
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 10, 2019

Copy link


Copy Markdown

Author
Failure of test here
_________________________ test_horovod_local_mode[2-2] _________________________

[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python

sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>

instances = 2, processes = 2

tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')
@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
[1, 2],
(2, 1),
(2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
output_path = 'file://%s' % tmpdir
job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
role='SageMakerRole',
train_instance_count=2,
train_instance_type='local',
sagemaker_session=sagemaker_local_session,
py_version=integ.PYTHON_VERSION,
script_mode=True,
output_path=output_path,
framework_version='1.12',
distributions={'mpi': {'enabled': True,
'processes_per_host': processes}})
with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
estimator.fit(job_name=job_name)
tmp = str(tmpdir)
extract_files(output_path.replace('file://', ''), tmp)
size = instances * processes
for rank in range(size):

 assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:
file = 'rank-0'

tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'
def read_json(file, tmp):

 with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'
tests/integ/test_horovod.py:102: FileNotFoundError

----------------------------- Captured stdout call -----------------------------
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 14, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 14, 2019

Copy link


Copy Markdown

Author
Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jmgray24@jesterhazy@imujjwal96

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feature: sts endpoint support - #784
Closed
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Closed
feature: sts endpoint support#784
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Conversation
@jmgray24
@jmgray24jmgray24
commented
May 8, 2019

Copy link


Copy Markdown

Issue #, if available: N/A
Description of changes:
Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I used the commit message format described in CONTRIBUTING
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24jmgray24
changed the title
STS Endpoint Supportfeature: sts endpoint supportMay 9, 2019
@jmgray24
jmgray24
commented
May 9, 2019

Copy link


Copy Markdown

Author
This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.
Thus ready for review.
@jesterhazy
jesterhazy
commented
May 9, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@imujjwal96
imujjwal96
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
The issue has been fixed in the PR #776

you may run the following commands to pull the latest commits:
git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

@jesterhazy
jesterhazy
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 10, 2019

Copy link


Copy Markdown

Author
Failure of test here
_________________________ test_horovod_local_mode[2-2] _________________________

[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python

sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>

instances = 2, processes = 2

tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')
@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
[1, 2],
(2, 1),
(2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
output_path = 'file://%s' % tmpdir
job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
role='SageMakerRole',
train_instance_count=2,
train_instance_type='local',
sagemaker_session=sagemaker_local_session,
py_version=integ.PYTHON_VERSION,
script_mode=True,
output_path=output_path,
framework_version='1.12',
distributions={'mpi': {'enabled': True,
'processes_per_host': processes}})
with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
estimator.fit(job_name=job_name)
tmp = str(tmpdir)
extract_files(output_path.replace('file://', ''), tmp)
size = instances * processes
for rank in range(size):

 assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:
file = 'rank-0'

tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'
def read_json(file, tmp):

 with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'
tests/integ/test_horovod.py:102: FileNotFoundError

----------------------------- Captured stdout call -----------------------------
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 14, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 14, 2019

Copy link


Copy Markdown

Author
Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jmgray24@jesterhazy@imujjwal96

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
feature: sts endpoint support - #784
Closed
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Closed
feature: sts endpoint support#784
jmgray24 wants to merge 10 commits into
aws:masterfrom 
jmgray24:master
Conversation
@jmgray24
@jmgray24jmgray24
commented
May 8, 2019

Copy link


Copy Markdown

Issue #, if available: N/A
Description of changes:
Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
  •  I have read the CONTRIBUTING doc
  •  I used the commit message format described in CONTRIBUTING
  •  I have added tests that prove my fix is effective or that my feature works (if appropriate)
  •  I have updated any necessary documentation (if appropriate)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 8, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24jmgray24
changed the title
STS Endpoint Supportfeature: sts endpoint supportMay 9, 2019
@jmgray24
jmgray24
commented
May 9, 2019

Copy link


Copy Markdown

Author
This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.
Thus ready for review.
@jesterhazy
jesterhazy
commented
May 9, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@imujjwal96
imujjwal96
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
The issue has been fixed in the PR #776

you may run the following commands to pull the latest commits:
git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

@jesterhazy
jesterhazy
commented
May 10, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 10, 2019

Copy link


Copy Markdown

Author
Failure of test here
_________________________ test_horovod_local_mode[2-2] _________________________

[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python

sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>

instances = 2, processes = 2

tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')
@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
[1, 2],
(2, 1),
(2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
output_path = 'file://%s' % tmpdir
job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
role='SageMakerRole',
train_instance_count=2,
train_instance_type='local',
sagemaker_session=sagemaker_local_session,
py_version=integ.PYTHON_VERSION,
script_mode=True,
output_path=output_path,
framework_version='1.12',
distributions={'mpi': {'enabled': True,
'processes_per_host': processes}})
with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
estimator.fit(job_name=job_name)
tmp = str(tmpdir)
extract_files(output_path.replace('file://', ''), tmp)
size = instances * processes
for rank in range(size):

 assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:
file = 'rank-0'

tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'
def read_json(file, tmp):

 with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'
tests/integ/test_horovod.py:102: FileNotFoundError

----------------------------- Captured stdout call -----------------------------
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 13, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jesterhazy
jesterhazy
commented
May 14, 2019

Copy link


Copy Markdown

Contributor
AWS CodeBuild CI Report
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository
@jmgray24
jmgray24
commented
May 14, 2019

Copy link


Copy Markdown

Author
Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jmgray24@jesterhazy@imujjwal96