Correctly handle TooManyBuckets error_code in default_bucket method - #40

Merged
owen-t merged 2 commits into
aws:masterfrom
jalabort:patch-1
Jan 12, 2018
Merged

Correctly handle TooManyBuckets error_code in default_bucket method#40
owen-t merged 2 commits into
aws:masterfrom
jalabort:patch-1

Conversation

@jalabort

@jalabortjalabort commented Jan 10, 2018

Copy link
Copy Markdown
Contributor

At my organisation we have reached our maximum number of allowed buckets on s3 and as a direct consequence of that we started getting the following error when using the sagemaker-python-sdk:

INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/bin/sagemaker", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/cli/bin/sagemaker", line 6, in <module>
fire.Fire(component=SageMakerRunner)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 314, in train
self._train(source_local_path, dataset_s3_path, job_name)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 364, in _train
run_tensorboard_locally=self._run_tensorboard_locally
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 162, in fit
fit_super()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 154, in fit_super
super(TensorFlow, self).fit(inputs, wait, logs, job_name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/estimator.py", line 498, in fit
code_bucket = self.sagemaker_session.default_bucket()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/session.py", line 159, in default_bucket
s3.create_bucket(Bucket=default_bucket)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

This little patch seems to take care of this issue.

I could not find much information in terms of the expected PR structure so I hope this suffices, happy to make the pertinent changes if it does not.

@jalabort
Update session.py
922dde0
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
@jalabortjalabort changed the title Correctly handle the TooManyBuckets error_code in default_bucketCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
@jalabortjalabort changed the title Correctly handle TooManyBuckets error_code in default_bucket methodCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
owen-t
owen-t previously approved these changes Jan 11, 2018
View reviewed changes

@owen-towen-t left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your submission!

This looks good to me.

Comment threadsrc/sagemaker/session.py Outdated
elif error_code == 'TooManyBuckets':
try:
s3.meta.client.head_bucket(Bucket=default_bucket)
LOGGER.info('S3 bucket {} already exists'.format(

@owen-towen-tJan 11, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this logging message, in the normal case we don't log this.

Comment threadsrc/sagemaker/session.py
pass
elif error_code == 'TooManyBuckets':
try:
s3.meta.client.head_bucket(Bucket=default_bucket)

@owen-towen-tJan 11, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little esoteric - if we remove the logging below, can you add some developer documentation:

# Succeed if the default bucket exists

@jalabort
Update session.py
40344b5
Remove logging message
@jalabort

jalabort commented Jan 11, 2018•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@owen-t Addressed your comments above.

@jalabort

jalabort commented Jan 12, 2018

Copy link
Copy Markdown
ContributorAuthor

Is there something I need to amend due to the failing teamcity build? I cannot see what's wrong with it.

@ChoiByungWook

ChoiByungWook commented Jan 12, 2018

Copy link
Copy Markdown
Contributor

Hello,
We don't publish our build logs externally. A test failed, but the issue is resolved and we're rerunning your build. We will merge it in once the build is complete. Thanks again for your contribution!

@owen-t
owen-t merged commit f43b9c3 into aws:masterJan 12, 2018
apacker pushed a commit to apacker/sagemaker-python-sdk that referenced this pull request Nov 15, 2018
@vrkhare
Merge pull requestaws#40from awslabs/xgboost-updates
d8e1060
removing references to auto-scaling
athewsey added a commit to athewsey/sagemaker-python-sdk that referenced this pull request Jul 16, 2021
@athewsey
Merge remote-tracking branch 'upstream/master'
4ded19c
Merge pull request aws#40 from athewsey/feat/fw-processor
Evan-W-ang added a commit to Evan-W-ang/sagemaker-python-sdk that referenced this pull request Jun 8, 2026
@Evan-W-ang
Correctly handle TooManyBuckets error_code in default_bucket method (a…
caa27a0
…ws#40)
* Update session.py
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@owen-towen-towen-t approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jalabort@ChoiByungWook@owen-t
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Correctly handle TooManyBuckets error_code in default_bucket method - #40
Merged
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Jan 12, 2018
Merged
Correctly handle TooManyBuckets error_code in default_bucket method#40
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Conversation
@jalabort
@jalabortjalabort
commented
Jan 10, 2018

Copy link


Copy Markdown

Contributor
At my organisation we have reached our maximum number of allowed buckets on s3 and as a direct consequence of that we started getting the following error when using the sagemaker-python-sdk:
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/bin/sagemaker", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/cli/bin/sagemaker", line 6, in <module>
fire.Fire(component=SageMakerRunner)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 314, in train
self._train(source_local_path, dataset_s3_path, job_name)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 364, in _train
run_tensorboard_locally=self._run_tensorboard_locally
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 162, in fit
fit_super()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 154, in fit_super
super(TensorFlow, self).fit(inputs, wait, logs, job_name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/estimator.py", line 498, in fit
code_bucket = self.sagemaker_session.default_bucket()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/session.py", line 159, in default_bucket
s3.create_bucket(Bucket=default_bucket)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

This little patch seems to take care of this issue.
I could not find much information in terms of the expected PR structure so I hope this suffices, happy to make the pertinent changes if it does not.
@jalabort
Update session.py
922dde0
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
@jalabortjalabort
changed the title
Correctly handle the TooManyBuckets error_code in default_bucketCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
@jalabortjalabort
changed the title
Correctly handle TooManyBuckets error_code in default_bucket methodCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
owen-t
owen-t
previously approved these changes
Jan 11, 2018
View reviewed changes
@owen-towen-t
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your submission!
This looks good to me.
Comment threadsrc/sagemaker/session.py
Outdated

 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
 LOGGER.info('S3 bucket {} already exists'.format(
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this logging message, in the normal case we don't log this.
Comment threadsrc/sagemaker/session.py
 pass
 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little esoteric - if we remove the logging below, can you add some developer documentation:
# Succeed if the default bucket exists
@jalabort
Update session.py
40344b5
Remove logging message
@jalabort
jalabort
commented
Jan 11, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@owen-t Addressed your comments above.
@jalabort
jalabort
commented
Jan 12, 2018

Copy link


Copy Markdown

ContributorAuthor
Is there something I need to amend due to the failing teamcity build? I cannot see what's wrong with it.
@ChoiByungWook
ChoiByungWook
commented
Jan 12, 2018

Copy link


Copy Markdown

Contributor
Hello,

We don't publish our build logs externally. A test failed, but the issue is resolved and we're rerunning your build. We will merge it in once the build is complete. Thanks again for your contribution!
@owen-t
owen-t
merged commit f43b9c3
into
aws:masterJan 12, 2018
apacker pushed a commit
to apacker/sagemaker-python-sdk
that referenced
this pull request
Nov 15, 2018
@vrkhare
Merge pull requestaws#40from awslabs/xgboost-updates
d8e1060
removing references to auto-scaling
athewsey added a commit
to athewsey/sagemaker-python-sdk
that referenced
this pull request
Jul 16, 2021
@athewsey
Merge remote-tracking branch 'upstream/master'
4ded19c
Merge pull request aws#40 from athewsey/feat/fw-processor
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Correctly handle TooManyBuckets error_code in default_bucket method (a…
caa27a0
…ws#40)
* Update session.py
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


1 more reviewer

@owen-towen-towen-t approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jalabort@ChoiByungWook@owen-t

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Correctly handle TooManyBuckets error_code in default_bucket method - #40
Merged
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Jan 12, 2018
Merged
Correctly handle TooManyBuckets error_code in default_bucket method#40
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Conversation
@jalabort
@jalabortjalabort
commented
Jan 10, 2018

Copy link


Copy Markdown

Contributor
At my organisation we have reached our maximum number of allowed buckets on s3 and as a direct consequence of that we started getting the following error when using the sagemaker-python-sdk:
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/bin/sagemaker", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/cli/bin/sagemaker", line 6, in <module>
fire.Fire(component=SageMakerRunner)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 314, in train
self._train(source_local_path, dataset_s3_path, job_name)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 364, in _train
run_tensorboard_locally=self._run_tensorboard_locally
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 162, in fit
fit_super()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 154, in fit_super
super(TensorFlow, self).fit(inputs, wait, logs, job_name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/estimator.py", line 498, in fit
code_bucket = self.sagemaker_session.default_bucket()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/session.py", line 159, in default_bucket
s3.create_bucket(Bucket=default_bucket)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

This little patch seems to take care of this issue.
I could not find much information in terms of the expected PR structure so I hope this suffices, happy to make the pertinent changes if it does not.
@jalabort
Update session.py
922dde0
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
@jalabortjalabort
changed the title
Correctly handle the TooManyBuckets error_code in default_bucketCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
@jalabortjalabort
changed the title
Correctly handle TooManyBuckets error_code in default_bucket methodCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
owen-t
owen-t
previously approved these changes
Jan 11, 2018
View reviewed changes
@owen-towen-t
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your submission!
This looks good to me.
Comment threadsrc/sagemaker/session.py
Outdated

 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
 LOGGER.info('S3 bucket {} already exists'.format(
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this logging message, in the normal case we don't log this.
Comment threadsrc/sagemaker/session.py
 pass
 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little esoteric - if we remove the logging below, can you add some developer documentation:
# Succeed if the default bucket exists
@jalabort
Update session.py
40344b5
Remove logging message
@jalabort
jalabort
commented
Jan 11, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@owen-t Addressed your comments above.
@jalabort
jalabort
commented
Jan 12, 2018

Copy link


Copy Markdown

ContributorAuthor
Is there something I need to amend due to the failing teamcity build? I cannot see what's wrong with it.
@ChoiByungWook
ChoiByungWook
commented
Jan 12, 2018

Copy link


Copy Markdown

Contributor
Hello,

We don't publish our build logs externally. A test failed, but the issue is resolved and we're rerunning your build. We will merge it in once the build is complete. Thanks again for your contribution!
@owen-t
owen-t
merged commit f43b9c3
into
aws:masterJan 12, 2018
apacker pushed a commit
to apacker/sagemaker-python-sdk
that referenced
this pull request
Nov 15, 2018
@vrkhare
Merge pull requestaws#40from awslabs/xgboost-updates
d8e1060
removing references to auto-scaling
athewsey added a commit
to athewsey/sagemaker-python-sdk
that referenced
this pull request
Jul 16, 2021
@athewsey
Merge remote-tracking branch 'upstream/master'
4ded19c
Merge pull request aws#40 from athewsey/feat/fw-processor
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Correctly handle TooManyBuckets error_code in default_bucket method (a…
caa27a0
…ws#40)
* Update session.py
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


1 more reviewer

@owen-towen-towen-t approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jalabort@ChoiByungWook@owen-t

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Correctly handle TooManyBuckets error_code in default_bucket method - #40
Merged
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Jan 12, 2018
Merged
Correctly handle TooManyBuckets error_code in default_bucket method#40
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Conversation
@jalabort
@jalabortjalabort
commented
Jan 10, 2018

Copy link


Copy Markdown

Contributor
At my organisation we have reached our maximum number of allowed buckets on s3 and as a direct consequence of that we started getting the following error when using the sagemaker-python-sdk:
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/bin/sagemaker", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/cli/bin/sagemaker", line 6, in <module>
fire.Fire(component=SageMakerRunner)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 314, in train
self._train(source_local_path, dataset_s3_path, job_name)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 364, in _train
run_tensorboard_locally=self._run_tensorboard_locally
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 162, in fit
fit_super()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 154, in fit_super
super(TensorFlow, self).fit(inputs, wait, logs, job_name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/estimator.py", line 498, in fit
code_bucket = self.sagemaker_session.default_bucket()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/session.py", line 159, in default_bucket
s3.create_bucket(Bucket=default_bucket)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

This little patch seems to take care of this issue.
I could not find much information in terms of the expected PR structure so I hope this suffices, happy to make the pertinent changes if it does not.
@jalabort
Update session.py
922dde0
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
@jalabortjalabort
changed the title
Correctly handle the TooManyBuckets error_code in default_bucketCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
@jalabortjalabort
changed the title
Correctly handle TooManyBuckets error_code in default_bucket methodCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
owen-t
owen-t
previously approved these changes
Jan 11, 2018
View reviewed changes
@owen-towen-t
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your submission!
This looks good to me.
Comment threadsrc/sagemaker/session.py
Outdated

 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
 LOGGER.info('S3 bucket {} already exists'.format(
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this logging message, in the normal case we don't log this.
Comment threadsrc/sagemaker/session.py
 pass
 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little esoteric - if we remove the logging below, can you add some developer documentation:
# Succeed if the default bucket exists
@jalabort
Update session.py
40344b5
Remove logging message
@jalabort
jalabort
commented
Jan 11, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@owen-t Addressed your comments above.
@jalabort
jalabort
commented
Jan 12, 2018

Copy link


Copy Markdown

ContributorAuthor
Is there something I need to amend due to the failing teamcity build? I cannot see what's wrong with it.
@ChoiByungWook
ChoiByungWook
commented
Jan 12, 2018

Copy link


Copy Markdown

Contributor
Hello,

We don't publish our build logs externally. A test failed, but the issue is resolved and we're rerunning your build. We will merge it in once the build is complete. Thanks again for your contribution!
@owen-t
owen-t
merged commit f43b9c3
into
aws:masterJan 12, 2018
apacker pushed a commit
to apacker/sagemaker-python-sdk
that referenced
this pull request
Nov 15, 2018
@vrkhare
Merge pull requestaws#40from awslabs/xgboost-updates
d8e1060
removing references to auto-scaling
athewsey added a commit
to athewsey/sagemaker-python-sdk
that referenced
this pull request
Jul 16, 2021
@athewsey
Merge remote-tracking branch 'upstream/master'
4ded19c
Merge pull request aws#40 from athewsey/feat/fw-processor
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Correctly handle TooManyBuckets error_code in default_bucket method (a…
caa27a0
…ws#40)
* Update session.py
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


1 more reviewer

@owen-towen-towen-t approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jalabort@ChoiByungWook@owen-t

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Correctly handle TooManyBuckets error_code in default_bucket method - #40
Merged
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Jan 12, 2018
Merged
Correctly handle TooManyBuckets error_code in default_bucket method#40
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Conversation
@jalabort
@jalabortjalabort
commented
Jan 10, 2018

Copy link


Copy Markdown

Contributor
At my organisation we have reached our maximum number of allowed buckets on s3 and as a direct consequence of that we started getting the following error when using the sagemaker-python-sdk:
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/bin/sagemaker", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/cli/bin/sagemaker", line 6, in <module>
fire.Fire(component=SageMakerRunner)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 314, in train
self._train(source_local_path, dataset_s3_path, job_name)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 364, in _train
run_tensorboard_locally=self._run_tensorboard_locally
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 162, in fit
fit_super()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 154, in fit_super
super(TensorFlow, self).fit(inputs, wait, logs, job_name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/estimator.py", line 498, in fit
code_bucket = self.sagemaker_session.default_bucket()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/session.py", line 159, in default_bucket
s3.create_bucket(Bucket=default_bucket)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

This little patch seems to take care of this issue.
I could not find much information in terms of the expected PR structure so I hope this suffices, happy to make the pertinent changes if it does not.
@jalabort
Update session.py
922dde0
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
@jalabortjalabort
changed the title
Correctly handle the TooManyBuckets error_code in default_bucketCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
@jalabortjalabort
changed the title
Correctly handle TooManyBuckets error_code in default_bucket methodCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
owen-t
owen-t
previously approved these changes
Jan 11, 2018
View reviewed changes
@owen-towen-t
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your submission!
This looks good to me.
Comment threadsrc/sagemaker/session.py
Outdated

 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
 LOGGER.info('S3 bucket {} already exists'.format(
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this logging message, in the normal case we don't log this.
Comment threadsrc/sagemaker/session.py
 pass
 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little esoteric - if we remove the logging below, can you add some developer documentation:
# Succeed if the default bucket exists
@jalabort
Update session.py
40344b5
Remove logging message
@jalabort
jalabort
commented
Jan 11, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@owen-t Addressed your comments above.
@jalabort
jalabort
commented
Jan 12, 2018

Copy link


Copy Markdown

ContributorAuthor
Is there something I need to amend due to the failing teamcity build? I cannot see what's wrong with it.
@ChoiByungWook
ChoiByungWook
commented
Jan 12, 2018

Copy link


Copy Markdown

Contributor
Hello,

We don't publish our build logs externally. A test failed, but the issue is resolved and we're rerunning your build. We will merge it in once the build is complete. Thanks again for your contribution!
@owen-t
owen-t
merged commit f43b9c3
into
aws:masterJan 12, 2018
apacker pushed a commit
to apacker/sagemaker-python-sdk
that referenced
this pull request
Nov 15, 2018
@vrkhare
Merge pull requestaws#40from awslabs/xgboost-updates
d8e1060
removing references to auto-scaling
athewsey added a commit
to athewsey/sagemaker-python-sdk
that referenced
this pull request
Jul 16, 2021
@athewsey
Merge remote-tracking branch 'upstream/master'
4ded19c
Merge pull request aws#40 from athewsey/feat/fw-processor
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Correctly handle TooManyBuckets error_code in default_bucket method (a…
caa27a0
…ws#40)
* Update session.py
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


1 more reviewer

@owen-towen-towen-t approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jalabort@ChoiByungWook@owen-t

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Correctly handle TooManyBuckets error_code in default_bucket method - #40
Merged
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Jan 12, 2018
Merged
Correctly handle TooManyBuckets error_code in default_bucket method#40
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Conversation
@jalabort
@jalabortjalabort
commented
Jan 10, 2018

Copy link


Copy Markdown

Contributor
At my organisation we have reached our maximum number of allowed buckets on s3 and as a direct consequence of that we started getting the following error when using the sagemaker-python-sdk:
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/bin/sagemaker", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/cli/bin/sagemaker", line 6, in <module>
fire.Fire(component=SageMakerRunner)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 314, in train
self._train(source_local_path, dataset_s3_path, job_name)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 364, in _train
run_tensorboard_locally=self._run_tensorboard_locally
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 162, in fit
fit_super()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 154, in fit_super
super(TensorFlow, self).fit(inputs, wait, logs, job_name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/estimator.py", line 498, in fit
code_bucket = self.sagemaker_session.default_bucket()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/session.py", line 159, in default_bucket
s3.create_bucket(Bucket=default_bucket)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

This little patch seems to take care of this issue.
I could not find much information in terms of the expected PR structure so I hope this suffices, happy to make the pertinent changes if it does not.
@jalabort
Update session.py
922dde0
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
@jalabortjalabort
changed the title
Correctly handle the TooManyBuckets error_code in default_bucketCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
@jalabortjalabort
changed the title
Correctly handle TooManyBuckets error_code in default_bucket methodCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
owen-t
owen-t
previously approved these changes
Jan 11, 2018
View reviewed changes
@owen-towen-t
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your submission!
This looks good to me.
Comment threadsrc/sagemaker/session.py
Outdated

 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
 LOGGER.info('S3 bucket {} already exists'.format(
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this logging message, in the normal case we don't log this.
Comment threadsrc/sagemaker/session.py
 pass
 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little esoteric - if we remove the logging below, can you add some developer documentation:
# Succeed if the default bucket exists
@jalabort
Update session.py
40344b5
Remove logging message
@jalabort
jalabort
commented
Jan 11, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@owen-t Addressed your comments above.
@jalabort
jalabort
commented
Jan 12, 2018

Copy link


Copy Markdown

ContributorAuthor
Is there something I need to amend due to the failing teamcity build? I cannot see what's wrong with it.
@ChoiByungWook
ChoiByungWook
commented
Jan 12, 2018

Copy link


Copy Markdown

Contributor
Hello,

We don't publish our build logs externally. A test failed, but the issue is resolved and we're rerunning your build. We will merge it in once the build is complete. Thanks again for your contribution!
@owen-t
owen-t
merged commit f43b9c3
into
aws:masterJan 12, 2018
apacker pushed a commit
to apacker/sagemaker-python-sdk
that referenced
this pull request
Nov 15, 2018
@vrkhare
Merge pull requestaws#40from awslabs/xgboost-updates
d8e1060
removing references to auto-scaling
athewsey added a commit
to athewsey/sagemaker-python-sdk
that referenced
this pull request
Jul 16, 2021
@athewsey
Merge remote-tracking branch 'upstream/master'
4ded19c
Merge pull request aws#40 from athewsey/feat/fw-processor
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Correctly handle TooManyBuckets error_code in default_bucket method (a…
caa27a0
…ws#40)
* Update session.py
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


1 more reviewer

@owen-towen-towen-t approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jalabort@ChoiByungWook@owen-t

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Correctly handle TooManyBuckets error_code in default_bucket method - #40
Merged
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Jan 12, 2018
Merged
Correctly handle TooManyBuckets error_code in default_bucket method#40
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Conversation
@jalabort
@jalabortjalabort
commented
Jan 10, 2018

Copy link


Copy Markdown

Contributor
At my organisation we have reached our maximum number of allowed buckets on s3 and as a direct consequence of that we started getting the following error when using the sagemaker-python-sdk:
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/bin/sagemaker", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/cli/bin/sagemaker", line 6, in <module>
fire.Fire(component=SageMakerRunner)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 314, in train
self._train(source_local_path, dataset_s3_path, job_name)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 364, in _train
run_tensorboard_locally=self._run_tensorboard_locally
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 162, in fit
fit_super()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 154, in fit_super
super(TensorFlow, self).fit(inputs, wait, logs, job_name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/estimator.py", line 498, in fit
code_bucket = self.sagemaker_session.default_bucket()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/session.py", line 159, in default_bucket
s3.create_bucket(Bucket=default_bucket)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

This little patch seems to take care of this issue.
I could not find much information in terms of the expected PR structure so I hope this suffices, happy to make the pertinent changes if it does not.
@jalabort
Update session.py
922dde0
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
@jalabortjalabort
changed the title
Correctly handle the TooManyBuckets error_code in default_bucketCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
@jalabortjalabort
changed the title
Correctly handle TooManyBuckets error_code in default_bucket methodCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
owen-t
owen-t
previously approved these changes
Jan 11, 2018
View reviewed changes
@owen-towen-t
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your submission!
This looks good to me.
Comment threadsrc/sagemaker/session.py
Outdated

 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
 LOGGER.info('S3 bucket {} already exists'.format(
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this logging message, in the normal case we don't log this.
Comment threadsrc/sagemaker/session.py
 pass
 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little esoteric - if we remove the logging below, can you add some developer documentation:
# Succeed if the default bucket exists
@jalabort
Update session.py
40344b5
Remove logging message
@jalabort
jalabort
commented
Jan 11, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@owen-t Addressed your comments above.
@jalabort
jalabort
commented
Jan 12, 2018

Copy link


Copy Markdown

ContributorAuthor
Is there something I need to amend due to the failing teamcity build? I cannot see what's wrong with it.
@ChoiByungWook
ChoiByungWook
commented
Jan 12, 2018

Copy link


Copy Markdown

Contributor
Hello,

We don't publish our build logs externally. A test failed, but the issue is resolved and we're rerunning your build. We will merge it in once the build is complete. Thanks again for your contribution!
@owen-t
owen-t
merged commit f43b9c3
into
aws:masterJan 12, 2018
apacker pushed a commit
to apacker/sagemaker-python-sdk
that referenced
this pull request
Nov 15, 2018
@vrkhare
Merge pull requestaws#40from awslabs/xgboost-updates
d8e1060
removing references to auto-scaling
athewsey added a commit
to athewsey/sagemaker-python-sdk
that referenced
this pull request
Jul 16, 2021
@athewsey
Merge remote-tracking branch 'upstream/master'
4ded19c
Merge pull request aws#40 from athewsey/feat/fw-processor
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Correctly handle TooManyBuckets error_code in default_bucket method (a…
caa27a0
…ws#40)
* Update session.py
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


1 more reviewer

@owen-towen-towen-t approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jalabort@ChoiByungWook@owen-t

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Correctly handle TooManyBuckets error_code in default_bucket method - #40
Merged
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Jan 12, 2018
Merged
Correctly handle TooManyBuckets error_code in default_bucket method#40
owen-t merged 2 commits into
aws:masterfrom 
jalabort:patch-1
Conversation
@jalabort
@jalabortjalabort
commented
Jan 10, 2018

Copy link


Copy Markdown

Contributor
At my organisation we have reached our maximum number of allowed buckets on s3 and as a direct consequence of that we started getting the following error when using the sagemaker-python-sdk:
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com
Traceback (most recent call last):
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/bin/sagemaker", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/cli/bin/sagemaker", line 6, in <module>
fire.Fire(component=SageMakerRunner)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 314, in train
self._train(source_local_path, dataset_s3_path, job_name)
File "/Users/joan.alabort/cvdev/beatrix/hudl_beatrix/sagemaker/base.py", line 364, in _train
run_tensorboard_locally=self._run_tensorboard_locally
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 162, in fit
fit_super()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/tensorflow/estimator.py", line 154, in fit_super
super(TensorFlow, self).fit(inputs, wait, logs, job_name)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/estimator.py", line 498, in fit
code_bucket = self.sagemaker_session.default_bucket()
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/sagemaker/session.py", line 159, in default_bucket
s3.create_bucket(Bucket=default_bucket)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/joan.alabort/miniconda3/envs/beatrix_27/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (TooManyBuckets) when calling the CreateBucket operation: You have attempted to create more buckets than allowed

This little patch seems to take care of this issue.
I could not find much information in terms of the expected PR structure so I hope this suffices, happy to make the pertinent changes if it does not.
@jalabort
Update session.py
922dde0
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
@jalabortjalabort
changed the title
Correctly handle the TooManyBuckets error_code in default_bucketCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
@jalabortjalabort
changed the title
Correctly handle TooManyBuckets error_code in default_bucket methodCorrectly handle TooManyBuckets error_code in default_bucket methodJan 10, 2018
owen-t
owen-t
previously approved these changes
Jan 11, 2018
View reviewed changes
@owen-towen-t
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your submission!
This looks good to me.
Comment threadsrc/sagemaker/session.py
Outdated

 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
 LOGGER.info('S3 bucket {} already exists'.format(
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this logging message, in the normal case we don't log this.
Comment threadsrc/sagemaker/session.py
 pass
 elif error_code == 'TooManyBuckets':
 try:
 s3.meta.client.head_bucket(Bucket=default_bucket)
@owen-towen-tJan 11, 2018

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little esoteric - if we remove the logging below, can you add some developer documentation:
# Succeed if the default bucket exists
@jalabort
Update session.py
40344b5
Remove logging message
@jalabort
jalabort
commented
Jan 11, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@owen-t Addressed your comments above.
@jalabort
jalabort
commented
Jan 12, 2018

Copy link


Copy Markdown

ContributorAuthor
Is there something I need to amend due to the failing teamcity build? I cannot see what's wrong with it.
@ChoiByungWook
ChoiByungWook
commented
Jan 12, 2018

Copy link


Copy Markdown

Contributor
Hello,

We don't publish our build logs externally. A test failed, but the issue is resolved and we're rerunning your build. We will merge it in once the build is complete. Thanks again for your contribution!
@owen-t
owen-t
merged commit f43b9c3
into
aws:masterJan 12, 2018
apacker pushed a commit
to apacker/sagemaker-python-sdk
that referenced
this pull request
Nov 15, 2018
@vrkhare
Merge pull requestaws#40from awslabs/xgboost-updates
d8e1060
removing references to auto-scaling
athewsey added a commit
to athewsey/sagemaker-python-sdk
that referenced
this pull request
Jul 16, 2021
@athewsey
Merge remote-tracking branch 'upstream/master'
4ded19c
Merge pull request aws#40 from athewsey/feat/fw-processor
Evan-W-ang added a commit
to Evan-W-ang/sagemaker-python-sdk
that referenced
this pull request
Jun 8, 2026
@Evan-W-ang
Correctly handle TooManyBuckets error_code in default_bucket method (a…
caa27a0
…ws#40)
* Update session.py
Correctly handles the `TooManyBuckets` error_code when the bucket already exists.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


1 more reviewer

@owen-towen-towen-t approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jalabort@ChoiByungWook@owen-t