Feat/gated model support - #4510

Merged
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support
Mar 18, 2024
Merged

Feat/gated model support#4510
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support

Conversation

@bencrabtree

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds support for gated models and models with training unsupported

Testing done:

Added unit tests for all new functions / logic pathways. Manually tested with Llama2-13b and other training unsupported

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

qidewenwhenand others added 12 commits March 15, 2024 21:27
* feat: add proprietary manifest/specs parsing
add unittests for test_cache
small refactoring
address comments and more unittests
fix linting and fix more tests
fix: pylint
feat: JumpStartModel class for prop models
* remove unused imports and fix docstyle
* fix: remove unused args
* fix: remove unused args
* fix: more unused vars
* fix: slow tests
* fix: unittests
* added more tests to cover some lines
* remove estimator warn check
* chore: address comments re performance
* fix: address comments
* complete list experience and other fixes
* fix: pylint
* add doc utils and fix pylint
* fix: docstyle
* fix: doc
* fix: default payloads
* fix: doc and tags and enums
* fix: jumpstart doc
* rename to open_weights and fix filtering
* update filter name
* doc update
* fix: black
* rename to proprietary model and fix unittests
* address comments
* fix: docstyle and flake8
* address more comments and fix doc
* put back doc utils for future refactoring
* add prop model title in doc
* doc update
---------
Co-authored-by: liujiaor <128006184+liujiaorr@users.noreply.github.com>
* first pass at sync function with util classes
* adding tests and update clases
* linting
* file generator class inheritance
* lint
* multipart copy and algorithm updates
* modularize sync
* reformatting folders
* testing for sync
* do not tolerate vulnerable
* remove prints
* handle multithreading progress bar
* update tests
* optimize function and add hub bucket prefix
* docstrings and linting
@bencrabtree
bencrabtree requested a review from a team as a code ownerMarch 18, 2024 13:50
@bencrabtree
bencrabtree requested review from knikure and removed request for a teamMarch 18, 2024 13:50
Comment threadsrc/sagemaker/jumpstart/curated_hub/accessors/public_model_data.py Outdated
@codecov

codecovBot commented Mar 18, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@d820b28). Click here to learn what that means.

Additional details and impacted files
@@ Coverage Diff @@## master-jumpstart-curated-hub #4510 +/- ##
===============================================================
Coverage ? 87.18% ===============================================================
Files ? 397 Lines ? 37256 Branches ? 0 ===============================================================
Hits ? 32481 Misses ? 4775 Partials ? 0 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 29733c8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aa8e4cb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

for dependency in HubContentDependencyType:
location: S3ObjectLocation = public_model_data_accessor.get_s3_reference(dependency)
# Training dependencies will return as None if training is unsupported
if not location or is_gated_bucket(location.bucket):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we checking for gated?

@bencrabtreebencrabtreeMar 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to / can't copy any files over from the gated bucket

model_version=self.model_specs.version,
script_scope=model_scope,
)
except ValueError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log something perhaps?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can log something here, though I don't think we'll ever reach this since we're only calling this function if training is supported

@erickb336erickb336 self-assigned this Mar 18, 2024
@erickb336
erickb336 merged commit a632795 into aws:master-jumpstart-curated-hubMar 18, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants

@bencrabtree@mufaddal-rohawala@mabunday@akozd@evakravi@erickb336@qidewenwhen@Captainia@jinyoung-lim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Feat/gated model support - #4510

Merged
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support
Mar 18, 2024
Merged

Feat/gated model support#4510
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support

Conversation

@bencrabtree

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds support for gated models and models with training unsupported

Testing done:

Added unit tests for all new functions / logic pathways. Manually tested with Llama2-13b and other training unsupported

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

qidewenwhenand others added 12 commits March 15, 2024 21:27
* feat: add proprietary manifest/specs parsing
add unittests for test_cache
small refactoring
address comments and more unittests
fix linting and fix more tests
fix: pylint
feat: JumpStartModel class for prop models
* remove unused imports and fix docstyle
* fix: remove unused args
* fix: remove unused args
* fix: more unused vars
* fix: slow tests
* fix: unittests
* added more tests to cover some lines
* remove estimator warn check
* chore: address comments re performance
* fix: address comments
* complete list experience and other fixes
* fix: pylint
* add doc utils and fix pylint
* fix: docstyle
* fix: doc
* fix: default payloads
* fix: doc and tags and enums
* fix: jumpstart doc
* rename to open_weights and fix filtering
* update filter name
* doc update
* fix: black
* rename to proprietary model and fix unittests
* address comments
* fix: docstyle and flake8
* address more comments and fix doc
* put back doc utils for future refactoring
* add prop model title in doc
* doc update
---------
Co-authored-by: liujiaor <128006184+liujiaorr@users.noreply.github.com>
* first pass at sync function with util classes
* adding tests and update clases
* linting
* file generator class inheritance
* lint
* multipart copy and algorithm updates
* modularize sync
* reformatting folders
* testing for sync
* do not tolerate vulnerable
* remove prints
* handle multithreading progress bar
* update tests
* optimize function and add hub bucket prefix
* docstrings and linting
@bencrabtree
bencrabtree requested a review from a team as a code ownerMarch 18, 2024 13:50
@bencrabtree
bencrabtree requested review from knikure and removed request for a teamMarch 18, 2024 13:50
Comment threadsrc/sagemaker/jumpstart/curated_hub/accessors/public_model_data.py Outdated
@codecov

codecovBot commented Mar 18, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@d820b28). Click here to learn what that means.

Additional details and impacted files
@@ Coverage Diff @@## master-jumpstart-curated-hub #4510 +/- ##
===============================================================
Coverage ? 87.18% ===============================================================
Files ? 397 Lines ? 37256 Branches ? 0 ===============================================================
Hits ? 32481 Misses ? 4775 Partials ? 0 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 29733c8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aa8e4cb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

for dependency in HubContentDependencyType:
location: S3ObjectLocation = public_model_data_accessor.get_s3_reference(dependency)
# Training dependencies will return as None if training is unsupported
if not location or is_gated_bucket(location.bucket):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we checking for gated?

@bencrabtreebencrabtreeMar 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to / can't copy any files over from the gated bucket

model_version=self.model_specs.version,
script_scope=model_scope,
)
except ValueError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log something perhaps?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can log something here, though I don't think we'll ever reach this since we're only calling this function if training is supported

@erickb336erickb336 self-assigned this Mar 18, 2024
@erickb336
erickb336 merged commit a632795 into aws:master-jumpstart-curated-hubMar 18, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants

@bencrabtree@mufaddal-rohawala@mabunday@akozd@evakravi@erickb336@qidewenwhen@Captainia@jinyoung-lim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feat/gated model support - #4510

Merged
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support
Mar 18, 2024
Merged

Feat/gated model support#4510
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support

Conversation

@bencrabtree

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds support for gated models and models with training unsupported

Testing done:

Added unit tests for all new functions / logic pathways. Manually tested with Llama2-13b and other training unsupported

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

qidewenwhenand others added 12 commits March 15, 2024 21:27
* feat: add proprietary manifest/specs parsing
add unittests for test_cache
small refactoring
address comments and more unittests
fix linting and fix more tests
fix: pylint
feat: JumpStartModel class for prop models
* remove unused imports and fix docstyle
* fix: remove unused args
* fix: remove unused args
* fix: more unused vars
* fix: slow tests
* fix: unittests
* added more tests to cover some lines
* remove estimator warn check
* chore: address comments re performance
* fix: address comments
* complete list experience and other fixes
* fix: pylint
* add doc utils and fix pylint
* fix: docstyle
* fix: doc
* fix: default payloads
* fix: doc and tags and enums
* fix: jumpstart doc
* rename to open_weights and fix filtering
* update filter name
* doc update
* fix: black
* rename to proprietary model and fix unittests
* address comments
* fix: docstyle and flake8
* address more comments and fix doc
* put back doc utils for future refactoring
* add prop model title in doc
* doc update
---------
Co-authored-by: liujiaor <128006184+liujiaorr@users.noreply.github.com>
* first pass at sync function with util classes
* adding tests and update clases
* linting
* file generator class inheritance
* lint
* multipart copy and algorithm updates
* modularize sync
* reformatting folders
* testing for sync
* do not tolerate vulnerable
* remove prints
* handle multithreading progress bar
* update tests
* optimize function and add hub bucket prefix
* docstrings and linting
@bencrabtree
bencrabtree requested a review from a team as a code ownerMarch 18, 2024 13:50
@bencrabtree
bencrabtree requested review from knikure and removed request for a teamMarch 18, 2024 13:50
Comment threadsrc/sagemaker/jumpstart/curated_hub/accessors/public_model_data.py Outdated
@codecov

codecovBot commented Mar 18, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@d820b28). Click here to learn what that means.

Additional details and impacted files
@@ Coverage Diff @@## master-jumpstart-curated-hub #4510 +/- ##
===============================================================
Coverage ? 87.18% ===============================================================
Files ? 397 Lines ? 37256 Branches ? 0 ===============================================================
Hits ? 32481 Misses ? 4775 Partials ? 0 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 29733c8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aa8e4cb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

for dependency in HubContentDependencyType:
location: S3ObjectLocation = public_model_data_accessor.get_s3_reference(dependency)
# Training dependencies will return as None if training is unsupported
if not location or is_gated_bucket(location.bucket):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we checking for gated?

@bencrabtreebencrabtreeMar 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to / can't copy any files over from the gated bucket

model_version=self.model_specs.version,
script_scope=model_scope,
)
except ValueError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log something perhaps?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can log something here, though I don't think we'll ever reach this since we're only calling this function if training is supported

@erickb336erickb336 self-assigned this Mar 18, 2024
@erickb336
erickb336 merged commit a632795 into aws:master-jumpstart-curated-hubMar 18, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants

@bencrabtree@mufaddal-rohawala@mabunday@akozd@evakravi@erickb336@qidewenwhen@Captainia@jinyoung-lim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feat/gated model support - #4510

Merged
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support
Mar 18, 2024
Merged

Feat/gated model support#4510
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support

Conversation

@bencrabtree

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds support for gated models and models with training unsupported

Testing done:

Added unit tests for all new functions / logic pathways. Manually tested with Llama2-13b and other training unsupported

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

qidewenwhenand others added 12 commits March 15, 2024 21:27
* feat: add proprietary manifest/specs parsing
add unittests for test_cache
small refactoring
address comments and more unittests
fix linting and fix more tests
fix: pylint
feat: JumpStartModel class for prop models
* remove unused imports and fix docstyle
* fix: remove unused args
* fix: remove unused args
* fix: more unused vars
* fix: slow tests
* fix: unittests
* added more tests to cover some lines
* remove estimator warn check
* chore: address comments re performance
* fix: address comments
* complete list experience and other fixes
* fix: pylint
* add doc utils and fix pylint
* fix: docstyle
* fix: doc
* fix: default payloads
* fix: doc and tags and enums
* fix: jumpstart doc
* rename to open_weights and fix filtering
* update filter name
* doc update
* fix: black
* rename to proprietary model and fix unittests
* address comments
* fix: docstyle and flake8
* address more comments and fix doc
* put back doc utils for future refactoring
* add prop model title in doc
* doc update
---------
Co-authored-by: liujiaor <128006184+liujiaorr@users.noreply.github.com>
* first pass at sync function with util classes
* adding tests and update clases
* linting
* file generator class inheritance
* lint
* multipart copy and algorithm updates
* modularize sync
* reformatting folders
* testing for sync
* do not tolerate vulnerable
* remove prints
* handle multithreading progress bar
* update tests
* optimize function and add hub bucket prefix
* docstrings and linting
@bencrabtree
bencrabtree requested a review from a team as a code ownerMarch 18, 2024 13:50
@bencrabtree
bencrabtree requested review from knikure and removed request for a teamMarch 18, 2024 13:50
Comment threadsrc/sagemaker/jumpstart/curated_hub/accessors/public_model_data.py Outdated
@codecov

codecovBot commented Mar 18, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@d820b28). Click here to learn what that means.

Additional details and impacted files
@@ Coverage Diff @@## master-jumpstart-curated-hub #4510 +/- ##
===============================================================
Coverage ? 87.18% ===============================================================
Files ? 397 Lines ? 37256 Branches ? 0 ===============================================================
Hits ? 32481 Misses ? 4775 Partials ? 0 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 29733c8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aa8e4cb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

for dependency in HubContentDependencyType:
location: S3ObjectLocation = public_model_data_accessor.get_s3_reference(dependency)
# Training dependencies will return as None if training is unsupported
if not location or is_gated_bucket(location.bucket):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we checking for gated?

@bencrabtreebencrabtreeMar 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to / can't copy any files over from the gated bucket

model_version=self.model_specs.version,
script_scope=model_scope,
)
except ValueError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log something perhaps?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can log something here, though I don't think we'll ever reach this since we're only calling this function if training is supported

@erickb336erickb336 self-assigned this Mar 18, 2024
@erickb336
erickb336 merged commit a632795 into aws:master-jumpstart-curated-hubMar 18, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants

@bencrabtree@mufaddal-rohawala@mabunday@akozd@evakravi@erickb336@qidewenwhen@Captainia@jinyoung-lim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Feat/gated model support - #4510

Merged
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support
Mar 18, 2024
Merged

Feat/gated model support#4510
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support

Conversation

@bencrabtree

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds support for gated models and models with training unsupported

Testing done:

Added unit tests for all new functions / logic pathways. Manually tested with Llama2-13b and other training unsupported

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

qidewenwhenand others added 12 commits March 15, 2024 21:27
* feat: add proprietary manifest/specs parsing
add unittests for test_cache
small refactoring
address comments and more unittests
fix linting and fix more tests
fix: pylint
feat: JumpStartModel class for prop models
* remove unused imports and fix docstyle
* fix: remove unused args
* fix: remove unused args
* fix: more unused vars
* fix: slow tests
* fix: unittests
* added more tests to cover some lines
* remove estimator warn check
* chore: address comments re performance
* fix: address comments
* complete list experience and other fixes
* fix: pylint
* add doc utils and fix pylint
* fix: docstyle
* fix: doc
* fix: default payloads
* fix: doc and tags and enums
* fix: jumpstart doc
* rename to open_weights and fix filtering
* update filter name
* doc update
* fix: black
* rename to proprietary model and fix unittests
* address comments
* fix: docstyle and flake8
* address more comments and fix doc
* put back doc utils for future refactoring
* add prop model title in doc
* doc update
---------
Co-authored-by: liujiaor <128006184+liujiaorr@users.noreply.github.com>
* first pass at sync function with util classes
* adding tests and update clases
* linting
* file generator class inheritance
* lint
* multipart copy and algorithm updates
* modularize sync
* reformatting folders
* testing for sync
* do not tolerate vulnerable
* remove prints
* handle multithreading progress bar
* update tests
* optimize function and add hub bucket prefix
* docstrings and linting
@bencrabtree
bencrabtree requested a review from a team as a code ownerMarch 18, 2024 13:50
@bencrabtree
bencrabtree requested review from knikure and removed request for a teamMarch 18, 2024 13:50
Comment threadsrc/sagemaker/jumpstart/curated_hub/accessors/public_model_data.py Outdated
@codecov

codecovBot commented Mar 18, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@d820b28). Click here to learn what that means.

Additional details and impacted files
@@ Coverage Diff @@## master-jumpstart-curated-hub #4510 +/- ##
===============================================================
Coverage ? 87.18% ===============================================================
Files ? 397 Lines ? 37256 Branches ? 0 ===============================================================
Hits ? 32481 Misses ? 4775 Partials ? 0 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 29733c8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aa8e4cb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

for dependency in HubContentDependencyType:
location: S3ObjectLocation = public_model_data_accessor.get_s3_reference(dependency)
# Training dependencies will return as None if training is unsupported
if not location or is_gated_bucket(location.bucket):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we checking for gated?

@bencrabtreebencrabtreeMar 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to / can't copy any files over from the gated bucket

model_version=self.model_specs.version,
script_scope=model_scope,
)
except ValueError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log something perhaps?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can log something here, though I don't think we'll ever reach this since we're only calling this function if training is supported

@erickb336erickb336 self-assigned this Mar 18, 2024
@erickb336
erickb336 merged commit a632795 into aws:master-jumpstart-curated-hubMar 18, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants

@bencrabtree@mufaddal-rohawala@mabunday@akozd@evakravi@erickb336@qidewenwhen@Captainia@jinyoung-lim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feat/gated model support - #4510

Merged
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support
Mar 18, 2024
Merged

Feat/gated model support#4510
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support

Conversation

@bencrabtree

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds support for gated models and models with training unsupported

Testing done:

Added unit tests for all new functions / logic pathways. Manually tested with Llama2-13b and other training unsupported

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

qidewenwhenand others added 12 commits March 15, 2024 21:27
* feat: add proprietary manifest/specs parsing
add unittests for test_cache
small refactoring
address comments and more unittests
fix linting and fix more tests
fix: pylint
feat: JumpStartModel class for prop models
* remove unused imports and fix docstyle
* fix: remove unused args
* fix: remove unused args
* fix: more unused vars
* fix: slow tests
* fix: unittests
* added more tests to cover some lines
* remove estimator warn check
* chore: address comments re performance
* fix: address comments
* complete list experience and other fixes
* fix: pylint
* add doc utils and fix pylint
* fix: docstyle
* fix: doc
* fix: default payloads
* fix: doc and tags and enums
* fix: jumpstart doc
* rename to open_weights and fix filtering
* update filter name
* doc update
* fix: black
* rename to proprietary model and fix unittests
* address comments
* fix: docstyle and flake8
* address more comments and fix doc
* put back doc utils for future refactoring
* add prop model title in doc
* doc update
---------
Co-authored-by: liujiaor <128006184+liujiaorr@users.noreply.github.com>
* first pass at sync function with util classes
* adding tests and update clases
* linting
* file generator class inheritance
* lint
* multipart copy and algorithm updates
* modularize sync
* reformatting folders
* testing for sync
* do not tolerate vulnerable
* remove prints
* handle multithreading progress bar
* update tests
* optimize function and add hub bucket prefix
* docstrings and linting
@bencrabtree
bencrabtree requested a review from a team as a code ownerMarch 18, 2024 13:50
@bencrabtree
bencrabtree requested review from knikure and removed request for a teamMarch 18, 2024 13:50
Comment threadsrc/sagemaker/jumpstart/curated_hub/accessors/public_model_data.py Outdated
@codecov

codecovBot commented Mar 18, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@d820b28). Click here to learn what that means.

Additional details and impacted files
@@ Coverage Diff @@## master-jumpstart-curated-hub #4510 +/- ##
===============================================================
Coverage ? 87.18% ===============================================================
Files ? 397 Lines ? 37256 Branches ? 0 ===============================================================
Hits ? 32481 Misses ? 4775 Partials ? 0 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 29733c8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aa8e4cb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

for dependency in HubContentDependencyType:
location: S3ObjectLocation = public_model_data_accessor.get_s3_reference(dependency)
# Training dependencies will return as None if training is unsupported
if not location or is_gated_bucket(location.bucket):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we checking for gated?

@bencrabtreebencrabtreeMar 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to / can't copy any files over from the gated bucket

model_version=self.model_specs.version,
script_scope=model_scope,
)
except ValueError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log something perhaps?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can log something here, though I don't think we'll ever reach this since we're only calling this function if training is supported

@erickb336erickb336 self-assigned this Mar 18, 2024
@erickb336
erickb336 merged commit a632795 into aws:master-jumpstart-curated-hubMar 18, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants

@bencrabtree@mufaddal-rohawala@mabunday@akozd@evakravi@erickb336@qidewenwhen@Captainia@jinyoung-lim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feat/gated model support - #4510

Merged
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support
Mar 18, 2024
Merged

Feat/gated model support#4510
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support

Conversation

@bencrabtree

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds support for gated models and models with training unsupported

Testing done:

Added unit tests for all new functions / logic pathways. Manually tested with Llama2-13b and other training unsupported

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

qidewenwhenand others added 12 commits March 15, 2024 21:27
* feat: add proprietary manifest/specs parsing
add unittests for test_cache
small refactoring
address comments and more unittests
fix linting and fix more tests
fix: pylint
feat: JumpStartModel class for prop models
* remove unused imports and fix docstyle
* fix: remove unused args
* fix: remove unused args
* fix: more unused vars
* fix: slow tests
* fix: unittests
* added more tests to cover some lines
* remove estimator warn check
* chore: address comments re performance
* fix: address comments
* complete list experience and other fixes
* fix: pylint
* add doc utils and fix pylint
* fix: docstyle
* fix: doc
* fix: default payloads
* fix: doc and tags and enums
* fix: jumpstart doc
* rename to open_weights and fix filtering
* update filter name
* doc update
* fix: black
* rename to proprietary model and fix unittests
* address comments
* fix: docstyle and flake8
* address more comments and fix doc
* put back doc utils for future refactoring
* add prop model title in doc
* doc update
---------
Co-authored-by: liujiaor <128006184+liujiaorr@users.noreply.github.com>
* first pass at sync function with util classes
* adding tests and update clases
* linting
* file generator class inheritance
* lint
* multipart copy and algorithm updates
* modularize sync
* reformatting folders
* testing for sync
* do not tolerate vulnerable
* remove prints
* handle multithreading progress bar
* update tests
* optimize function and add hub bucket prefix
* docstrings and linting
@bencrabtree
bencrabtree requested a review from a team as a code ownerMarch 18, 2024 13:50
@bencrabtree
bencrabtree requested review from knikure and removed request for a teamMarch 18, 2024 13:50
Comment threadsrc/sagemaker/jumpstart/curated_hub/accessors/public_model_data.py Outdated
@codecov

codecovBot commented Mar 18, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@d820b28). Click here to learn what that means.

Additional details and impacted files
@@ Coverage Diff @@## master-jumpstart-curated-hub #4510 +/- ##
===============================================================
Coverage ? 87.18% ===============================================================
Files ? 397 Lines ? 37256 Branches ? 0 ===============================================================
Hits ? 32481 Misses ? 4775 Partials ? 0 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 29733c8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aa8e4cb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

for dependency in HubContentDependencyType:
location: S3ObjectLocation = public_model_data_accessor.get_s3_reference(dependency)
# Training dependencies will return as None if training is unsupported
if not location or is_gated_bucket(location.bucket):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we checking for gated?

@bencrabtreebencrabtreeMar 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to / can't copy any files over from the gated bucket

model_version=self.model_specs.version,
script_scope=model_scope,
)
except ValueError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log something perhaps?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can log something here, though I don't think we'll ever reach this since we're only calling this function if training is supported

@erickb336erickb336 self-assigned this Mar 18, 2024
@erickb336
erickb336 merged commit a632795 into aws:master-jumpstart-curated-hubMar 18, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants

@bencrabtree@mufaddal-rohawala@mabunday@akozd@evakravi@erickb336@qidewenwhen@Captainia@jinyoung-lim
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Feat/gated model support - #4510

Merged
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support
Mar 18, 2024
Merged

Feat/gated model support#4510
erickb336 merged 17 commits into
aws:master-jumpstart-curated-hubfrom
bencrabtree:feat/gated-model-support

Conversation

@bencrabtree

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds support for gated models and models with training unsupported

Testing done:

Added unit tests for all new functions / logic pathways. Manually tested with Llama2-13b and other training unsupported

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

qidewenwhenand others added 12 commits March 15, 2024 21:27
* feat: add proprietary manifest/specs parsing
add unittests for test_cache
small refactoring
address comments and more unittests
fix linting and fix more tests
fix: pylint
feat: JumpStartModel class for prop models
* remove unused imports and fix docstyle
* fix: remove unused args
* fix: remove unused args
* fix: more unused vars
* fix: slow tests
* fix: unittests
* added more tests to cover some lines
* remove estimator warn check
* chore: address comments re performance
* fix: address comments
* complete list experience and other fixes
* fix: pylint
* add doc utils and fix pylint
* fix: docstyle
* fix: doc
* fix: default payloads
* fix: doc and tags and enums
* fix: jumpstart doc
* rename to open_weights and fix filtering
* update filter name
* doc update
* fix: black
* rename to proprietary model and fix unittests
* address comments
* fix: docstyle and flake8
* address more comments and fix doc
* put back doc utils for future refactoring
* add prop model title in doc
* doc update
---------
Co-authored-by: liujiaor <128006184+liujiaorr@users.noreply.github.com>
* first pass at sync function with util classes
* adding tests and update clases
* linting
* file generator class inheritance
* lint
* multipart copy and algorithm updates
* modularize sync
* reformatting folders
* testing for sync
* do not tolerate vulnerable
* remove prints
* handle multithreading progress bar
* update tests
* optimize function and add hub bucket prefix
* docstrings and linting
@bencrabtree
bencrabtree requested a review from a team as a code ownerMarch 18, 2024 13:50
@bencrabtree
bencrabtree requested review from knikure and removed request for a teamMarch 18, 2024 13:50
Comment threadsrc/sagemaker/jumpstart/curated_hub/accessors/public_model_data.py Outdated
@codecov

codecovBot commented Mar 18, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (master-jumpstart-curated-hub@d820b28). Click here to learn what that means.

Additional details and impacted files
@@ Coverage Diff @@## master-jumpstart-curated-hub #4510 +/- ##
===============================================================
Coverage ? 87.18% ===============================================================
Files ? 397 Lines ? 37256 Branches ? 0 ===============================================================
Hits ? 32481 Misses ? 4775 Partials ? 0 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 29733c8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala

Copy link
Copy Markdown
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aa8e4cb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

for dependency in HubContentDependencyType:
location: S3ObjectLocation = public_model_data_accessor.get_s3_reference(dependency)
# Training dependencies will return as None if training is unsupported
if not location or is_gated_bucket(location.bucket):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we checking for gated?

@bencrabtreebencrabtreeMar 18, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to / can't copy any files over from the gated bucket

model_version=self.model_specs.version,
script_scope=model_scope,
)
except ValueError:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log something perhaps?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can log something here, though I don't think we'll ever reach this since we're only calling this function if training is supported

@erickb336erickb336 self-assigned this Mar 18, 2024
@erickb336
erickb336 merged commit a632795 into aws:master-jumpstart-curated-hubMar 18, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants

@bencrabtree@mufaddal-rohawala@mabunday@akozd@evakravi@erickb336@qidewenwhen@Captainia@jinyoung-lim