Skip to content

Support Python wheels larger than 10MB - #879

Merged
andrewnester merged 2 commits into
mainfrom
larger-wheel
Oct 18, 2023
Merged

Support Python wheels larger than 10MB#879
andrewnester merged 2 commits into
mainfrom
larger-wheel

Conversation

@andrewnester

Copy link
Copy Markdown
Contributor

Changes

Previously we only supported uploading Python wheels smaller than 10mb due to using Workspace.Import API and content field https://docs.databricks.com/api/workspace/workspace/import

By switching to use WorkspaceFilesClient we overcome the limit because it uses POST body for the API instead.

Tests

TestAccUploadArtifactFileToCorrectRemotePath integration test passes

=== RUN TestAccUploadArtifactFileToCorrectRemotePath
artifacts_test.go:28: gcp
2023/10/17 15:24:04 INFO Using Google Credentials sdk=true
helpers.go:356: Creating /Users/.../integration-test-wsfs-ekggbkcfdkid
artifacts.Upload(test.whl): Uploading...
2023/10/17 15:24:06 INFO Using Google Credentials mutator=artifacts.Upload(test) sdk=true
artifacts.Upload(test.whl): Upload succeeded
helpers.go:362: Removing /Users/.../integration-test-wsfs-ekggbkcfdkid
--- PASS: TestAccUploadArtifactFileToCorrectRemotePath (5.66s)
PASS
coverage: 14.9% of statements in ./...
ok github.com/databricks/cli/internal 6.109s coverage: 14.9% of statements in ./...

Comment threadbundle/artifacts/artifacts.go
Comment threadinternal/artifacts_test.go
@pieternpietern changed the title Support Python wheels larger than 10mbSupport Python wheels larger than 10MBOct 18, 2023
@andrewnester
andrewnester added this pull request to the merge queueOct 18, 2023
Merged via the queue into main with commit 5273d0cOct 18, 2023
@andrewnester
andrewnester deleted the larger-wheel branch October 18, 2023 10:26
shreyas-goenka added a commit that referenced this pull request Oct 19, 2023
CLI:
* Fix rendering of streaming response ([#876](#876)).
* Rename MLOps Stack to MLOps Stacks ([#881](#881)).
* Support Python wheels larger than 10MB ([#879](#879)).
* Improve the output of the `databricks bundle init` command ([#795](#795)).
Bundles:
* **FILL THIS IN MANUALLY BY MOVING RELEVANT ITEMS FROM ABOVE LIST**
Internal:
* **FILL THIS IN MANUALLY BY MOVING RELEVANT ITEMS FROM ABOVE LIST**
@shreyas-goenkashreyas-goenka mentioned this pull request Oct 19, 2023
github-merge-queueBot pushed a commit that referenced this pull request Oct 19, 2023
CLI:
* Fix rendering of streaming response
([#876](#876)).
Bundles:
* Rename MLOps Stack to MLOps Stacks
([#881](#881)).
* Support Python wheels larger than 10MB
([#879](#879)).
* Improve the output of the `databricks bundle init` command
([#795](#795)).
denik pushed a commit that referenced this pull request May 20, 2026
## Changes
Previously we only supported uploading Python wheels smaller than 10mb
due to using Workspace.Import API and `content ` field
https://docs.databricks.com/api/workspace/workspace/import
By switching to use `WorkspaceFilesClient` we overcome the limit because
it uses POST body for the API instead.
## Tests
`TestAccUploadArtifactFileToCorrectRemotePath` integration test passes
```
=== RUN TestAccUploadArtifactFileToCorrectRemotePath
artifacts_test.go:28: gcp
2023/10/17 15:24:04 INFO Using Google Credentials sdk=true
helpers.go:356: Creating /Users/.../integration-test-wsfs-ekggbkcfdkid
artifacts.Upload(test.whl): Uploading...
2023/10/17 15:24:06 INFO Using Google Credentials mutator=artifacts.Upload(test) sdk=true
artifacts.Upload(test.whl): Upload succeeded
helpers.go:362: Removing /Users/.../integration-test-wsfs-ekggbkcfdkid
--- PASS: TestAccUploadArtifactFileToCorrectRemotePath (5.66s)
PASS
coverage: 14.9% of statements in ./...
ok github.com/databricks/cli/internal 6.109s coverage: 14.9% of statements in ./...
```
denik pushed a commit that referenced this pull request May 20, 2026
CLI:
* Fix rendering of streaming response
([#876](#876)).
Bundles:
* Rename MLOps Stack to MLOps Stacks
([#881](#881)).
* Support Python wheels larger than 10MB
([#879](#879)).
* Improve the output of the `databricks bundle init` command
([#795](#795)).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@andrewnester@pietern