Uh oh!
There was an error while loading. Please reload this page.
Add "stale wheel" acc/integration test for serverless - #2519
Merged
Conversation
denik
requested review from
andrewnester, pietern and shreyas-goenka
as code ownersMarch 19, 2025 09:28
denikforce-pushed
the
denik/acc-whl-serverless
branch
from
March 19, 2025 09:31
59d6a1a to
8faa723CompareUh oh!
There was an error while loading. Please reload this page.
| @@ -0,0 +1,87 @@ | |||
| >>> cp -r [TESTROOT]/bundle/integration_whl/serverless/../base/setup.py [TESTROOT]/bundle/integration_whl/serverless/../base/my_test_code . | |||
Contributor
There was a problem hiding this comment.
I guess we need to add a repl for Window forward slashes in path?
| @@ -0,0 +1,12 @@ | |||
| envsubst < databricks.yml.tmpl > databricks.yml | |||
| trace cp -r $TESTDIR/../base/{setup.py,my_test_code} . | |||
Contributor
There was a problem hiding this comment.
Does the path work on Windows?
denikforce-pushed
the
denik/acc-whl-serverless
branch
from
March 19, 2025 11:31
8faa723 to
a9b1826Comparedenikforce-pushed
the
denik/acc-whl-serverless
branch
from
March 19, 2025 11:38
4423c40 to
068d3ccComparedenikforce-pushed
the
denik/acc-whl-serverless
branch
from
March 19, 2025 15:54
7932ac4 to
055fe37Comparedenik
enabled auto-merge
March 19, 2025 15:56
andrewnester
approved these changes
Mar 20, 2025
denik
commented
Mar 20, 2025
ContributorAuthor
Integration test failure seems unrelated (TestSyncNestedFolderSync) |
github-merge-queueBot
pushed a commit
that referenced
this pull request
Mar 26, 2025
## Changes Add opt-in field `artifacts.*.dynamic_version` (bool). If set to true, then wheels produced or referenced by this artifact will be "patched" to break dependency caching. The patching involves modifying the version to append current timestamp after `+` sign and rewriting internals of wheel, such as METADATA and RECORD. Details of patching: #2427 The patching involves two phases: 1. During build phase we check what wheels where produced (or references via files.source field) and apply PatchWheel on those. 2. During deploy phase we go through places where wheels are referenced and update unpatched files to patched. The places are jobs/task/libraries, jobs/task/for_each_task/libraries, jobs/environments/spec/dependencies. ## Why We would like to switch default template to pyproject.toml without having to carry on setuptools dependency. Currently we're relying on setuptools to patch the version at build time: #1034 Previously I added integration tests showing this problem: #2477#2519 Those tests showed that the problem exists only for interactive clusters and only when DATA_SECURITY_MODE=USER_ISOLATION. However, this PR enables patching everywhere, since we don't have any guarantees about this behaviour. ## Tests New local acceptance test whl_dynamic to check that patching works across all 3 places. New integration tests to check that patched wheels can be run properly in classic and serverless environments: interactive_cluster_dynamic_version, serverless_dynamic_version.
denik added a commit
that referenced
this pull request
May 20, 2026
## Changes Expand integration_whl tests with serverless configuration ## Why Record "stale wheel" situation. Going to use this test for patched wheel functionality. ## Tests Enabling this test on PR by temp setting CLOUD_SLOW = false (http://go/deco/tests/13948235349/job/39040568795)
denik added a commit
that referenced
this pull request
May 20, 2026
## Changes Add opt-in field `artifacts.*.dynamic_version` (bool). If set to true, then wheels produced or referenced by this artifact will be "patched" to break dependency caching. The patching involves modifying the version to append current timestamp after `+` sign and rewriting internals of wheel, such as METADATA and RECORD. Details of patching: #2427 The patching involves two phases: 1. During build phase we check what wheels where produced (or references via files.source field) and apply PatchWheel on those. 2. During deploy phase we go through places where wheels are referenced and update unpatched files to patched. The places are jobs/task/libraries, jobs/task/for_each_task/libraries, jobs/environments/spec/dependencies. ## Why We would like to switch default template to pyproject.toml without having to carry on setuptools dependency. Currently we're relying on setuptools to patch the version at build time: #1034 Previously I added integration tests showing this problem: #2477#2519 Those tests showed that the problem exists only for interactive clusters and only when DATA_SECURITY_MODE=USER_ISOLATION. However, this PR enables patching everywhere, since we don't have any guarantees about this behaviour. ## Tests New local acceptance test whl_dynamic to check that patching works across all 3 places. New integration tests to check that patched wheels can be run properly in classic and serverless environments: interactive_cluster_dynamic_version, serverless_dynamic_version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Expand integration_whl tests with serverless configuration
Why
Record "stale wheel" situation. Going to use this test for patched wheel functionality.
Tests
Enabling this test on PR by temp setting CLOUD_SLOW = false (http://go/deco/tests/13948235349/job/39040568795)