Added transformation mutator for Python wheel task for them to work on DBR <13.1 by andrewnester · Pull Request #635 · databricks/cli

andrewnester · 2023-08-03T14:08:48Z

Changes

Note: this PR relies on sync.include functionality from here: #671

Added transformation mutator for Python wheel task for them to work on DBR <13.1

Using wheels upload to Workspace file system as cluster libraries is not supported in DBR < 13.1

In order to make Python wheel work correctly on DBR < 13.1 we do the following:

Build and upload python wheel as usual
Transform python wheel task into special notebook task which does the following
a. Installs all necessary wheels with %pip magic
b. Executes defined entry point with all provided parameters
Upload this notebook file to workspace file system
Deploy transformed job task

This is also beneficial for executing on existing clusters because this notebook always reinstall wheels so if there are any changes to the wheel package, they are correctly picked up

Tests

bundle.yml

bundle:
name: wheel-taskworkspace:
host: ****resources:
jobs:
test_job:
name: "[${bundle.environment}] My Wheel Job"tasks:
- task_key: TestTaskexisting_cluster_id: "***"python_wheel_task:
package_name: "my_test_code"entry_point: "run"parameters: ["first argument","first value","second argument","second value"]libraries:
- whl: ./dist/*.whl

Output

andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job
Run URL: ***
2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS Output:
=======
Task TestTask:
Hello from my func
Got arguments v1:
['python', 'first argument', 'first value', 'second argument', 'second value']

…n DBR <13.1

#674) ## Changes Fixes#673 It also includes a change for `libraries` from #635 to get the list of wheel tasks

mgyucht

May be a good idea to wait for @pietern to get back, but this looks good to me. A couple small (non-blocking) suggestions from me.

pietern · 2023-08-30T10:07:13Z

+	remotePath := path.Join(b.Config.Workspace.FilesPath, filepath.ToSlash(internalDirRel), notebookName)
+
+	task.NotebookTask = &jobs.NotebookTask{
+ NotebookPath: remotePath,


Can you comment on how we expect the wheel args, or task params, to be passed in the template?

As a reference example, in Python trampoline we do it with sys.argv as you can see here
https://github.com/databricks/cli/pull/635/files#diff-8ddf2564fcd580399d61df5283d0c0c7b614c476e0803f63f9f785b62dc69a04R23
I'd expect it should work similarly for task params

CLI: * Fixed --environment flag ([#705](#705)). Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)).

Also see #635.

## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Fixed --environment flag ([#705](#705)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)). * Test transform when no Python wheel tasks defined ([#714](#714)). * Pin Terraform binary version to 1.5.5 ([#715](#715)). * Cleanup after "Add a foundation for built-in templates" ([#707](#707)). * Filter down to Python wheel tasks only for trampoline ([#712](#712)). * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).

…on DBR <13.1 (#635) ## Changes ***Note: this PR relies on sync.include functionality from here: #671 Added transformation mutator for Python wheel task for them to work on DBR <13.1 Using wheels upload to Workspace file system as cluster libraries is not supported in DBR < 13.1 In order to make Python wheel work correctly on DBR < 13.1 we do the following: 1. Build and upload python wheel as usual 2. Transform python wheel task into special notebook task which does the following a. Installs all necessary wheels with %pip magic b. Executes defined entry point with all provided parameters 3. Upload this notebook file to workspace file system 4. Deploy transformed job task This is also beneficial for executing on existing clusters because this notebook always reinstall wheels so if there are any changes to the wheel package, they are correctly picked up ## Tests bundle.yml ```yaml bundle: name: wheel-task workspace: host: **** resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" parameters: ["first argument","first value","second argument","second value"] libraries: - whl: ./dist/*.whl ``` Output ``` andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job Run URL: *** 2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS Output: ======= Task TestTask: Hello from my func Got arguments v1: ['python', 'first argument', 'first value', 'second argument', 'second value'] ```

## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Fixed --environment flag ([#705](#705)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)). * Test transform when no Python wheel tasks defined ([#714](#714)). * Pin Terraform binary version to 1.5.5 ([#715](#715)). * Cleanup after "Add a foundation for built-in templates" ([#707](#707)). * Filter down to Python wheel tasks only for trampoline ([#712](#712)). * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).

…on DBR <13.1 (databricks#635) ## Changes ***Note: this PR relies on sync.include functionality from here: databricks#671 Added transformation mutator for Python wheel task for them to work on DBR <13.1 Using wheels upload to Workspace file system as cluster libraries is not supported in DBR < 13.1 In order to make Python wheel work correctly on DBR < 13.1 we do the following: 1. Build and upload python wheel as usual 2. Transform python wheel task into special notebook task which does the following a. Installs all necessary wheels with %pip magic b. Executes defined entry point with all provided parameters 3. Upload this notebook file to workspace file system 4. Deploy transformed job task This is also beneficial for executing on existing clusters because this notebook always reinstall wheels so if there are any changes to the wheel package, they are correctly picked up ## Tests bundle.yml ```yaml bundle: name: wheel-task workspace: host: **** resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" parameters: ["first argument","first value","second argument","second value"] libraries: - whl: ./dist/*.whl ``` Output ``` andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job Run URL: *** 2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS Output: ======= Task TestTask: Hello from my func Got arguments v1: ['python', 'first argument', 'first value', 'second argument', 'second value'] ``` Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

## Changes Fixes issue introduced in databricks#635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle. Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

Bundles: * Support cluster overrides with cluster_key and compute_key ([databricks#696](databricks#696)). * Allow referencing local Python wheels without artifacts section defined ([databricks#703](databricks#703)). * Fixed --environment flag ([databricks#705](databricks#705)). * Correctly identify local paths in libraries section ([databricks#702](databricks#702)). * Fixed path joining in FindFilesWithSuffixInPath ([databricks#704](databricks#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([databricks#635](databricks#635)). Internal: * Add a foundation for built-in templates ([databricks#685](databricks#685)). * Test transform when no Python wheel tasks defined ([databricks#714](databricks#714)). * Pin Terraform binary version to 1.5.5 ([databricks#715](databricks#715)). * Cleanup after "Add a foundation for built-in templates" ([databricks#707](databricks#707)). * Filter down to Python wheel tasks only for trampoline ([databricks#712](databricks#712)). * Update Terraform provider schema structs from 1.23.0 ([databricks#713](databricks#713)). Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

stawo · 2023-09-19T13:19:10Z

@andrewnester , @pietern , I believe this PR introduced some breaking behavior. See #783

## Summary The `wrapper` and `wrapper_custom_params` tests pin Spark to `12.2.x-scala2.12` to exercise the trampoline workaround for DBR <13.1 ([PR #635](#635)). The AWS test workspaces have disabled legacy access, so 12.2.x is rejected at job submission time: ``` INVALID_PARAMETER_VALUE: Spark version 12.2.x-scala2.12 isn't supported because legacy access is disabled in your workspace. Please use Databricks Runtime 13.3 LTS or above when any legacy features are disabled. ``` Bumping the runtime would defeat the purpose of the test (the trampoline only triggers for DBR <13.1). Restrict the test matrix to Azure where 12.2.x is still bookable. GCP was already excluded for an unrelated DBR release issue. The trampoline itself is covered by unit tests in `bundle/trampoline/` and acceptance tests under `acceptance/bundle/trampoline/`. ## Test plan - [ ] Confirm next nightly: wrapper and wrapper_custom_params no longer in the failures list on aws-prod-is or aws-prod-ucws-is. - [ ] They continue to pass on Azure (azure-prod-is, azure-prod-ucws-is) where they had been passing before. This pull request was AI-assisted by Isaac.

#674) ## Changes Fixes#673 It also includes a change for `libraries` from #635 to get the list of wheel tasks

…on DBR <13.1 (#635) ## Changes ***Note: this PR relies on sync.include functionality from here: #671 Added transformation mutator for Python wheel task for them to work on DBR <13.1 Using wheels upload to Workspace file system as cluster libraries is not supported in DBR < 13.1 In order to make Python wheel work correctly on DBR < 13.1 we do the following: 1. Build and upload python wheel as usual 2. Transform python wheel task into special notebook task which does the following a. Installs all necessary wheels with %pip magic b. Executes defined entry point with all provided parameters 3. Upload this notebook file to workspace file system 4. Deploy transformed job task This is also beneficial for executing on existing clusters because this notebook always reinstall wheels so if there are any changes to the wheel package, they are correctly picked up ## Tests bundle.yml ```yaml bundle: name: wheel-task workspace: host: **** resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" parameters: ["first argument","first value","second argument","second value"] libraries: - whl: ./dist/*.whl ``` Output ``` andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job Run URL: *** 2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS Output: ======= Task TestTask: Hello from my func Got arguments v1: ['python', 'first argument', 'first value', 'second argument', 'second value'] ```

## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Fixed --environment flag ([#705](#705)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)). * Test transform when no Python wheel tasks defined ([#714](#714)). * Pin Terraform binary version to 1.5.5 ([#715](#715)). * Cleanup after "Add a foundation for built-in templates" ([#707](#707)). * Filter down to Python wheel tasks only for trampoline ([#712](#712)). * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).

## Summary The `wrapper` and `wrapper_custom_params` tests pin Spark to `12.2.x-scala2.12` to exercise the trampoline workaround for DBR <13.1 ([PR #635](#635)). The AWS test workspaces have disabled legacy access, so 12.2.x is rejected at job submission time: ``` INVALID_PARAMETER_VALUE: Spark version 12.2.x-scala2.12 isn't supported because legacy access is disabled in your workspace. Please use Databricks Runtime 13.3 LTS or above when any legacy features are disabled. ``` Bumping the runtime would defeat the purpose of the test (the trampoline only triggers for DBR <13.1). Restrict the test matrix to Azure where 12.2.x is still bookable. GCP was already excluded for an unrelated DBR release issue. The trampoline itself is covered by unit tests in `bundle/trampoline/` and acceptance tests under `acceptance/bundle/trampoline/`. ## Test plan - [ ] Confirm next nightly: wrapper and wrapper_custom_params no longer in the failures list on aws-prod-is or aws-prod-ucws-is. - [ ] They continue to pass on Azure (azure-prod-is, azure-prod-ucws-is) where they had been passing before. This pull request was AI-assisted by Isaac.

…abricks#5152) ## Summary The `wrapper` and `wrapper_custom_params` tests pin Spark to `12.2.x-scala2.12` to exercise the trampoline workaround for DBR <13.1 ([PR databricks#635](databricks#635)). The AWS test workspaces have disabled legacy access, so 12.2.x is rejected at job submission time: ``` INVALID_PARAMETER_VALUE: Spark version 12.2.x-scala2.12 isn't supported because legacy access is disabled in your workspace. Please use Databricks Runtime 13.3 LTS or above when any legacy features are disabled. ``` Bumping the runtime would defeat the purpose of the test (the trampoline only triggers for DBR <13.1). Restrict the test matrix to Azure where 12.2.x is still bookable. GCP was already excluded for an unrelated DBR release issue. The trampoline itself is covered by unit tests in `bundle/trampoline/` and acceptance tests under `acceptance/bundle/trampoline/`. ## Test plan - [ ] Confirm next nightly: wrapper and wrapper_custom_params no longer in the failures list on aws-prod-is or aws-prod-ucws-is. - [ ] They continue to pass on Azure (azure-prod-is, azure-prod-ucws-is) where they had been passing before. This pull request was AI-assisted by Isaac.

Added transformation mutator for Python wheel task for them to work o…
78b75f6
…n DBR <13.1

andrewnester requested review from pietern and shreyas-goenka August 3, 2023 14:08

pietern reviewed Aug 10, 2023
View reviewed changes

use sync include and templates
2be69af

andrewnester requested a review from pietern August 14, 2023 15:37

andrewnester changed the title ~~Added transformation mutator for Python wheel task for them to work on DBR <13.1~~Added Sync.Include and Sync.Excluded + Added transformation mutator for Python wheel task for them to work on DBR <13.1Aug 14, 2023

pietern reviewed Aug 16, 2023
View reviewed changes

andrewnester changed the title ~~Added Sync.Include and Sync.Excluded + Added transformation mutator for Python wheel task for them to work on DBR <13.1~~ Added transformation mutator for Python wheel task for them to work on DBR <13.1Aug 16, 2023

andrewnester added 2 commits August 16, 2023 15:50

removed sync include code
2e3c5ca

add tests
a9ed5df

andrewnester requested a review from pietern August 16, 2023 14:35

andrewnester mentioned this pull request Aug 16, 2023
Do not try auto detect Python package if no Python wheel tasks defined #674
Merged

pietern reviewed Aug 17, 2023
View reviewed changes

Comment threadbundle/python/transform.go
Comment threadbundle/python/transform.go Outdated
Comment threadbundle/python/transform.go Outdated
Comment threadbundle/python/transform_test.go
Comment threadbundle/python/transform.go Outdated

github-merge-queueBot pushed a commit that referenced this pull request Aug 17, 2023
Do not try auto detect Python package if no Python wheel tasks defined (
4694832
#674) ## Changes Fixes#673 It also includes a change for `libraries` from #635 to get the list of wheel tasks

andrewnester added 3 commits August 17, 2023 17:50

moved to separate function
476439c

Merge branch 'main' into transform-wheel-task
b10d50f

more reusable code
aeb77d9

andrewnester requested a review from pietern August 18, 2023 09:27

andrewnester enabled auto-merge August 18, 2023 09:38

andrewnester requested a review from mgyucht August 23, 2023 15:06

mgyucht approved these changes Aug 25, 2023
View reviewed changes

Comment threadbundle/config/mutator/trampoline.go
Comment threadbundle/config/mutator/trampoline.go Outdated
Comment threadbundle/config/mutator/trampoline.go Outdated

andrewnester added this pull request to the merge queue Aug 25, 2023

github-merge-queueBot removed this pull request from the merge queue due to failed status checks Aug 25, 2023

kartikgupta-db reviewed Aug 29, 2023
View reviewed changes

Comment threadbundle/config/mutator/trampoline.go Outdated

pietern reviewed Aug 30, 2023
View reviewed changes

andrewnester added 2 commits August 30, 2023 12:24

Merge branch 'main' into transform-wheel-task
36732ac

addressed feedback
de391b2

andrewnester requested review from kartikgupta-db and pietern August 30, 2023 11:04

Merged via the queue into main with commit 12368e3Aug 30, 2023

andrewnester deleted the transform-wheel-task branch August 30, 2023 12:32

andrewnester mentioned this pull request Aug 30, 2023
Release v0.203.3 #711
Closed

pietern added a commit that referenced this pull request Aug 30, 2023
Filter down to Python wheel tasks only for trampoline
cc364a8
Also see #635.

pietern mentioned this pull request Aug 30, 2023
Filter down to Python wheel tasks only for trampoline #712
Merged

github-merge-queueBot pushed a commit that referenced this pull request Aug 30, 2023
Filter down to Python wheel tasks only for trampoline (#712)
ca2f1dc
## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

andrewnester mentioned this pull request Aug 30, 2023
Release v0.203.3 #716
Merged

lennartkats-db pushed a commit that referenced this pull request Sep 1, 2023
Filter down to Python wheel tasks only for trampoline (#712)
2c9d687
## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

stawo mentioned this pull request Sep 15, 2023
Python wheel based tasks no longer deployed correctly #783
Closed

andrewnester mentioned this pull request Nov 21, 2023
bundle run fails for non unity-catalog workspace #1000
Closed

andrewnester mentioned this pull request Dec 11, 2023
Python code is not updated when using Python wheel tasks with existing Cluster ID in the task definition #1050
Closed

andrewnester mentioned this pull request Mar 12, 2025
Convert python_wheel_test.go to acceptance test #2471
Merged

shreyas-goenka mentioned this pull request May 1, 2026
acceptance: skip integration_whl/wrapper{,_custom_params} on AWS #5152
Merged
2 tasks

denik pushed a commit that referenced this pull request May 20, 2026
Do not try auto detect Python package if no Python wheel tasks defined (
97fed43
#674) ## Changes Fixes#673 It also includes a change for `libraries` from #635 to get the list of wheel tasks

denik pushed a commit that referenced this pull request May 20, 2026
Filter down to Python wheel tasks only for trampoline (#712)
49284b9
## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

Uh oh!

Conversation

andrewnester commented Aug 3, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgyucht left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternAug 30, 2023

Choose a reason for hiding this comment

Uh oh!

andrewnesterAug 30, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stawo commented Sep 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

andrewnester commented Aug 3, 2023•
edited
Loading