Added support for experimental scripts section by andrewnester · Pull Request #632 · databricks/cli

andrewnester · 2023-08-02T15:34:38Z

Changes

Added support for experimental scripts section

It allows execution of arbitrary bash commands during certain bundle lifecycle steps.

Tests

Example of configuration

bundle:
name: wheel-taskworkspace:
host: ***experimental:
scripts:
prebuild: | echo 'Prebuild 1' echo 'Prebuild 2'postbuild: "echo 'Postbuild 1' && echo 'Postbuild 2'"predeploy: | echo 'Checking go version...' go versionpostdeploy: | echo 'Checking python version...' python --versionresources:
jobs:
test_job:
name: "[${bundle.environment}] My Wheel Job"tasks:
- task_key: TestTaskexisting_cluster_id: "***"python_wheel_task:
package_name: "my_test_code"entry_point: "run"libraries:
- whl: ./dist/*.whl

Output

andrew.nester@HFW9Y94129 wheel % databricks bundle deploy
artifacts.whl.AutoDetect: Detecting Python wheel project...
artifacts.whl.AutoDetect: Found Python wheel project at /Users/andrew.nester/dabs/wheel
'Prebuild 1''Prebuild 2'
artifacts.whl.Build(my_test_code): Building...
artifacts.whl.Build(my_test_code): Build succeeded
'Postbuild 1''Postbuild 2''Checking go version...'
go version go1.19.9 darwin/arm64
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/wheel-task/default/files!
artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Uploading...
artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Upload succeeded
Starting resource deployment
Resource deployment completed!'Checking python version...'
Python 2.7.18

shreyas-goenka

LGTM!

kanterov · 2023-08-31T12:37:21Z

I've given it a try, and it worked for "databricks bundle deploy". I've found a few issues:

Files generated in "prebuild" can't be used in "include"
"prebuild" isn't called for "databricks bundle validate", so it can't be used in CI checks

andrewnester · 2023-09-12T12:27:40Z

@kanterov Build is not part of validate command so the hook is not executed. I've added postinit hook which can be used in CI and executed after initialisation.

pietern · 2023-09-14T09:52:48Z


 func DefaultMutators() []bundle.Mutator {
 	return []bundle.Mutator{
+ scripts.Execute(config.ScriptPreInit),


Note: this is not where this hook should be executed in the non-experimental version.
Before executing anything, we'd need to have already done:
target selection,
variable interpolation,
confirmed that our workspace is configured correctly.
Then to pick up new configuration (e.g. resources) from these scripts, we'd need a different mechanism. For example, we could set an environment variable DATABRICKS_BUNDLE_RESOURCES to some temporary file path that the script can write to, and we subsequently load + merge.
Given that this will be empty/a no-op for regular usage, I'm OK merging this and addressing this later.

Agree with the above. Let's merge at current version as an experimental and iterate over it

Bundles: * Use enums for default python template ([#765](#765)). * Make bundle deploy work if no resources are defined ([#767](#767)). * Added support for experimental scripts section ([#632](#632)). * Error when unknown keys are encounters during template execution ([#766](#766)). * Fall back to full Git clone if shallow clone is not supported ([#775](#775)). * Enable environment overrides for job tasks ([#779](#779)). * Increase timeout waiting for job run to 1 day ([#786](#786)). Internal: * Update Go SDK to v0.19.3 (unreleased) ([#778](#778)).

## Changes Added support for experimental scripts section It allows execution of arbitrary bash commands during certain bundle lifecycle steps. ## Tests Example of configuration ```yaml bundle: name: wheel-task workspace: host: *** experimental: scripts: prebuild: | echo 'Prebuild 1' echo 'Prebuild 2' postbuild: "echo 'Postbuild 1' && echo 'Postbuild 2'" predeploy: | echo 'Checking go version...' go version postdeploy: | echo 'Checking python version...' python --version resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" libraries: - whl: ./dist/*.whl ``` Output ```bash andrew.nester@HFW9Y94129 wheel % databricks bundle deploy artifacts.whl.AutoDetect: Detecting Python wheel project... artifacts.whl.AutoDetect: Found Python wheel project at /Users/andrew.nester/dabs/wheel 'Prebuild 1' 'Prebuild 2' artifacts.whl.Build(my_test_code): Building... artifacts.whl.Build(my_test_code): Build succeeded 'Postbuild 1' 'Postbuild 2' 'Checking go version...' go version go1.19.9 darwin/arm64 Starting upload of bundle files Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/wheel-task/default/files! artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Uploading... artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Upload succeeded Starting resource deployment Resource deployment completed! 'Checking python version...' Python 2.7.18 ```

Bundles: * Use enums for default python template ([#765](#765)). * Make bundle deploy work if no resources are defined ([#767](#767)). * Added support for experimental scripts section ([#632](#632)). * Error when unknown keys are encounters during template execution ([#766](#766)). * Fall back to full Git clone if shallow clone is not supported ([#775](#775)). * Enable environment overrides for job tasks ([#779](#779)). * Increase timeout waiting for job run to 1 day ([#786](#786)). Internal: * Update Go SDK to v0.19.3 (unreleased) ([#778](#778)).

## Changes Added support for experimental scripts section It allows execution of arbitrary bash commands during certain bundle lifecycle steps. ## Tests Example of configuration ```yaml bundle: name: wheel-task workspace: host: *** experimental: scripts: prebuild: | echo 'Prebuild 1' echo 'Prebuild 2' postbuild: "echo 'Postbuild 1' && echo 'Postbuild 2'" predeploy: | echo 'Checking go version...' go version postdeploy: | echo 'Checking python version...' python --version resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" libraries: - whl: ./dist/*.whl ``` Output ```bash andrew.nester@HFW9Y94129 wheel % databricks bundle deploy artifacts.whl.AutoDetect: Detecting Python wheel project... artifacts.whl.AutoDetect: Found Python wheel project at /Users/andrew.nester/dabs/wheel 'Prebuild 1' 'Prebuild 2' artifacts.whl.Build(my_test_code): Building... artifacts.whl.Build(my_test_code): Build succeeded 'Postbuild 1' 'Postbuild 2' 'Checking go version...' go version go1.19.9 darwin/arm64 Starting upload of bundle files Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/wheel-task/default/files! artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Uploading... artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Upload succeeded Starting resource deployment Resource deployment completed! 'Checking python version...' Python 2.7.18 ```

Bundles: * Use enums for default python template ([#765](#765)). * Make bundle deploy work if no resources are defined ([#767](#767)). * Added support for experimental scripts section ([#632](#632)). * Error when unknown keys are encounters during template execution ([#766](#766)). * Fall back to full Git clone if shallow clone is not supported ([#775](#775)). * Enable environment overrides for job tasks ([#779](#779)). * Increase timeout waiting for job run to 1 day ([#786](#786)). Internal: * Update Go SDK to v0.19.3 (unreleased) ([#778](#778)).

Added support for experimental scripts section
5d21ceb

andrewnester requested review from mgyucht, pietern and shreyas-goenka August 2, 2023 15:34

shreyas-goenka approved these changes Aug 2, 2023
View reviewed changes

Comment threadbundle/scripts/scripts.go Outdated

mgyucht reviewed Aug 4, 2023
View reviewed changes

Comment threadbundle/scripts/scripts.go Outdated
Comment threadbundle/scripts/scripts.go Outdated

lennartkats-db reviewed Aug 7, 2023
View reviewed changes

Comment threadbundle/scripts/scripts.go Outdated
Comment threadbundle/scripts/scripts.go Outdated
Comment threadbundle/scripts/scripts.go Outdated

pietern reviewed Aug 10, 2023
View reviewed changes

Comment threadbundle/scripts/scripts.go Outdated
Comment threadbundle/scripts/scripts.go Outdated
Comment threadbundle/scripts/scripts.go Outdated

andrewnester added 3 commits September 12, 2023 13:14

Merge branch 'main' into experimental-scripts
0255afb

Use sh
17d4472

streaming and preinit
180b51b

andrewnester requested a review from lennartkats-db September 12, 2023 12:21

added post init
c5b3caf

andrewnester requested a review from pietern September 12, 2023 12:30

pietern reviewed Sep 14, 2023
View reviewed changes

moved to experimental.go
745e861

andrewnester enabled auto-merge September 14, 2023 10:03

andrewnester added this pull request to the merge queue Sep 14, 2023

Merged via the queue into main with commit 953dcb4Sep 14, 2023

andrewnester deleted the experimental-scripts branch September 14, 2023 10:20

pietern mentioned this pull request Sep 20, 2023
Release v0.205.1 #787
Merged

Navigation Menu

Uh oh!

Added support for experimental scripts section - #632

Added support for experimental scripts section#632
andrewnester merged 6 commits into
mainfrom
experimental-scripts

andrewnester commented Aug 2, 2023

Uh oh!

shreyas-goenka left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kanterov commented Aug 31, 2023

Uh oh!

andrewnester commented Sep 12, 2023

Uh oh!

Uh oh!

pieternSep 14, 2023

Uh oh!

andrewnesterSep 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

andrewnester commented Aug 2, 2023

Changes

Tests

Uh oh!

shreyas-goenka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kanterov commented Aug 31, 2023

Uh oh!

andrewnester commented Sep 12, 2023

Uh oh!

Uh oh!

pieternSep 14, 2023

Choose a reason for hiding this comment

Uh oh!

andrewnesterSep 14, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants