Skip to content

Add scripts to DABs - #2813

Merged
shreyas-goenka merged 62 commits into
mainfrom
scripts-section
Jul 9, 2025
Merged

Add scripts to DABs#2813
shreyas-goenka merged 62 commits into
mainfrom
scripts-section

Conversation

@shreyas-goenka

@shreyas-goenkashreyas-goenka commented May 5, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

Scripts can be defined in a bundle YAML file as:

scripts:
my_script:
content: |
echo "one"
echo "two"

They can then be run by calling databricks bundle run my_script

Why

There are four primary benefits of having the scripts section in DABs:

  1. Customers can scripts (like integration testing scripts) live and be versioned with their bundle. Since integration tests based on spark connect require authentication, scripts run via this mechanism will automatically have the appropriate authentication configured.
  2. Simplifies setup for users who might not have their own build tools configured.
  3. Allows us to stardarize documentation to always refer to the scripts section.
  4. Make DABs in the workspace integration possible. Since DABs in the workspace cannot read arbitrary build file formats like Makefile.

Why do we have an additional level of indentation with the content field? Why not inline it?

The two main reasons are:

  1. This mirrors what the artifacts block looks like. Users in the artifact block are able to specify which shell they want to run their build command in. We should offer the same functionality in the scripts section.
  2. Leaves the room open for us to provide users with additional modifiers they might like to apply to their scripts.

Tests

Acceptance tests.

@shreyas-goenka
shreyas-goenka marked this pull request as draft May 5, 2025 12:44
@shreyas-goenkashreyas-goenka changed the title -[wip] scripts sectionMay 5, 2025
pietern
pietern reviewed Jul 3, 2025
View reviewed changes
Comment threadNEXT_CHANGELOG.md Outdated
Comment threadacceptance/bundle/run/scripts/databricks-cli/target-is-passed/from_flag/script
Comment threadacceptance/bundle/run/scripts/io/databricks.yml Outdated
Comment threadacceptance/bundle/run/scripts/no-auth/script Outdated
Comment threadcmd/bundle/run.go
shreyas-goenkaand others added 5 commits July 3, 2025 12:36
@shreyas-goenka@pietern
Update NEXT_CHANGELOG.md
9b15d61
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
@shreyas-goenka
address comments
e79622e
@shreyas-goenka
clarify test usecase
2c5aba2
@shreyas-goenka
-
968b282
@shreyas-goenka
merge'
4c348a9
andrewnester
andrewnester approved these changes Jul 8, 2025
View reviewed changes
Comment threadacceptance/bundle/run/scripts/cwd/databricks.yml Outdated
@shreyas-goenka
shreyas-goenka merged commit 6f14799 into mainJul 9, 2025
12 of 13 checks passed
@shreyas-goenka
shreyas-goenka deleted the scripts-section branch July 9, 2025 16:21
deco-sdk-taggingBot added a commit that referenced this pull request Jul 10, 2025
@deco-sdk-tagging
[Release] Release v0.259.0
9290d60
## Release v0.259.0
### Notable Changes
* Add support for arbitrary scripts in DABs. Users can now define scripts in their bundle configuration. These scripts automatically inherit the bundle's credentials for authentication. They can be invoked with the `bundle run` command. ([#2813](#2813))
* Error when the absolute path to `databricks.yml` contains a glob character. These are: `*`, `?`, `[`, `]` and `^`. If the path to the `databricks.yml` file on your local filesystem contains one of these characters, that could lead to incorrect computation of glob patterns for the `includes` block and might cause resources to be deleted. After this patch users will not be at risk for unexpected deletions due to this issue. ([#3096](#3096))
* Diagnostics messages are no longer buffered to be printed at the end of command, flushed after every mutator ([#3175](#3175))
* Diagnostics are now always rendered with forward slashes in file paths, even on Windows ([#3175](#3175))
* "bundle summary" now prints diagnostics to stderr instead of stdout in text output mode ([#3175](#3175))
* "bundle summary" no longer prints recommendations, it will only prints warnings and errors ([#3175](#3175))
### Bundles
* Fix default search location for whl artifacts ([#3184](#3184)). This was a regression introduced in 0.255.0.
* The job tasks are now sorted by task key in "bundle validate/summary" output ([#3212](#3212))
@shreyas-goenkashreyas-goenka mentioned this pull request Jul 11, 2025
Closed
denik pushed a commit that referenced this pull request May 20, 2026
@shreyas-goenka
Add ShellExecv method (#2862)
1ed60e7
## Changes
ShellExecv is a variant of the Execv method that also configures a shell
to be used before running the command.
## Why
We need it for the scripts section in DABs:
#2813.
Why:
1. We'll need to perform lexical analysis to figure out where the
boundaries for args in the script are. It's better to defer that to a
shell.
2. Direct calls to execv do not perform any environment variable
interpolation, so we need to go through a shell. This is neat because it
provides customers a way to configure inputs to their scripts.
3. Multiline scripts require a shell since, without them, we cannot
parse where a command ends.
Note: This uses the same priority order for shells as the
`artifacts.build` field, that is bash > sh > powershell.
## Tests
Unit test. Also acceptance tests in
#2813
denik pushed a commit that referenced this pull request May 20, 2026
@shreyas-goenka@pietern
Add scripts to DABs (#2813)
783d3f9
## Changes
Scripts can be defined in a bundle YAML file as:
```
scripts:
my_script:
content: |
echo "one"
echo "two"
```
They can then be run by calling `databricks bundle run my_script`
## Why
There are four primary benefits of having the scripts section in DABs:
1. Customers can scripts (like integration testing scripts) live and be
versioned with their bundle. Since integration tests based on spark
connect require authentication, scripts run via this mechanism will
automatically have the appropriate authentication configured.
2. Simplifies setup for users who might not have their own build tools
configured.
3. Allows us to stardarize documentation to always refer to the scripts
section.
4. Make DABs in the workspace integration possible. Since DABs in the
workspace cannot read arbitrary build file formats like Makefile.
### Why do we have an additional level of indentation with the `content`
field? Why not inline it?
The two main reasons are:
1. This mirrors what the artifacts block looks like. Users in the
artifact block are able to specify which shell they want to run their
build command in. We should offer the same functionality in the scripts
section.
5. Leaves the room open for us to provide users with additional
modifiers they might like to apply to their scripts.
## Tests
Acceptance tests.
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
denik pushed a commit that referenced this pull request May 20, 2026
@deco-sdk-tagging
[Release] Release v0.259.0
3b657c7
## Release v0.259.0
### Notable Changes
* Add support for arbitrary scripts in DABs. Users can now define scripts in their bundle configuration. These scripts automatically inherit the bundle's credentials for authentication. They can be invoked with the `bundle run` command. ([#2813](#2813))
* Error when the absolute path to `databricks.yml` contains a glob character. These are: `*`, `?`, `[`, `]` and `^`. If the path to the `databricks.yml` file on your local filesystem contains one of these characters, that could lead to incorrect computation of glob patterns for the `includes` block and might cause resources to be deleted. After this patch users will not be at risk for unexpected deletions due to this issue. ([#3096](#3096))
* Diagnostics messages are no longer buffered to be printed at the end of command, flushed after every mutator ([#3175](#3175))
* Diagnostics are now always rendered with forward slashes in file paths, even on Windows ([#3175](#3175))
* "bundle summary" now prints diagnostics to stderr instead of stdout in text output mode ([#3175](#3175))
* "bundle summary" no longer prints recommendations, it will only prints warnings and errors ([#3175](#3175))
### Bundles
* Fix default search location for whl artifacts ([#3184](#3184)). This was a regression introduced in 0.255.0.
* The job tasks are now sorted by task key in "bundle validate/summary" output ([#3212](#3212))
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewnesterandrewnesterandrewnester approved these changes
@anton-107anton-107Awaiting requested review from anton-107
@denikdenikAwaiting requested review from denik
@pieternpieternAwaiting requested review from pietern

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@shreyas-goenka@eng-dev-ecosystem-bot@pietern@denik@andrewnester