Skip to content

Add env section to bundle scripts for DABs interpolation - #5299

Merged
shreyas-goenka merged 6 commits into
mainfrom
script-env-section
Jul 13, 2026
Merged

Add env section to bundle scripts for DABs interpolation#5299
shreyas-goenka merged 6 commits into
mainfrom
script-env-section

Conversation

@shreyas-goenka

@shreyas-goenkashreyas-goenka commented May 21, 2026

Copy link
Copy Markdown
Contributor

Fixes#4179.

Adds an env: map to scripts.<name>. Values may reference ${bundle.*}, ${workspace.*}, and ${var.*}; they're resolved before the script runs and exported into its shell, where content uses plain $NAME:

scripts:
show_env:
env:
REGION: ${var.region}content: echo "region=$REGION"

content is still passed to the shell as-is, so ${...} in content stays unsupported (the error now points to the env: section). Env values with any other prefix (e.g. ${resources.*}) are rejected. On a name collision with the CLI-injected auth vars, the script's env: wins.

This pull request was AI-assisted by Isaac.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: a32d68c

Run: 29084501634

Env🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
💚​aws linux4423010804:59
💚​aws windows4423210787:06
💚​aws-ucws linux443149985:31
💚​aws-ucws windows443169967:22
💚​azure linux4423010794:51
💚​azure windows4423210777:01
💚​azure-ucws linux443169955:31
🔄​azure-ucws windows1443179937:56
💚​gcp linux4422910815:17
💚​gcp windows4423110797:09
9 interesting tests: 4 RECOVERED, 4 SKIP, 1 flaky
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
💚​TestAccept💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
🙈​TestAccept/bundle/invariant/no_drift🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/ssh/connection🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🔄​TestSyncIncrementalFileOverwritesFolder✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p
💚​TestFetchRepositoryInfoAPI_FromRepo💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
💚​TestFetchRepositoryInfoAPI_FromRepo/root💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
💚​TestFetchRepositoryInfoAPI_FromRepo/subdir💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
Top 10 slowest tests (at least 2 minutes):
durationenvtestname
6:31azure-ucws windowsTestAccept
6:18aws-ucws windowsTestAccept
6:11aws windowsTestAccept
6:09gcp windowsTestAccept
6:03azure windowsTestAccept
2:56azure linuxTestAccept
2:55aws linuxTestAccept
2:54gcp linuxTestAccept
2:50azure-ucws linuxTestAccept
2:47aws-ucws linuxTestAccept

@shreyas-goenka
shreyas-goenkaforce-pushed the script-env-section branch 5 times, most recently from f373d61 to b68897dCompareJune 29, 2026 16:37
scripts.<name> now accepts an env: map whose values may reference
${bundle.*}, ${workspace.*}, and ${var.*}. The script content is still
passed to the shell as-is (no DABs interpolation), removing the ambiguity
between bundle variables and shell variables that previously forced all
${...} usage to be rejected.
When an env value references an unsupported prefix (e.g. ${resources.*}),
validation reports a clear error pointing at the field. The content-side
error now suggests the new env section instead of just stating ${...} is
unsupported.
Fixes#4179
Co-authored-by: Isaac
@shreyas-goenka
shreyas-goenka marked this pull request as ready for review July 6, 2026 13:30
Resolved conflicts in NEXT_CHANGELOG.md (kept both entry sets) and
bundle/internal/schema/annotations.yml (took main's regenerated layout,
re-added the scripts env field annotation).
Co-authored-by: Isaac
Comment threadacceptance/bundle/run/scripts/env-bad-prefix/output.txt
Comment threadNEXT_CHANGELOG.md
Fix the earlier bad merge in NEXT_CHANGELOG.md: the prior merge
resurrected Bundles entries that had already been swept into the
v1.6.0 release. Reset the changelog to origin/main and re-added only
the scripts env entry.
Co-authored-by: Isaac
PLACEHOLDER
"env":
"description": |-
PLACEHOLDER

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a description here so it shows up in tooltips via the jsonschema

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added a description for scripts.<name>.env in annotations.yml (regenerated jsonschema.json).

Comment threadcmd/bundle/run.go
Comment on lines +281 to +286
env := scriptEnv(cmd, b)
// Append after the auth/target variables so a script's env: section takes
// precedence on collision (os/exec uses the last value for a duplicate key).
for _, name := range slices.Sorted(maps.Keys(script.Env)) {
env = append(env, name+"="+script.Env[name])
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add an acceptance test for this

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the env-precedence acceptance test: a script's env: entry sets DATABRICKS_BUNDLE_TARGET=from-script-env and the output confirms it wins over the CLI-injected value.

@@ -0,0 +1 @@
errcode trace $CLI bundle validate

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
errcode trace $CLI bundle validate
musterr trace $CLI bundle validate

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied musterr (dropped the redundant errcode).

- Add a real description for scripts.<name>.env in annotations.yml so it
surfaces in jsonschema tooltips (regenerated jsonschema.json).
- Use musterr instead of errcode in the env-bad-prefix script to assert
the command must fail.
- Add an env-precedence acceptance test proving a script's env: entry
overrides a CLI-injected variable (DATABRICKS_BUNDLE_TARGET).
Co-authored-by: Isaac
v1.8.0 was cut, emptying the Bundles section of NEXT_CHANGELOG.md on
main. Kept only the scripts env entry.
Co-authored-by: Isaac
@shreyas-goenka
shreyas-goenka added this pull request to the merge queueJul 13, 2026
Merged via the queue into main with commit a152b60Jul 13, 2026
26 checks passed
@shreyas-goenka
shreyas-goenka deleted the script-env-section branch July 13, 2026 08:42
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: a152b60

Run: 29236447430

Env❌​FAIL🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
❌​aws linux6134523101045:27
❌​aws windows61434476102365:50
❌​aws-ucws linux11152992820148:32
❌​aws-ucws windows111452930838185:30
🔄​azure linux334521101142:06
💚​azure windows44480102454:37
💚​azure-ucws linux62898855129:22
🔄​azure-ucws windows342839873149:05
💚​gcp linux44514101651:19
🔄​gcp windows334469102959:41
33 interesting tests: 17 FAIL, 9 flaky, 4 RECOVERED, 2 SKIP, 1 KNOWN
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K🟨​K🟨​K🔄​f💚​R💚​R🔄​f💚​R🔄​f
💚​TestAccept/bundle/invariant/no_drift🙈​S🙈​S💚​R💚​R🙈​S🙈​S💚​R💚​R🙈​S🙈​S
🔄​TestAccept/bundle/resources/dashboards/change-name✅​p🔄​f✅​p🔄​f🔄​f✅​p✅​p✅​p✅​p🔄​f
🔄​TestAccept/bundle/resources/dashboards/change-name/DATABRICKS_BUNDLE_ENGINE=direct✅​p🔄​f✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestAccept/bundle/resources/dashboards/change-name/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p🔄​f
❌​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions❌​F❌​F✅​p✅​p🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
❌​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct❌​F❌​F✅​p✅​p
❌​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform❌​F❌​F✅​p✅​p
❌​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions❌​F❌​F✅​p✅​p🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
❌​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct❌​F❌​F✅​p✅​p
❌​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform❌​F❌​F✅​p✅​p
❌​TestAccept/bundle/resources/postgres_databases/live_errors/missing_role🙈​s🙈​s❌​F❌​F🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
❌​TestAccept/bundle/resources/postgres_databases/live_errors/missing_role/DATABRICKS_BUNDLE_ENGINE=direct❌​F❌​F
🔄​TestAccept/bundle/resources/postgres_databases/recreate🙈​s🙈​s✅​p🔄​f🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
🔄​TestAccept/bundle/resources/postgres_databases/recreate/DATABRICKS_BUNDLE_ENGINE=direct✅​p🔄​f
❌​TestAccept/bundle/resources/postgres_endpoints/basic🙈​s🙈​s❌​F❌​F🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
❌​TestAccept/bundle/resources/postgres_endpoints/basic/DATABRICKS_BUNDLE_ENGINE=direct❌​F❌​F
❌​TestAccept/bundle/resources/postgres_endpoints/basic/DATABRICKS_BUNDLE_ENGINE=terraform❌​F❌​F
❌​TestAccept/bundle/resources/postgres_endpoints/replace_existing🙈​s🙈​s❌​F❌​F🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
❌​TestAccept/bundle/resources/postgres_endpoints/replace_existing/DATABRICKS_BUNDLE_ENGINE=direct❌​F❌​F
❌​TestAccept/bundle/resources/postgres_endpoints/replace_existing/DATABRICKS_BUNDLE_ENGINE=terraform❌​F❌​F
❌​TestAccept/bundle/resources/postgres_endpoints/update_autoscaling🙈​s🙈​s❌​F❌​F🙈​s🙈​s🙈​s🙈​s🙈​s🙈​s
❌​TestAccept/bundle/resources/postgres_endpoints/update_autoscaling/DATABRICKS_BUNDLE_ENGINE=direct❌​F❌​F
❌​TestAccept/bundle/resources/postgres_endpoints/update_autoscaling/DATABRICKS_BUNDLE_ENGINE=terraform❌​F❌​F
🔄​TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name🙈​S🙈​S💚​R💚​R🙈​S🙈​S💚​R🔄​f🙈​S🙈​S
🔄​TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p🔄​f
🙈​TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/ssh/connection🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🔄​TestSecretsPutSecretBytesValue✅​p🔄​f🙈​s🙈​s✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestSecretsPutSecretStringValue✅​p🔄​f🙈​s🙈​s✅​p✅​p✅​p✅​p✅​p✅​p
💚​TestFetchRepositoryInfoAPI_FromRepo💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
💚​TestFetchRepositoryInfoAPI_FromRepo/root💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
💚​TestFetchRepositoryInfoAPI_FromRepo/subdir💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
Top 50 slowest tests (at least 2 minutes):
durationenvtestname
15:39aws windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
14:20aws linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
13:25gcp linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
12:19gcp linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
11:27gcp windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
11:06aws windowsTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
10:01aws-ucws windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
9:15azure-ucws windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:52aws-ucws windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:24gcp windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:20gcp linuxTestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
8:10azure-ucws windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:42aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
7:37aws linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:30aws windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:28azure windowsTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:22gcp windowsTestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct
7:19aws-ucws linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:17azure-ucws linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:16azure linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:03azure-ucws linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:02aws windowsTestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct
7:01azure linuxTestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:52aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
6:51aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
6:48gcp windowsTestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
6:48gcp linuxTestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct
6:40azure windowsTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:29gcp linuxTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:11aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
6:11aws-ucws linuxTestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
5:58azure windowsTestAccept
5:50azure-ucws linuxTestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct
5:48aws-ucws linuxTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
5:41aws-ucws linuxTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
5:38aws windowsTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
5:36aws-ucws windowsTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
5:33aws-ucws linuxTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
5:28aws-ucws windowsTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
5:25aws-ucws linuxTestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
5:12aws linuxTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
5:08aws-ucws windowsTestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
5:07aws linuxTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
5:02aws linuxTestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct
4:57gcp linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:56gcp windowsTestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
4:54aws-ucws windowsTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
4:53gcp linuxTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
4:53aws linuxTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
4:52gcp linuxTestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct

ronaldz-db pushed a commit to ronaldz-db/cli that referenced this pull request Jul 15, 2026
…sue databricks#4179 (databricks#5923)
The fragment added by databricks#5831 linked the `env:` section changelog entry to
the original issue (databricks#4179) rather than the PR that implemented the
feature (databricks#5299). This includes both links.
deco-sdk-taggingBot added a commit that referenced this pull request Jul 15, 2026
## Release v1.8.0
### Notable Changes
* Auto-migrate a bundle from terraform to the direct engine when `bundle.engine` is `"direct"` (or `DATABRICKS_BUNDLE_ENGINE=direct`) and the post-deploy dry-run migration is clean; a warning is emitted if the dry-run surfaces errors or warnings so the automatic migration is skipped.
### CLI
* experimental `ssh connect`: bare `python`/`pip` in an interactive session now resolve to the environment interpreter (`$DATABRICKS_VIRTUAL_ENV`) instead of the system or cluster-libraries interpreter, so packages installed in the environment are importable without extra setup. The interactive shell is now non-login (`bash -i`) and the server seeds a `~/.bashrc` snippet that re-prepends the environment's bin directory to `PATH` ([#5888](#5888)).
* When Claude Code runs the CLI without the Databricks AI tooling installed, the CLI now prints a one-line recommendation on stderr to run `databricks aitools install`. The recommendation is shown at most once per hour per Claude session, and never for human callers or `aitools` commands.
* Fixed `databricks auth describe` misattributing a profile selected via `DATABRICKS_CONFIG_PROFILE` as `(from bundle)` when run inside a bundle root ([#5904](#5904)).
### Bundles
* `bundle generate` now warns when the generated configuration file is not matched by any pattern in the `include` section of `databricks.yml` ([#5868](#5868)).
* direct: Match UC Auto Upgrade managed property defaults with a wildcard pattern instead of enumerating each key ([#5877](#5877)).
* Recognize `ssh://` template URLs in `databricks bundle init` ([#5891](#5891)).
* `databricks bundle init` now reports an actionable error when given a template URL with an unsupported protocol (`http://`, `git://`, `ftp://`, `ftps://`) instead of failing with a confusing "not a bundle template" message ([#5902](#5902)).
* Added an `env:` section to `scripts.<name>` for declaring environment variables that may reference `${bundle.*}`, `${workspace.*}`, and `${var.*}` ([#4179](#4179), [#5299](#5299)).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature suggestion - variable interpolation in DAB script

4 participants

@shreyas-goenka@eng-dev-ecosystem-bot@denik@janniklasrose