Skip to content

Default to DATA_SECURITY_MODE_AUTO in bundle templates - #5452

Merged
pietern merged 4 commits into
mainfrom
dab-template-auto-dsm
Jun 5, 2026
Merged

Default to DATA_SECURITY_MODE_AUTO in bundle templates#5452
pietern merged 4 commits into
mainfrom
dab-template-auto-dsm

Conversation

@pietern

@pieternpietern commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Changes

The default data_security_mode for job clusters created by the bundle templates is now DATA_SECURITY_MODE_AUTO. With Auto, the access mode is selected automatically: Standard by default, and Dedicated when the cluster configuration requires it (for example ML Runtime, GPUs, R, Databricks Container Services, or global init scripts). See Access modes.

Why

Auto selects an access mode that works across the broadest set of configurations without the user having to pick one. The job definition stores and returns auto verbatim; the access mode is resolved per run when the job cluster launches, and that resolved value is never written back to the job — so bundles see no drift.

Tests

  • Acceptance tests updated and passing for both deploy engines.
  • Deployed against a production workspace on both engines and confirmed the value round-trips as DATA_SECURITY_MODE_AUTO with no drift, and that Auto resolves to a concrete access mode at cluster launch.
  • Manually verified the dbt-sql template's classic (non-serverless) job cluster by deploying and running the job: Auto round-trips and resolves correctly and the run completes.

This pull request and its description were written by Isaac.

Set the default data_security_mode for job clusters in the built-in bundle
templates (default and dbt-sql) to DATA_SECURITY_MODE_AUTO, so new projects
get an access mode that is selected automatically.
Co-authored-by: Isaac
@pietern
pieterntemporarily deployed to test-trigger-is June 5, 2026 12:53 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 5, 2026 12:53 — with GitHub Actions Inactive
Co-authored-by: Isaac
@pietern
pieterntemporarily deployed to test-trigger-is June 5, 2026 12:55 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 5, 2026 12:55 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 5, 2026 12:59 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 5, 2026 12:59 — with GitHub Actions Inactive
@pietern
pietern enabled auto-merge June 5, 2026 13:00
@pietern
pieterntemporarily deployed to test-trigger-is June 5, 2026 13:20 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 5, 2026 13:20 — with GitHub Actions Inactive
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Commit: 251bf31

Run: 27017311628

Ignoring .gh-logs/27017311628/update-check-action/action.yml: cannot extract env
Ignoring .gh-logs/27017311628/gh-report-action/gh_pr_comment.py: cannot extract env
Ignoring .gh-logs/27017311628/gh-report-action/action.yml: cannot extract env

Env💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
💚​aws linux7152619235:52
💚​aws windows7152639217:53
💚​aws-ucws linux7153578377:17
💚​aws-ucws windows7153598359:41
💚​azure linux1172649215:44
💚​azure windows1172669197:42
💚​azure-ucws linux1173628336:59
💚​azure-ucws windows1173648319:09
💚​gcp linux1172609246:38
💚​gcp windows1172629229:48
22 interesting tests: 15 SKIP, 7 RECOVERED
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
💚​TestAccept💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
🙈​TestAccept/bundle/invariant/no_drift🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
💚​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions💚​R💚​R💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
💚​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct💚​R💚​R💚​R💚​R
💚​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform💚​R💚​R💚​R💚​R
💚​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions💚​R💚​R💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
💚​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct💚​R💚​R💚​R💚​R
💚​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform💚​R💚​R💚​R💚​R
🙈​TestAccept/bundle/resources/postgres_branches/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/recreate🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/replace_existing🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/update_protected🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/without_branch_id🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_endpoints/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_endpoints/recreate🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_projects/update_display_name🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/synced_database_tables/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_indexes/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_indexes/grants/select🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/ssh/connection🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
Top 26 slowest tests (at least 2 minutes):
durationenvtestname
4:58gcp linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:40gcp windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:35gcp linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:15gcp windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:23aws-ucws windowsTestAccept
3:21aws windowsTestAccept
3:19gcp windowsTestAccept
3:16azure windowsTestAccept
3:14azure-ucws windowsTestAccept
3:11aws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:04aws-ucws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:58azure-ucws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:55azure windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:54aws-ucws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:53aws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:52azure windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:47azure linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:42azure-ucws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:34azure-ucws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:34aws-ucws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:33aws-ucws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:32aws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:32azure-ucws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:31azure linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:25aws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:01aws windowsTestSecretsPutSecretStringValue

@pietern
pietern added this pull request to the merge queueJun 5, 2026
Merged via the queue into main with commit ec7c659Jun 5, 2026
25 checks passed
@pietern
pietern deleted the dab-template-auto-dsm branch June 5, 2026 15:03
deco-sdk-taggingBot added a commit that referenced this pull request Jun 10, 2026
## Release v1.3.0
### Notable Changes
* The `direct` deployment engine is now Generally Available and the default for new deployments. To opt out, set `engine: terraform` under `bundle` in your `databricks.yml` or set `DATABRICKS_BUNDLE_ENGINE=terraform`. Existing deployments keep their current engine; see https://docs.databricks.com/aws/en/dev-tools/bundles/direct to migrate.
### CLI
* Added the `databricks quickstart` command, a short introduction to the CLI that prints a human-friendly guide interactively and an agent-oriented version when run non-interactively ([#5464](#5464)).
* Add `databricks version --check` to report whether a newer CLI version is available and print the upgrade command for the detected install method ([#5469](#5469)).
* `databricks auth describe` now verifies credentials against both the workspace and account endpoints before reporting a failure, fixing false "Unable to authenticate" errors for account console profiles ([#5479](#5479)).
* `databricks auth login` no longer prompts for workspace selection when logging in to an account console host (`https://accounts.*`). Pass `--workspace-id` explicitly to store a workspace ID on such a profile ([#5504](#5504)).
* `databricks auth profiles --skip-validate` no longer makes any network calls; the host metadata fetch is skipped along with validation ([#5530](#5530)).
### Bundles
* Set the default `data_security_mode` to `DATA_SECURITY_MODE_AUTO` in bundle templates ([#5452](#5452)).
* Mark vector search index index_subtype as backend_default to prevent drift after deployment ([#5454](#5454)).
* `bundle deployment migrate`: handle resources added to or removed from `databricks.yml` since the last Terraform deploy ([#5463](#5463)).
* Add the `genie_spaces` bundle resource for managing Databricks Genie spaces as code, plus `bundle generate genie-space` to import an existing space. Direct deployment engine only ([#5282](#5282)).
* Fix spurious recreate of schemas and volumes whose names use mixed case ([#5531](#5531)).
pietern added a commit that referenced this pull request Jun 17, 2026
#5452 shipped and was moved to CHANGELOG.md upstream; it was carried over here
through the changelog merge conflicts.
Co-authored-by: Isaac
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@pietern@eng-dev-ecosystem-bot@lennartkats-db