Skip to content

bundle migrate: handle added/removed resources gracefully - #5463

Merged
denik merged 2 commits into
mainfrom
denik/migration-added-removed
Jun 8, 2026
Merged

bundle migrate: handle added/removed resources gracefully#5463
denik merged 2 commits into
mainfrom
denik/migration-added-removed

Conversation

@denik

@denikdenik commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Changes

  • Update "bundle deployment migrate" to support added/removed resources:
    • If resource is present in databricks.yml but not in terraform state the migrate command no longer fails, it skips the resource, it'll be deployed by direct engine.
    • If resource is present in the state but not in databricks.yml, migration preserves their ID in the state so that it can be deleted by direct engine.

Why

Towards auto-migration: we should migrate any situation without errors.

Tests

New acceptance tests.

@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 09:36 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 09:36 — with GitHub Actions Inactive
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Commit: 1ad2f9d

Run: 27144759214

Env🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux7152619268:16
🟨​aws windows711526292416:33
💚​aws-ucws linux7153578408:52
💚​aws-ucws windows71535983813:09
💚​azure linux1172649246:33
💚​azure windows11726692211:46
💚​azure-ucws linux1173628368:21
💚​azure-ucws windows11736483413:15
🔄​gcp linux211725892717:31
🔄​gcp windows31726092514:30
27 interesting tests: 15 SKIP, 7 KNOWN, 5 flaky
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K💚​R💚​R💚​R💚​R💚​R💚​R💚​R🔄​f
🔄​TestAccept/bundle/deployment/bind/alert🙈​s🙈​s🙈​s🙈​s✅​p✅​p✅​p✅​p✅​p🔄​f
🔄​TestAccept/bundle/deployment/bind/alert/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p✅​p🔄​f
🙈​TestAccept/bundle/invariant/no_drift🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🔄​TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p
🔄​TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🙈​TestAccept/bundle/resources/postgres_branches/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/recreate🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/replace_existing🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/update_protected🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/without_branch_id🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_endpoints/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_endpoints/recreate🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_projects/update_display_name🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/synced_database_tables/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_indexes/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_indexes/grants/select🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/ssh/connection🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🔄​TestSecretsPutSecretBytesValue✅​p🔄​f🙈​s🙈​s✅​p✅​p✅​p✅​p✅​p✅​p
Top 28 slowest tests (at least 2 minutes):
durationenvtestname
7:17azure windowsTestAccept
6:05aws-ucws windowsTestAccept
6:02azure-ucws windowsTestAccept
4:50gcp linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:42gcp windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:33gcp linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:23gcp windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:25aws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:18aws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:16aws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:12aws-ucws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:11azure-ucws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:00aws-ucws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:58azure windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:56azure linuxTestAccept
2:52azure linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:51azure-ucws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:48azure-ucws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:47aws-ucws linuxTestAccept
2:46azure-ucws linuxTestAccept
2:43azure windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:41aws-ucws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:39aws-ucws windowsTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:35azure linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:32azure-ucws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:30aws linuxTestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:06aws-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/basic/DATABRICKS_BUNDLE_ENGINE=terraform
2:06aws-ucws windowsTestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=terraform

@denik
denikforce-pushed the denik/migration-added-removed branch from 1117995 to fe0d059CompareJune 8, 2026 11:05
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 11:06 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 11:06 — with GitHub Actions Inactive
@denik
denikforce-pushed the denik/migration-added-removed branch from fe0d059 to b4d3e40CompareJune 8, 2026 11:18
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 11:19 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 11:19 — with GitHub Actions Inactive
@denik
denikforce-pushed the denik/migration-added-removed branch from b4d3e40 to 7bf336cCompareJune 8, 2026 12:23
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 12:24 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 12:24 — with GitHub Actions Inactive
@denik
denikforce-pushed the denik/migration-added-removed branch from 7bf336c to f7c3b39CompareJune 8, 2026 12:26
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 12:27 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 12:27 — with GitHub Actions Inactive
@denik
denikforce-pushed the denik/migration-added-removed branch from f7c3b39 to f9b0ae3CompareJune 8, 2026 12:50
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 12:51 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 12:51 — with GitHub Actions Inactive
Resources in databricks.yml but not in terraform state (added since
last deploy) are now skipped during migration and will be created on
the first direct deploy, instead of failing the migration.
Resources in terraform state but not in databricks.yml (removed since
last deploy) now have their ID preserved in the direct state so the
next direct deploy will plan and execute deletion, instead of failing
the migration.
Co-authored-by: Isaac
@denik
denikforce-pushed the denik/migration-added-removed branch from f9b0ae3 to bb2f595CompareJune 8, 2026 12:53

@pieternpietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this mesh with automigration?

@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 14:28 — with GitHub Actions Inactive
@denik
deniktemporarily deployed to test-trigger-is June 8, 2026 14:28 — with GitHub Actions Inactive
@denik

denik commented Jun 8, 2026

Copy link
Copy Markdown
ContributorAuthor

How does this mesh with automigration?

This directly enables auto-migration by not requiring a prior deploy. (Although we might still choose to auto-migration after terraform deployment while we still have terraform implementation in).

@denik
denik enabled auto-merge June 8, 2026 14:34
@denik
denik added this pull request to the merge queueJun 8, 2026
Merged via the queue into main with commit 7826da9Jun 8, 2026
23 checks passed
@denik
denik deleted the denik/migration-added-removed branch June 8, 2026 15:51
deco-sdk-taggingBot added a commit that referenced this pull request Jun 10, 2026
## Release v1.3.0
### Notable Changes
* The `direct` deployment engine is now Generally Available and the default for new deployments. To opt out, set `engine: terraform` under `bundle` in your `databricks.yml` or set `DATABRICKS_BUNDLE_ENGINE=terraform`. Existing deployments keep their current engine; see https://docs.databricks.com/aws/en/dev-tools/bundles/direct to migrate.
### CLI
* Added the `databricks quickstart` command, a short introduction to the CLI that prints a human-friendly guide interactively and an agent-oriented version when run non-interactively ([#5464](#5464)).
* Add `databricks version --check` to report whether a newer CLI version is available and print the upgrade command for the detected install method ([#5469](#5469)).
* `databricks auth describe` now verifies credentials against both the workspace and account endpoints before reporting a failure, fixing false "Unable to authenticate" errors for account console profiles ([#5479](#5479)).
* `databricks auth login` no longer prompts for workspace selection when logging in to an account console host (`https://accounts.*`). Pass `--workspace-id` explicitly to store a workspace ID on such a profile ([#5504](#5504)).
* `databricks auth profiles --skip-validate` no longer makes any network calls; the host metadata fetch is skipped along with validation ([#5530](#5530)).
### Bundles
* Set the default `data_security_mode` to `DATA_SECURITY_MODE_AUTO` in bundle templates ([#5452](#5452)).
* Mark vector search index index_subtype as backend_default to prevent drift after deployment ([#5454](#5454)).
* `bundle deployment migrate`: handle resources added to or removed from `databricks.yml` since the last Terraform deploy ([#5463](#5463)).
* Add the `genie_spaces` bundle resource for managing Databricks Genie spaces as code, plus `bundle generate genie-space` to import an existing space. Direct deployment engine only ([#5282](#5282)).
* Fix spurious recreate of schemas and volumes whose names use mixed case ([#5531](#5531)).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@denik@eng-dev-ecosystem-bot@pietern