Skip to content

Reverse dependency order for delete operations in direct engine - #4105

Merged
denik merged 12 commits into
databricks:mainfrom
varundeepsaini:fix/dependency-ordered-deletion
Dec 15, 2025
Merged

Reverse dependency order for delete operations in direct engine#4105
denik merged 12 commits into
databricks:mainfrom
varundeepsaini:fix/dependency-ordered-deletion

Conversation

@varundeepsaini

@varundeepsainivarundeepsaini commented Dec 6, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes: #4089

Changes

When deleting resources with dependencies, the direct deployment engine now deletes dependents before their dependencies. Previously, resources were deleted in deployment order, which could cause errors when a dependency was deleted before resources that depend on it.

Edge direction in the dependency graph is now determined per-node based on action type:

  • Delete actions: reverse edge (dependent → dependency)
  • Other actions: normal edge (dependency → dependent)

@varundeepsaini
varundeepsainiforce-pushed the fix/dependency-ordered-deletion branch from d6ebbc5 to 2042b3aCompareDecember 9, 2025 17:45
@varundeepsaini

varundeepsaini commented Dec 9, 2025

Copy link
Copy Markdown
ContributorAuthor

@andrewnester@denik
could you please review this, and queue the tests

@denik

denik commented Dec 10, 2025

Copy link
Copy Markdown
Contributor

Thanks @varundeepsaini this looks on the right track.

You need to update output of all acceptance tests (make test-update) and deduplicate the entries:

=== FAIL: acceptance TestAccept/bundle/migrate/basic/DATABRICKS_BUNDLE_ENGINE=direct (re-run 1) (5.35s)
acceptance_test.go:1401: No effective updates from BundleConfig.default_name
acceptance_test.go:819: Diff:
--- bundle/migrate/basic/out.new_state.json
+++ /tmp/TestAcceptbundlemigratebasicDATABRICKS_BUNDLE_ENGINE=direct2716390764/001/out.new_state.json
@@ -51,7 +51,15 @@
"volume_catalog_name": "mycat",
"volume_storage_location": "s3://deco-uc-prod-isolated-aws-us-east-1/metastore/[UUID]/volumes/[UUID]"
}
- }
+ },
+ "depends_on": [
+ "resources.jobs.test_job",
+ "resources.jobs.test_job",
+ "resources.jobs.test_job",
+ "resources.volumes.test_volume",
+ "resources.volumes.test_volume",
+ "resources.volumes.test_volume"
+ ]
},
"resources.volumes.test_volume": {
"__id__": "mycat.myschema.myvol",

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Dec 10, 2025•
edited
Loading

Copy link
Copy Markdown
Collaborator

Commit: 7a2ff5e

Run: 20229246583

Env🟨​KNOWN💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux7138064620:08
🟨​aws windows7138264418:08
🟨​aws-ucws linux34152353127:37
🟨​aws-ucws windows34152552924:01
💚​azure linux1338164418:49
💚​azure windows1338364218:01
💚​azure-ucws linux1352052925:44
💚​azure-ucws windows1352252724:48
💚​gcp linux1337065018:25
💚​gcp windows1337264817:40
8 interesting tests: 7 KNOWN, 1 SKIP
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K🟨​K🟨​K💚​R💚​R💚​R💚​R💚​R💚​R
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K🟨​K🟨​K🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K🟨​K🟨​K
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
Top 25 slowest tests (at least 2 minutes):
durationenvtestname
7:19aws-ucws linuxTestAccept/bundle/resources/synced_database_tables/basic
6:24aws-ucws windowsTestAccept/bundle/resources/synced_database_tables/basic
6:02aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:49aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:39aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:37gcp windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:37aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:36aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:32aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:26gcp windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:25gcp linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:25aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:57gcp linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:25azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:19azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
4:15azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:57azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:55azure linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:35azure linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:49azure-ucws windowsTestAccept/bundle/resources/synced_database_tables/basic
2:45azure-ucws linuxTestAccept/bundle/resources/synced_database_tables/basic
2:29azure-ucws linuxTestAccept
2:23aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:14gcp linuxTestAccept
2:12azure linuxTestAccept

denik
denik reviewed Dec 10, 2025
View reviewed changes
Comment threadacceptance/bundle/resource_deps/job_id/out.plan_delete.direct.json Outdated
@varundeepsaini

varundeepsaini commented Dec 11, 2025

Copy link
Copy Markdown
ContributorAuthor

@denik
Could you please review this, and queue the tests

denik
denik reviewed Dec 11, 2025
View reviewed changes

@denikdenik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, minor comments remaining. Please also rebase.

Added bundle/direct/graph_test.go

I don't see this file.

Comment threadacceptance/bundle/resource_deps/job_id_big_graph/script Outdated
Comment threadacceptance/bundle/resource_deps/job_id_big_graph/script Outdated
Comment threadacceptance/bundle/resource_deps/job_id_big_graph/script Outdated
denik
denik reviewed Dec 12, 2025
View reviewed changes
Comment threadbundle/direct/dstate/state.go Outdated
Comment threadacceptance/test.toml Outdated
@varundeepsaini

varundeepsaini commented Dec 12, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@denik can you review once again, and queue the tests (the integration tests as well)

denik
denik reviewed Dec 12, 2025
View reviewed changes
Comment threadacceptance/bundle/resource_deps/job_id_chain/databricks.yml Outdated
@denik

denik commented Dec 12, 2025

Copy link
Copy Markdown
Contributor

Looks great! Please also update NEXT_CHANGELOG.md

@varundeepsaini

varundeepsaini commented Dec 12, 2025

Copy link
Copy Markdown
ContributorAuthor

updated the changelog

@varundeepsainivarundeepsaini mentioned this pull request Dec 12, 2025
Closed
@github-actions

github-actionsBot commented Dec 15, 2025

Copy link
Copy Markdown
Contributor

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 4105
  • Commit SHA: 7a2ff5e923241891b768f2c9a278245d8ba16014

Checks will be approved automatically on success.

denik
denik approved these changes Dec 15, 2025
View reviewed changes

@denikdenik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks!

@denik
denik added this pull request to the merge queueDec 15, 2025
Merged via the queue into databricks:main with commit 700366bDec 15, 2025
19 checks passed
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Dec 15, 2025

Copy link
Copy Markdown
Collaborator

Commit: 700366b

Run: 20231484559

Env🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux7141663246:37
🟨​aws windows7141863046:24
🟨​aws-ucws linux34157751170:27
🟨​aws-ucws windows34157950959:23
💚​azure linux1341763050:23
💚​azure windows1341962848:01
💚​azure-ucws linux1357450966:26
🔄​azure-ucws windows3357450770:19
💚​gcp linux1339763947:33
💚​gcp windows1339963752:08
10 interesting tests: 7 KNOWN, 2 flaky, 1 SKIP
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K🟨​K🟨​K💚​R💚​R💚​R🔄​f💚​R💚​R
🔄​TestAccept/bundle/resources/clusters/run/spark_python_task✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p
🔄​TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K🟨​K🟨​K🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K🟨​K🟨​K
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
Top 50 slowest tests (at least 2 minutes):
durationenvtestname
20:10azure-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
19:46azure-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
15:51azure-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
14:57gcp windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
14:53azure-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
14:02aws-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
13:04gcp linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
12:38gcp windowsTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
11:13aws-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
10:54aws-ucws windowsTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
10:53gcp windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
10:30aws-ucws linuxTestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
10:18gcp linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
10:14azure-ucws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
9:20gcp windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
9:05azure windowsTestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
9:03gcp linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:48azure linuxTestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
8:46azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
8:45azure-ucws linuxTestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
8:42aws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:39azure-ucws windowsTestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_15.4_LTS
8:27aws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:26gcp linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
8:22aws linuxTestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_15.4_LTS
8:15aws-ucws linuxTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform
8:14aws-ucws linuxTestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_15.4_LTS
8:10azure-ucws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
8:09aws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
8:08gcp linuxTestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_14.3_LTS
8:07gcp windowsTestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_15.4_LTS
8:06azure-ucws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
8:05aws-ucws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
8:02azure linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:02aws linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
8:01aws-ucws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:00azure-ucws linuxTestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_14.3_LTS
8:00azure-ucws linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:59gcp windowsTestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_14.3_LTS
7:56azure-ucws windowsTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:55aws windowsTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform
7:54gcp windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
7:52gcp linuxTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
7:50aws windowsTestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct
7:46azure linuxTestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:44aws-ucws windowsTestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_15.4_LTS
7:44gcp windowsTestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
7:42azure-ucws linuxTestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_15.4_LTS
7:42azure-ucws linuxTestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_13.3_LTS
7:39aws-ucws linuxTestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_13.3_LTS

@varundeepsaini
varundeepsaini deleted the fix/dependency-ordered-deletion branch December 15, 2025 14:10
deco-sdk-taggingBot added a commit that referenced this pull request Dec 18, 2025
@deco-sdk-tagging
[Release] Release v0.281.0
0f1bd8e
## Release v0.281.0
### CLI
* Fix lakeview publish to default `embed_credentials` to false ([#4066](#4066))
### Bundles
* Add support for configurable catalog/schema for dashboards ([#4130](#4130))
* Pass SYSTEM\_ACCESSTOKEN from env to the Terraform provider ([#4135](#4135))
* `bundle deployment migrate`: when running `bundle plan` propagate `-var` arguments.
* engine/direct: New option --plan to `bundle deploy` to deploy previously saved plan (saved plan with `bundle plan -o json`) ([#4134](#4134))
* engine/direct: Fix dependency-ordered deletion by persisting depends\_on in state ([#4105](#4105))
### Dependency updates
* Upgrade Go SDK to 0.94.0 ([#4148](#4148))
* Upgrade Terraform provider to 1.100.0 ([#4150](#4150))
denik added a commit that referenced this pull request May 20, 2026
Varun Deep Saini@denik
Reverse dependency order for delete operations in direct engine (#4105)
49ee7d7
Fixes: #4089 ## Changes
When deleting resources with dependencies, the direct deployment engine
now deletes dependents before their dependencies. Previously, resources
were deleted in deployment order, which could cause errors when a
dependency was deleted before resources that depend on it.
Edge direction in the dependency graph is now determined per-node based
on action type:
- Delete actions: reverse edge (dependent → dependency) - Other actions: normal edge (dependency → dependent)
---------
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>
denik pushed a commit that referenced this pull request May 20, 2026
@deco-sdk-tagging
[Release] Release v0.281.0
52c4551
## Release v0.281.0
### CLI
* Fix lakeview publish to default `embed_credentials` to false ([#4066](#4066))
### Bundles
* Add support for configurable catalog/schema for dashboards ([#4130](#4130))
* Pass SYSTEM\_ACCESSTOKEN from env to the Terraform provider ([#4135](#4135))
* `bundle deployment migrate`: when running `bundle plan` propagate `-var` arguments.
* engine/direct: New option --plan to `bundle deploy` to deploy previously saved plan (saved plan with `bundle plan -o json`) ([#4134](#4134))
* engine/direct: Fix dependency-ordered deletion by persisting depends\_on in state ([#4105](#4105))
### Dependency updates
* Upgrade Go SDK to 0.94.0 ([#4148](#4148))
* Upgrade Terraform provider to 1.100.0 ([#4150](#4150))
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@denikdenikdenik approved these changes
@andrewnesterandrewnesterAwaiting requested review from andrewnester
@anton-107anton-107Awaiting requested review from anton-107
@pieternpieternAwaiting requested review from pietern
@shreyas-goenkashreyas-goenkaAwaiting requested review from shreyas-goenka

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Deleting resources should respect the dependencies between resources

3 participants

@varundeepsaini@denik@eng-dev-ecosystem-bot