Skip to content

Use local Terraform state only when lineage match - #1588

Merged
shreyas-goenka merged 15 commits into
mainfrom
fix/tf-lineage
Jul 18, 2024
Merged

Use local Terraform state only when lineage match#1588
shreyas-goenka merged 15 commits into
mainfrom
fix/tf-lineage

Conversation

@shreyas-goenka

@shreyas-goenkashreyas-goenka commented Jul 10, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

DABs deployments should be isolated if root_path and workspace host are different. This PR fixes a bug where local terraform state gets piggybacked if the same cwd is used to deploy two isolated deployments for the same bundle target. This can happen if:

  1. A user switches to a different identity on the same machine.
  2. The workspace host URL the bundle/target points to is changed.
  3. A user changes the root_path while doing bundle development.

To solve this problem we rely on the lineage field available in the terraform state, which is a uuid identifying unique terraform deployments. There's a 1:1 mapping between a terraform deployment and a bundle deployment.

For more details on how lineage works in terraform, see: https://developer.hashicorp.com/terraform/language/state/backends#manual-state-pull-push

Tests

Manually verified that changing the identity no longer results in the incorrect terraform state being used. Also, new unit tests are added.

@shreyas-goenka

shreyas-goenka commented Jul 10, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Integration tests are passing.

@shreyas-goenka
shreyas-goenka marked this pull request as ready for review July 10, 2024 17:44
@shreyas-goenkashreyas-goenka changed the title Invalidate local terraform state only when lineage matchUse local terraform state only when lineage matchJul 10, 2024
@pieternpietern changed the title Use local terraform state only when lineage matchUse local Terraform state only when lineage matchJul 15, 2024
pietern
pietern reviewed Jul 15, 2024
View reviewed changes
Comment threadbundle/deploy/terraform/state_pull.go Outdated
Comment threadbundle/deploy/terraform/state_pull.go Outdated
Comment threadbundle/deploy/terraform/state_pull.go
Comment threadbundle/deploy/terraform/state_pull_test.go
pietern
pietern reviewed Jul 17, 2024
View reviewed changes
Comment threadbundle/deploy/terraform/state_pull.go Outdated
Comment threadbundle/deploy/terraform/state_pull.go Outdated
Comment threadbundle/deploy/terraform/state_pull.go Outdated
Comment threadbundle/deploy/terraform/state_pull.go Outdated
@shreyas-goenka
shreyas-goenka added this pull request to the merge queueJul 18, 2024
Merged via the queue into main with commit 5b65358Jul 18, 2024
@shreyas-goenka
shreyas-goenka deleted the fix/tf-lineage branch July 18, 2024 09:52
andrewnester added a commit that referenced this pull request Jul 18, 2024
@andrewnester
Release v0.223.3
6617c7c
CLI:
* [Fix] Do not buffer files in memory when downloading ([#1599](#1599)).
Bundles:
* Allow artifacts (JARs, wheels) to be uploaded to UC Volumes ([#1591](#1591)).
* Upgrade TF provider to 1.48.3 ([#1600](#1600)).
* Fixed job name normalisation for bundle generate ([#1601](#1601)).
Internal:
* Add UUID to uniquely identify a deployment state ([#1595](#1595)).
* Track multiple locations associated with a `dyn.Value` ([#1510](#1510)).
* Attribute Terraform API requests the CLI ([#1598](#1598)).
* Use local Terraform state only when lineage match ([#1588](#1588)).
* Implement readahead cache for Workspace API calls ([#1582](#1582)).
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.43.0 to 0.43.2 ([#1594](#1594)).
@andrewnesterandrewnester mentioned this pull request Jul 18, 2024
Closed
andrewnester added a commit that referenced this pull request Jul 18, 2024
@andrewnester
Release v0.224.0
44022b3
CLI:
* Do not buffer files in memory when downloading ([#1599](#1599)).
Bundles:
* Allow artifacts (JARs, wheels) to be uploaded to UC Volumes ([#1591](#1591)).
* Upgrade TF provider to 1.48.3 ([#1600](#1600)).
* Fixed job name normalisation for bundle generate ([#1601](#1601)).
Internal:
* Add UUID to uniquely identify a deployment state ([#1595](#1595)).
* Track multiple locations associated with a `dyn.Value` ([#1510](#1510)).
* Attribute Terraform API requests the CLI ([#1598](#1598)).
* Implement readahead cache for Workspace API calls ([#1582](#1582)).
* Use local Terraform state only when lineage match ([#1588](#1588)).
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.43.0 to 0.43.2 ([#1594](#1594)).
@andrewnesterandrewnester mentioned this pull request Jul 18, 2024
Merged
github-merge-queueBot pushed a commit that referenced this pull request Jul 18, 2024
@andrewnester
Release v0.224.0 (#1604)
c8ce18f
CLI:
* Do not buffer files in memory when downloading
([#1599](#1599)).
Bundles:
* Allow artifacts (JARs, wheels) to be uploaded to UC Volumes
([#1591](#1591)).
* Upgrade TF provider to 1.48.3
([#1600](#1600)).
* Fixed job name normalisation for bundle generate
([#1601](#1601)).
Internal:
* Add UUID to uniquely identify a deployment state
([#1595](#1595)).
* Track multiple locations associated with a `dyn.Value`
([#1510](#1510)).
* Attribute Terraform API requests the CLI
([#1598](#1598)).
* Implement readahead cache for Workspace API calls
([#1582](#1582)).
* Use local Terraform state only when lineage match
([#1588](#1588)).
* Add read-only mode for extension aware workspace filer
([#1609](#1609)).
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.43.0 to 0.43.2
([#1594](#1594)).
denik pushed a commit that referenced this pull request May 20, 2026
@shreyas-goenka
Use local Terraform state only when lineage match (#1588)
3782edb
## Changes
DABs deployments should be isolated if `root_path` and workspace host
are different. This PR fixes a bug where local terraform state gets
piggybacked if the same cwd is used to deploy two isolated deployments
for the same bundle target. This can happen if:
1. A user switches to a different identity on the same machine. 2. The workspace host URL the bundle/target points to is changed.
3. A user changes the `root_path` while doing bundle development.
To solve this problem we rely on the lineage field available in the
terraform state, which is a uuid identifying unique terraform
deployments. There's a 1:1 mapping between a terraform deployment and a
bundle deployment.
For more details on how lineage works in terraform, see:
https://developer.hashicorp.com/terraform/language/state/backends#manual-state-pull-push
## Tests
Manually verified that changing the identity no longer results in the
incorrect terraform state being used. Also, new unit tests are added.
denik pushed a commit that referenced this pull request May 20, 2026
@andrewnester
Release v0.224.0 (#1604)
75f5044
CLI:
* Do not buffer files in memory when downloading
([#1599](#1599)).
Bundles:
* Allow artifacts (JARs, wheels) to be uploaded to UC Volumes
([#1591](#1591)).
* Upgrade TF provider to 1.48.3
([#1600](#1600)).
* Fixed job name normalisation for bundle generate
([#1601](#1601)).
Internal:
* Add UUID to uniquely identify a deployment state
([#1595](#1595)).
* Track multiple locations associated with a `dyn.Value`
([#1510](#1510)).
* Attribute Terraform API requests the CLI
([#1598](#1598)).
* Implement readahead cache for Workspace API calls
([#1582](#1582)).
* Use local Terraform state only when lineage match
([#1588](#1588)).
* Add read-only mode for extension aware workspace filer
([#1609](#1609)).
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.43.0 to 0.43.2
([#1594](#1594)).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pieternpieternpietern approved these changes
@andrewnesterandrewnesterAwaiting requested review from andrewnester

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shreyas-goenka@pietern