Uh oh!
There was an error while loading. Please reload this page.
Configure __apply_policy_default_values_allow_list for task and job clusters - #3255
Merged
Conversation
pietern
marked this pull request as ready for review
July 15, 2025 14:29
pietern
requested review from
andrewnester, anton-107, denik and shreyas-goenka
as code ownersJuly 15, 2025 14:29
andrewnester
approved these changes
Jul 16, 2025
| // If the field "policy_id" is not set, do nothing. | ||
| if _, ok := v.Get("policy_id").AsString(); !ok { | ||
| return v, nil |
Contributor
There was a problem hiding this comment.
Is it worth to log a warning here because apply_policy_default_values is set but not policy_id?
ContributorAuthor
There was a problem hiding this comment.
Thanks for flagging. I think not because it doesn't make sense to use one without the other.
Any validation for a condition like this we should apply earlier, as it will apply to either deployment backend. The stuff here is really TF specific to work around the issue.
pietern
enabled auto-merge
July 16, 2025 10:37
Collaborator
23 failing tests:
|
auto-merge was automatically disabled
July 16, 2025 12:23
Pull request was closed
github-merge-queueBot
removed this pull request from the merge queue due to a manual request
Jul 16, 2025
Uh oh!
There was an error while loading. Please reload this page.
github-actionsBot
pushed a commit
that referenced
this pull request
Jul 16, 2025
## Changes Upgrade TF provider to 1.85.0 Important changes: * Added support for `database` resource for Apps * Includes a change needed for enabling `allow_policy_default_values` fix#3255 ## Tests Existing tests pass <!-- If your PR needs to be included in the release notes for next release, add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
Uh oh!
There was an error while loading. Please reload this page.
deco-sdk-taggingBot
added a commit
that referenced
this pull request
Jul 16, 2025
## Release v0.260.0 ### Notable Changes * Added support for creating SQL warehouses in DABs ([#3129](#3129)) ### Dependency updates * Upgrade Go SDK to 0.75.0 ([#3256](#3256)) ### CLI * Add `databricks psql` command to connect to Lakebase with a single command ([#3128](#3128)) ### Bundles * Jobs that use cluster policy default values for their cluster configuration now correctly update those defaults on every deployment ([#3255](#3255)). * Add upper and lower helper methods for bundle templates ([#3242](#3242))
denik pushed a commit
that referenced
this pull request
May 20, 2026
## Changes Upgrade TF provider to 1.85.0 Important changes: * Added support for `database` resource for Apps * Includes a change needed for enabling `allow_policy_default_values` fix#3255 ## Tests Existing tests pass <!-- If your PR needs to be included in the release notes for next release, add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
denik pushed a commit
that referenced
this pull request
May 20, 2026
… clusters (#3255) ## Changes Automatically reapply cluster policy defaults for task and job clusters when updating existing jobs. Note: if you relied on task and job cluster settings that originate from a cluster policy _not_ changing on deploy, even though the cluster policy defaults they were drawn from did change, then you'll need to explicitly include those values in your bundle configuration from now on. ## Why The latest version of the Terraform provider comes with a feature specifically for re-applying cluster policy defaults for task and job clusters when updating a job (databricks/terraform-provider-databricks#4834). Before this change, it was not possible to reapply cluster policy defaults for an existing job, as Terraform keeps server-side default values in its state. Users had to destroy and recreate jobs for new defaults to take effect. With the Terraform dependency in DABs, the same happened for DABs users, and cluster policy defaults were effectively "snapshotted" on the first deploy, and never updated thereafter. This is contrary to the behavior we expect to see. Unless a field is explicitly configured in the bundle configuration, cluster policy defaults should be applied on every deploy, regardless of the job being created or updated. This change passes the list of fields explicitly specified in the bundle configuration to Terraform via the `__allow_policy_default_values_allow_list` property introduced in the linked PR. ## Tests * Unit tests pass. * Manually confirmed that when the cluster policy for a job cluster is changed, the fields that aren't explicitly specified in the bundle configuration draw their default values from the new cluster policy. <!-- If your PR needs to be included in the release notes for next release, add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
denik pushed a commit
that referenced
this pull request
May 20, 2026
## Release v0.260.0 ### Notable Changes * Added support for creating SQL warehouses in DABs ([#3129](#3129)) ### Dependency updates * Upgrade Go SDK to 0.75.0 ([#3256](#3256)) ### CLI * Add `databricks psql` command to connect to Lakebase with a single command ([#3128](#3128)) ### Bundles * Jobs that use cluster policy default values for their cluster configuration now correctly update those defaults on every deployment ([#3255](#3255)). * Add upper and lower helper methods for bundle templates ([#3242](#3242))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Automatically reapply cluster policy defaults for task and job clusters when updating existing jobs.
Note: if you relied on task and job cluster settings that originate from a cluster policy not changing on deploy, even though the cluster policy defaults they were drawn from did change, then you'll need to explicitly include those values in your bundle configuration from now on.
Why
The latest version of the Terraform provider comes with a feature specifically for re-applying cluster policy defaults for task and job clusters when updating a job (databricks/terraform-provider-databricks#4834). Before this change, it was not possible to reapply cluster policy defaults for an existing job, as Terraform keeps server-side default values in its state. Users had to destroy and recreate jobs for new defaults to take effect.
With the Terraform dependency in DABs, the same happened for DABs users, and cluster policy defaults were effectively "snapshotted" on the first deploy, and never updated thereafter.
This is contrary to the behavior we expect to see. Unless a field is explicitly configured in the bundle configuration, cluster policy defaults should be applied on every deploy, regardless of the job being created or updated.
This change passes the list of fields explicitly specified in the bundle configuration to Terraform via the
__allow_policy_default_values_allow_listproperty introduced in the linked PR.Tests