Cluster gets recreated during deployment

[GewoonMaarten]

Describe the issue

When I deploy the cluster resource it triggers a Compute edited by ... event and I need to wait a bit before the cluster is running again.

The event:

{
 "previous_attributes": {
 "cluster_name": "[dev ...] Development cluster",
 "spark_version": "17.3.x-scala2.13",
 "spark_conf": {
 "spark.sql.shuffle.partitions": "auto"
 },
 "azure_attributes": {
 "first_on_demand": 1,
 "availability": "SPOT_WITH_FALLBACK_AZURE",
 "spot_bid_max_price": -1
 },
 "node_type_id": "Standard_D4ds_v5",
 "driver_node_type_id": "Standard_D4ds_v5",
 "custom_tags": {},
 "autotermination_minutes": 60,
 "enable_elastic_disk": true,
 "disk_spec": {},
 "cluster_source": "UI",
 "policy_id": "...",
 "enable_local_disk_encryption": false,
 "instance_source": {
 "node_type_id": "Standard_D4ds_v5"
 },
 "driver_instance_source": {
 "node_type_id": "Standard_D4ds_v5"
 },
 "data_security_mode": "USER_ISOLATION",
 "effective_spark_version": "17.3.x-scala2.13",
 "release_version": "17.3.10"
 },
 "attributes": {
 "cluster_name": "[dev ...] Development cluster",
 "spark_version": "17.3.x-scala2.13",
 "spark_conf": {
 "spark.sql.shuffle.partitions": "auto"
 },
 "azure_attributes": {
 "first_on_demand": 1,
 "availability": "SPOT_WITH_FALLBACK_AZURE",
 "spot_bid_max_price": -1
 },
 "node_type_id": "Standard_D4ds_v5",
 "driver_node_type_id": "Standard_D4ds_v5",
 "custom_tags": {},
 "autotermination_minutes": 60,
 "enable_elastic_disk": true,
 "disk_spec": {},
 "cluster_source": "UI",
 "policy_id": "...",
 "enable_local_disk_encryption": false,
 "instance_source": {
 "node_type_id": "Standard_D4ds_v5"
 },
 "driver_instance_source": {
 "node_type_id": "Standard_D4ds_v5"
 },
 "data_security_mode": "USER_ISOLATION",
 "effective_spark_version": "17.3.x-scala2.13"
 },
 "previous_cluster_size": {
 "autoscale": {
 "min_workers": 2,
 "max_workers": 20,
 "target_workers": 13
 }
 },
 "cluster_size": {
 "autoscale": {
 "min_workers": 2,
 "max_workers": 20,
 "target_workers": 13
 }
 },
 "user": "..."
}

The only property that has been updated is release_version, which I cannot set. I also checked #4933 and #3286, but I think I have a different problem databricks bundle plan -o json does not show anything intresetting:

❯ databricks bundle plan -o json
Building default...
{
 "plan": {
 "resources.clusters.development_cluster": {
 "action": "update"
 },
 "resources.jobs.test": {
 "action": "update"
 }
 }
}

Configuration

variables:
 policy_id:
 description: Cluster policy id for cost allocation
 lookup:
 cluster_policy: "test"
 spark_version:
 description: Shared Spark runtime version for all clusters
 default: "17.3.x-scala2.13"
 node_type_id:
 description: Shared node type for all clusters
 default: "Standard_D4ds_v5"

targets:
 dev:
 mode: development
 default: true

 cluster_id: ${resources.clusters.development_cluster.id}

 resources:
 clusters:
 development_cluster:
 cluster_name: Development cluster
 spark_version: ${var.spark_version}
 node_type_id: ${var.node_type_id}
 spark_conf:
 spark.sql.shuffle.partitions: auto
 autoscale:
 min_workers: 2
 max_workers: 20
 autotermination_minutes: 60
 azure_attributes:
 availability: SPOT_WITH_FALLBACK_AZURE
 data_security_mode: DATA_SECURITY_MODE_AUTO
 policy_id: ${var.policy_id}

Steps to reproduce the behavior

1. Run databricks bundle deploy
2. On the second deployment cluster gets recreated

Expected Behavior

No updates to config don't trigger a cluster update

Actual Behavior

Cluster is updated

OS and CLI version

• Databricks CLI v0.278.0
• Databricks CLI v0.299.0

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed

[andrewnester]

@GewoonMaarten thanks for reporting! Do you use Terraform or direct deployment mode? My assumption it's a Terraform mode. In this case I would suggest you to try to use direct deployment, where this type of the issues should not exist
https://docs.databricks.com/aws/en/dev-tools/bundles/direct

[GewoonMaarten]

@andrewnester Thanks for the response! I migrated to the direct deployment. It still recreates the cluster, but for a different reason. There are some custom_tags automatically applied by our policy which I have not defined in our databricks.yml. Is there a way to ignore those during deployment or is the only choice I have to also define them in our config?

[andrewnester]

@GewoonMaarten do you have custom_tags section in the databricks.yml but it has a different set of tags or you don't have custom_tags in databricks.yml at all?

[GewoonMaarten]

@andrewnester I dont have any custom_tags. This is my full config:

 resources:
 clusters:
 development_cluster:
 cluster_name: Development cluster
 apply_policy_default_values: true
 spark_version: ${var.spark_version}
 node_type_id: ${var.node_type_id}
 spark_conf:
 spark.sql.shuffle.partitions: auto
 autoscale:
 min_workers: 2
 max_workers: 20
 autotermination_minutes: 60
 azure_attributes:
 availability: SPOT_WITH_FALLBACK_AZURE
 data_security_mode: DATA_SECURITY_MODE_AUTO
 policy_id: ${var.policy_id}

[grniss]

Hello, are there any updates on these? I faced a similar problem that might be useful for this.

Currently I have problem with databricks bundle plan that always displays some jobs as update even after we just deployed.

What I assume is that the policy in job_cluster is not yet fetched and overridden to the incoming cluster state, so the comparison is always not equal.

I've pivoted my job in these scenarios, deploy and plan:

• serverless_cluster → no change
• job_cluster without policy_id and apply_policy_default_values: false → no change
• job_cluster with policy_id lookup (didn't test by explicit define policy_id, but I think it same) → always update
• job_cluster with apply_policy_default_values: true → always update

This is also happen on Terraform engine, but I think that is intentional as we didn't apply resource lookup.

[jonmozley-ow]

FWIW I'm also seeing this issue as I can't migrate from TF to the direct engine if it thinks a change is going to apply every time. If I try it with a new target defined with the direct engine, I get this in output json which doesn't really tell us much:

 "job_clusters[job_cluster_key='regression_test_etl_cluster'].new_cluster.apply_policy_default_values": {
 "action": "update",
 "old": true,
 "new": true
 },

As you found @grniss , if I set apply_policy_default_values: false then I don't see the issue.

There also seems to still be issues where not all policy defaults are actually applied such as spark_env_vars or custom tags. I got the impression from this issue that it's only a TF issue but I'm seeing it in direct as well.
#3125

[denik]

The apply_policy_default_values issue is fixed in #5693, it'll be released today as 1.5.0

[jonmozley-ow]

Hi, I've tried updating to 1.5.0 today but unfortunately still seeing the same issue. I set apply_policy_default_values as true, then per my screenshot below:

1. Ran a plan and saw 3 job changes pending
2. Ran a deploy
3. Ran another plan, it still thinks 3 job changes are pending
4. Ran a plan with output to json - you can see at the top of file it's planning to replace "true" with "true" and manually apply a custom tag that should come from a policy.

The same happens if I try to deploy it under a different target name as well, so it's not specific to any remote/cached state.

[denik]

@jonmozley-ow thanks, I see what's going on. The fix only targets clusters not jobs with jobs_clusters. I'll make another PR.