apply_policy_default_values not working for spark_version

[AndreSepulveda]

Describe the issue

Documentation states that When set to true, fixed and default values from the policy will be used for fields that are omitted. When set to false, only fixed values from the policy will be applied.

When using apply_policy_default_values: true and omitting the spark_version, deployment fails even if there's a default spark_version set in the policy being used.

Configuration

bundle:
 name: my-bundle

targets:
 dev:
 default: true
 workspace:
 host: https://my-workspace.cloud.databricks.com

resources:
 jobs:
 my_job:
 name: my_job
 tasks:
 - task_key: my_task
 job_cluster_key: my_cluster
 spark_python_task:
 python_file: ${workspace.file_path}/src/main.py

 job_clusters:
 - job_cluster_key: my_cluster
 new_cluster:
 policy_id: my-policy-id
 apply_policy_default_values: true
 runtime_engine: STANDARD
 data_security_mode: SINGLE_USER
 num_workers: 0

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Update policy id and host url from configs
2. Run databricks bundle deploy

Expected Behavior

Expected deployment to use the default spark_version value defined in the policy.

Actual Behavior

Deployment fails saying the argument "spark_version" is required, but no definition was found.

OS and CLI version

Databricks CLI v0.248.0
macOS 15.3.1

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed

[andrewnester]

Related to this change in TF provider which is needed in order for DAB to correctly support this databricks/terraform-provider-databricks#4643

[AndreSepulveda]

Thanks for the reply @andrewnester

Do you know if #1327 is being addressed in the Go SDK / API ?

[pietern]

FYI -- the next CLI release happening today includes a fix for this issue.

@AndreSepulveda Regarding the issue you mention, the SDK and API definitions have changed in the meantime. If you are interested in using a policy that configures the Spark version to an auto:* value, I manually confirmed this works as of the next release.