Uh oh!
There was an error while loading. Please reload this page.
Add validation for single node clusters - #1909
Merged
Merged
Conversation
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 18, 2024 14:56 — with 
GitHub Actions
Inactive
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 18, 2024 14:56 — with GitHub Actions
Inactive
| } | ||
| } | ||
| func TestValidateSingleNodePipelineClustersFail(t *testing.T) { |
ContributorAuthor
There was a problem hiding this comment.
We duplicate the test cases for pipeline clusters because they have a separate proto / Go SDK type for pipeline clusters that mirrors the spec from the cluster APIs.
Duplicating the validation code and test seemed easier and more robust than generalizing the code by converting between the two cluster structs or having a common representation.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
pietern
reviewed
Nov 19, 2024
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 19, 2024 22:07 — with GitHub Actions
Inactive
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 19, 2024 22:07 — with GitHub Actions
Inactive
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 19, 2024 22:11 — with GitHub Actions
Inactive
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 19, 2024 22:11 — with GitHub Actions
Inactive
pietern
reviewed
Nov 22, 2024
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 22, 2024 13:54 — with GitHub Actions
Inactive
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 22, 2024 13:54 — with GitHub Actions
Inactive
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 22, 2024 14:10 — with GitHub Actions
Inactive
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 22, 2024 14:10 — with GitHub Actions
Inactive
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 22, 2024 14:28 — with GitHub Actions
Inactive
Contributor
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
shreyas-goenkatemporarily deployed
to
test-trigger-is
November 22, 2024 14:28 — with GitHub Actions
Inactive
eng-dev-ecosystem-bot
commented
Nov 22, 2024
Collaborator
Test Details: go/deco-tests/11974490258 |
pietern
approved these changes
Nov 22, 2024
shreyas-goenka
enabled auto-merge
November 22, 2024 14:36
pietern added a commit
that referenced
this pull request
Dec 5, 2024
**New features for Databricks Asset Bundles:** This release adds support for managing Unity Catalog volumes as part of your bundle configuration. Bundles: * Add DABs support for Unity Catalog volumes ([#1762](#1762)). * Support lookup by name of notification destinations ([#1922](#1922)). * Extend "notebook not found" error to warn about missing extension ([#1920](#1920)). * Skip sync warning if no sync paths are defined ([#1926](#1926)). * Add validation for single node clusters ([#1909](#1909)). * Fix segfault in bundle summary command ([#1937](#1937)). * Add the `bundle_uuid` helper function for templates ([#1947](#1947)). * Add default value for `volume_type` for DABs ([#1952](#1952)). * Properly read Git metadata when running inside workspace ([#1945](#1945)). * Upgrade TF provider to 1.59.0 ([#1960](#1960)). Internal: * Breakout variable lookup into separate files and tests ([#1921](#1921)). * Add golangci-lint v1.62.2 ([#1953](#1953)). Dependency updates: * Bump golang.org/x/term from 0.25.0 to 0.26.0 ([#1907](#1907)). * Bump github.com/Masterminds/semver/v3 from 3.3.0 to 3.3.1 ([#1930](#1930)). * Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 ([#1932](#1932)). * Bump github.com/databricks/databricks-sdk-go from 0.51.0 to 0.52.0 ([#1931](#1931)).
github-merge-queueBot
pushed a commit
that referenced
this pull request
Dec 5, 2024
**New features for Databricks Asset Bundles:** This release adds support for managing Unity Catalog volumes as part of your bundle configuration. Bundles: * Add DABs support for Unity Catalog volumes ([#1762](#1762)). * Support lookup by name of notification destinations ([#1922](#1922)). * Extend "notebook not found" error to warn about missing extension ([#1920](#1920)). * Skip sync warning if no sync paths are defined ([#1926](#1926)). * Add validation for single node clusters ([#1909](#1909)). * Fix segfault in bundle summary command ([#1937](#1937)). * Add the `bundle_uuid` helper function for templates ([#1947](#1947)). * Add default value for `volume_type` for DABs ([#1952](#1952)). * Properly read Git metadata when running inside workspace ([#1945](#1945)). * Upgrade TF provider to 1.59.0 ([#1960](#1960)). Internal: * Breakout variable lookup into separate files and tests ([#1921](#1921)). * Add golangci-lint v1.62.2 ([#1953](#1953)). Dependency updates: * Bump golang.org/x/term from 0.25.0 to 0.26.0 ([#1907](#1907)). * Bump github.com/Masterminds/semver/v3 from 3.3.0 to 3.3.1 ([#1930](#1930)). * Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 ([#1932](#1932)). * Bump github.com/databricks/databricks-sdk-go from 0.51.0 to 0.52.0 ([#1931](#1931)).
denik pushed a commit
that referenced
this pull request
May 20, 2026
## Changes This PR adds a warning validating that the configuration for a single node cluster is valid for interactive, job, job-task, and pipeline clusters. Note: We skip the validation if a cluster policy is configured because the policy is likely to configure `spark_conf` / `custom_tags` itself. Note: Terrform originally only had validation for interactive, job, and job-task clusters. This PR adding the validation for pipeline clusters as well is new. This PR follows the same logic as we used to have in Terraform. The validation was removed from Terraform because we had no way to demote the error to a warning: databricks/terraform-provider-databricks#4222 ### Background Single-node clusters require `spark_conf` and `custom_tags` to be correctly set in the cluster definition for them to function optimally. The cluster will be created even if incorrectly configured, but its performance will not be great. For example, if both `spark_conf` and `custom_tags` are not set and `num_workers` is 0, then only the driver process will be launched on the cluster compute instance thus leading to sub-optimal utilization of available compute resources and no parallelization across worker processes when processing a spark query. ### Issue This PR addresses some issues reported in #1546 ## Tests Unit tests and manually. Example output of the warning: ``` ➜ bundle-playground git:(master) ✗ cli bundle validate Warning: Single node cluster is not correctly configured at resources.pipelines.bar.clusters[0] in databricks.yml:29:11 num_workers should be 0 only for single-node clusters. To create a valid single node cluster please ensure that the following properties are correctly set in the cluster specification: spark_conf: spark.databricks.cluster.profile: singleNode spark.master: local[*] custom_tags: ResourceClass: SingleNode Name: foobar Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Workspace/Users/shreyas.goenka@databricks.com/.bundle/foobar/default Found 1 warning ```
denik pushed a commit
that referenced
this pull request
May 20, 2026
**New features for Databricks Asset Bundles:** This release adds support for managing Unity Catalog volumes as part of your bundle configuration. Bundles: * Add DABs support for Unity Catalog volumes ([#1762](#1762)). * Support lookup by name of notification destinations ([#1922](#1922)). * Extend "notebook not found" error to warn about missing extension ([#1920](#1920)). * Skip sync warning if no sync paths are defined ([#1926](#1926)). * Add validation for single node clusters ([#1909](#1909)). * Fix segfault in bundle summary command ([#1937](#1937)). * Add the `bundle_uuid` helper function for templates ([#1947](#1947)). * Add default value for `volume_type` for DABs ([#1952](#1952)). * Properly read Git metadata when running inside workspace ([#1945](#1945)). * Upgrade TF provider to 1.59.0 ([#1960](#1960)). Internal: * Breakout variable lookup into separate files and tests ([#1921](#1921)). * Add golangci-lint v1.62.2 ([#1953](#1953)). Dependency updates: * Bump golang.org/x/term from 0.25.0 to 0.26.0 ([#1907](#1907)). * Bump github.com/Masterminds/semver/v3 from 3.3.0 to 3.3.1 ([#1930](#1930)). * Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 ([#1932](#1932)). * Bump github.com/databricks/databricks-sdk-go from 0.51.0 to 0.52.0 ([#1931](#1931)).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
This PR adds a warning validating that the configuration for a single node cluster is valid for interactive, job, job-task, and pipeline clusters.
Note: We skip the validation if a cluster policy is configured because the policy is likely to configure
spark_conf/custom_tagsitself.Note: Terrform originally only had validation for interactive, job, and job-task clusters. This PR adding the validation for pipeline clusters as well is new.
This PR follows the same logic as we used to have in Terraform. The validation was removed from Terraform because we had no way to demote the error to a warning:
databricks/terraform-provider-databricks#4222
Background
Single-node clusters require
spark_confandcustom_tagsto be correctly set in the cluster definition for them to function optimally. The cluster will be created even if incorrectly configured, but its performance will not be great.For example, if both
spark_confandcustom_tagsare not set andnum_workersis 0, then only the driver process will be launched on the cluster compute instance thus leading to sub-optimal utilization of available compute resources and no parallelization across worker processes when processing a spark query.Issue
This PR addresses some issues reported in #1546
Tests
Unit tests and manually.
Example output of the warning: