direct: Add dashboards support by shreyas-goenka · Pull Request #3725 · databricks/cli

shreyas-goenka · 2025-10-06T11:24:08Z

Summary

This PR adds support for Databricks Lakeview Dashboards in the direct deployment mode (DATABRICKS_BUNDLE_ENGINE=direct-exp). Dashboards can now be deployed, updated, and deleted directly via API calls without using Terraform.

Key Features

1. Full Dashboard Lifecycle Management

Create: Creates draft dashboards and automatically publishes them with warehouse and credential settings
Update: Updates both dashboard content and publish settings (warehouse_id, embed_credentials)
Delete: Safely trashes dashboards with idempotent handling
Refresh: Fetches dashboard state from both draft and published endpoints in parallel

2. Smart Field Change Detection

The implementation uses FieldTriggers to determine the correct action for each field change:

Update Actions (in-place modifications):

display_name - Dashboard name changes
warehouse_id - SQL warehouse assignment changes
embed_credentials - Credential embedding toggle
serialized_dashboard - Dashboard content/layout changes

Recreate Actions (delete + create):

parent_path - Moving dashboard to different folder requires recreation

Skip Actions (ignored for diff):

Output-only fields: dashboard_id, path, create_time, update_time, lifecycle_state
etag - Skipped for local changes (user edits), but triggers updates for remote drift detection
serialized_dashboard - Skipped for remote drift (etag is sufficient)

3. Advanced Capabilities

Parallel API Calls: Refresh operations fetch both draft and published dashboard states concurrently using goroutines for better performance.

Automatic Parent Directory Creation: If the parent directory doesn't exist, it's automatically created before dashboard creation.

Idempotent Delete: Handles already-trashed dashboards gracefully by checking the dashboard state on 403 errors.

embed_credentials Edge Case: Unlike Terraform, this implementation properly detects changes to the embed_credentials field by fetching published dashboard state separately.

Serialized Dashboard Flexibility: Supports both string and map/object formats for serialized_dashboard in bundle configuration.

4. Comprehensive Test Coverage

Added 5 new acceptance test suites testing field change behavior:

change-name - Tests display_name updates
change-warehouse-id - Tests warehouse assignment updates
change-embed-credentials - Tests credential embedding toggle
change-serialized-dashboard - Tests dashboard content updates
change-parent-path - Tests parent path changes trigger recreate

Each test verifies:

Correct API methods (POST/PATCH/DELETE) are called
Dashboard ID changes (or doesn't change) as expected
Field values are properly updated in remote state

Testing

Existing and new unit and integration tests.

## Why Used in #3725 to provide local test coverage. ## Tests Existing tests pass with slight modifications.

eng-dev-ecosystem-bot · 2025-10-13T07:38:02Z

Run: 19101439812

	Env	❌FAIL	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip
💚	aws linux				1	1	356	592
💚	aws windows				1	1	357	591
💚	aws-ucws linux				1	1	479	487
💚	aws-ucws windows				1	1	480	486
❌	azure linux	2		4		1	351	591
❌	azure windows	2		4		1	352	590
💚	azure-ucws linux				1	1	475	486
🔄	azure-ucws windows			5	1	1	471	485
❌	gcp linux	14	1			1	341	593
❌	gcp windows	20	1			1	336	592

39 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚R	💚R	💚R	💚R	🔄f	🔄f	💚R	💚R	🟨K	🟨K
TestAccept/bundle/deployment/bind/experiment	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/deployment/bind/job/job-abort-bind	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/deployment/unbind/permissions	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/clusters/deploy/data_security_mode	✅p	✅p	✅p	✅p	🔄f	🔄f	✅p	🔄f	✅p	✅p
TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	🔄f	🔄f	✅p	🔄f	✅p	✅p
TestAccept/bundle/resources/clusters/deploy/update-after-create	✅p	✅p	✅p	✅p	❌F	❌F	✅p	🔄f	✅p	✅p
TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	❌F	❌F	✅p	🔄f	✅p	✅p
TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	🔄f	🔄f	✅p	🔄f	✅p	✅p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/pipelines/update	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/pipelines/update/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/pipelines/update/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/run/app-with-job	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
TestAccept/bundle/templates/default-python/combinations/classic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestSparkJarTaskDeployAndRunOnWorkspace	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestSparkJarTaskDeployAndRunOnWorkspace/Databricks_Runtime_15.4_LTS	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFsCpDir	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFsCpDir/dbfs_to_dbfs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFsRmEmptyDir	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFsRmEmptyDir/dbfs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFsRmFile	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFsRmFile/dbfs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F

…ange This merge incorporates changes from PR #3741 which adds an isLocal parameter to FieldTriggers and ClassifyChange methods to differentiate between local changes (user modifications) and remote changes (drift detection). Key changes: - Renamed ActionTypeUnset to ActionTypeUndefined - Updated FieldTriggers to accept isLocal bool parameter - Updated ClassifyChange to accept isLocal bool parameter - Consolidated dashboard's separate FieldTriggersLocal/Remote methods into single FieldTriggers(isLocal bool) method - Updated adapter to store separate trigger maps for local and remote 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Resolved conflicts: - bundle/direct/dresources/adapter.go: Merged documentation for FieldTriggers method, combining detailed explanation from HEAD with note about initialization timing from main - acceptance/bundle/resources/dashboards/test.toml: Kept HEAD version with proper warehouse requirements and etag replacements for new dashboard tests

direct: Add dashboards support
92fac76

shreyas-goenka temporarily deployed to test-trigger-is October 6, 2025 11:24 — with GitHub Actions Inactive

make change detection work
cbf97d3

shreyas-goenka temporarily deployed to test-trigger-is October 6, 2025 15:30 — with GitHub Actions Inactive

shreyas-goenka added 6 commits October 8, 2025 11:35

add tests for dahsboard generate
6e1d4f2

Replace FieldTriggers with FieldTriggersLocal and FieldTriggersRemote
574939c

better comments
d2f2566

remove golangci
bf3b696

update files
7f77809

Merge remote-tracking branch 'origin' into field-trigger-remote
33e06c7

shreyas-goenka mentioned this pull request Oct 8, 2025
Improve dashboards test server to match to the real server #3743
Merged

shreyas-goenka added 6 commits October 9, 2025 10:57

address comments
b59cd5d

address comments
c62217d

address comments
3405b17

better validation
d9d2cf1

address comments
f2720d3

fix lint
046a699

github-merge-queueBot pushed a commit that referenced this pull request Oct 13, 2025
Improve dashboards test server to match to the real server (#3743)
bd47132
## Why Used in #3725 to provide local test coverage. ## Tests Existing tests pass with slight modifications.

merge
0c20964

shreyas-goenka temporarily deployed to test-trigger-is October 13, 2025 07:34 — with GitHub Actions Inactive

shreyas-goenkaand others added 4 commits October 13, 2025 10:01

Merge remote-tracking branch 'origin/main' into direct/dashboard
64c6473

tests for all change; improrevements
a0bd138

shreyas-goenka temporarily deployed to test-trigger-is October 14, 2025 11:18 — with GitHub Actions Inactive

-
dff5d2b

shreyas-goenka temporarily deployed to test-trigger-is October 14, 2025 11:42 — with GitHub Actions Inactive

-
173c559

shreyas-goenka temporarily deployed to test-trigger-is October 14, 2025 14:44 — with GitHub Actions Inactive

shreyas-goenka added 5 commits October 28, 2025 17:36

split post and patch
8bb3d7e

Add plan for trashed dashboard
935b113

use real time replacements
41ca336

fix test
5d94f00

merge
2eb8b86

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 10:42 — with GitHub Actions Inactive

shreyas-goenka requested a review from denik October 29, 2025 10:42

fix test
79163e8

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 10:44 — with GitHub Actions Inactive

fix test
00084c4

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 11:47 — with GitHub Actions Inactive

add omitempty back
6b0422d

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 11:49 — with GitHub Actions Inactive

denik approved these changes Oct 29, 2025
View reviewed changes

shreyas-goenka added 16 commits October 29, 2025 13:37

fix windows test
ddff1a1

Merge remote-tracking branch 'origin' into direct/dashboard
5661607

Merge remote-tracking branch 'origin' into direct/dashboard
af20a88

address comments
d3f360e

Fix some tests
758ff98

fix windows test
5c5990b

merge
a2242f0

merge
2f971d2

fix other tests
e11fd37

merge
ade72f5

-
8f0c6ad

merge
8b0c9e2

cleaup
f8b7052

merge
f96117d

merge
8b8e505

fix windows test
13ff5c9

Navigation Menu

Uh oh!

direct: Add dashboards support - #3725

direct: Add dashboards support#3725
shreyas-goenka merged 104 commits into
mainfrom
direct/dashboard

shreyas-goenka commented Oct 6, 2025•
edited
Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 13, 2025•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

shreyas-goenka commented Oct 6, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Features

1. Full Dashboard Lifecycle Management

2. Smart Field Change Detection

Update Actions (in-place modifications):

Recreate Actions (delete + create):

Skip Actions (ignored for diff):

3. Advanced Capabilities

4. Comprehensive Test Coverage

Testing

Uh oh!

eng-dev-ecosystem-bot commented Oct 13, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shreyas-goenka commented Oct 6, 2025•
edited
Loading

eng-dev-ecosystem-bot commented Oct 13, 2025•
edited
Loading