direct: Support partial plan/deploy with --select

NEXT_CHANGELOG.md

@@ -11,5 +11,6 @@
 * Retry transient HTTP 5xx and 408 errors in direct deployment engine ([#5349](https://github.com/databricks/cli/pull/5349), [#5364](https://github.com/databricks/cli/pull/5364)).
 * Preserve `.designer.ipynb` suffix when translating notebook task paths so Lakeflow Designer files referenced from a `notebook_task` resolve correctly in the workspace ([#5370](https://github.com/databricks/cli/pull/5370)).
 * Fix script output dropping last line without trailing newline ([#4995](https://github.com/databricks/cli/pull/4995)).
+* Add `--select` flag to `bundle plan` and `bundle deploy` to plan/deploy a subset of resources (e.g. `--select my_job` or `--select jobs.my_job`); resources referenced by the selection are included transitively. Direct engine only ([#5413](https://github.com/databricks/cli/pull/5413)).
 
 ### Dependency updates

acceptance/bundle/help/bundle-deploy/output.txt

@@ -20,6 +20,7 @@ Flags:
  --force-lock Force acquisition of deployment lock.
  -h, --help help for deploy
  --plan string Path to a JSON plan file to apply instead of planning (direct engine only).
+ --select strings Deploy only the specified resource (e.g. 'my_job' or 'jobs.my_job'). Can be repeated or comma-separated.
 
 Global Flags:
  --debug enable debug logging

acceptance/bundle/select/ambiguous/databricks.yml

@@ -0,0 +1,13 @@
+bundle:
+ name: select-ambiguous
+
+# "thing" is defined as both a job and a pipeline. This is what an ambiguous
+# --select selector would resolve to, but UniqueResourceKeys rejects it at load
+# time, so the bundle never reaches selector resolution.
+resources:
+ jobs:
+ thing:
+ name: job-thing
+ pipelines:
+ thing:
+ name: pipeline-thing

acceptance/bundle/select/ambiguous/output.txt

@@ -0,0 +1,16 @@
+
+>>> musterr [CLI] bundle plan --select thing
+Error: multiple resources or scripts have been defined with the same key: thing
+ at resources.jobs.thing
+ resources.pipelines.thing
+ in databricks.yml:10:7
+ databricks.yml:13:7
+
+
+>>> musterr [CLI] bundle deploy --select thing
+Error: multiple resources or scripts have been defined with the same key: thing
+ at resources.jobs.thing
+ resources.pipelines.thing
+ in databricks.yml:10:7
+ databricks.yml:13:7
+

acceptance/bundle/select/ambiguous/script

@@ -0,0 +1,2 @@
+trace musterr $CLI bundle plan --select thing
+trace musterr $CLI bundle deploy --select thing

acceptance/bundle/select/basic/databricks.yml.tmpl

@@ -0,0 +1,17 @@
+bundle:
+ name: select-$UNIQUE_NAME
+
+resources:
+ jobs:
+ bar:
+ name: bar-$UNIQUE_NAME
+
+ foo:
+ name: foo-$UNIQUE_NAME
+ tasks:
+ - task_key: run_bar
+ run_job_task:
+ job_id: ${resources.jobs.bar.id}
+
+ baz:
+ name: baz-$UNIQUE_NAME

acceptance/bundle/select/basic/output.txt

@@ -0,0 +1,141 @@
+
+>>> [CLI] bundle plan --select bar
+create jobs.bar
+
+Plan: 1 to add, 0 to change, 0 to delete, 0 unchanged
+
+>>> [CLI] bundle plan --select jobs.bar --select jobs.baz
+create jobs.bar
+create jobs.baz
+
+Plan: 2 to add, 0 to change, 0 to delete, 0 unchanged
+
+>>> [CLI] bundle plan --select jobs.bar,jobs.baz
+create jobs.bar
+create jobs.baz
+
+Plan: 2 to add, 0 to change, 0 to delete, 0 unchanged
+
+>>> [CLI] bundle plan --select jobs.foo
+create jobs.bar
+create jobs.foo
+
+Plan: 2 to add, 0 to change, 0 to delete, 0 unchanged
+
+=== bundle deploy --select jobs.foo
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
+
+=== Telemetry:
+select_used true
+
+>>> print_requests.py --sort //jobs
+{
+ "method": "POST",
+ "path": "/api/2.2/jobs/create",
+ "body": {
+ "deployment": {
+ "kind": "BUNDLE",
+ "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/state/metadata.json"
+ },
+ "edit_mode": "UI_LOCKED",
+ "format": "MULTI_TASK",
+ "max_concurrent_runs": 1,
+ "name": "bar-[UNIQUE_NAME]",
+ "queue": {
+ "enabled": true
+ }
+ }
+}
+{
+ "method": "POST",
+ "path": "/api/2.2/jobs/create",
+ "body": {
+ "deployment": {
+ "kind": "BUNDLE",
+ "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/state/metadata.json"
+ },
+ "edit_mode": "UI_LOCKED",
+ "format": "MULTI_TASK",
+ "max_concurrent_runs": 1,
+ "name": "foo-[UNIQUE_NAME]",
+ "queue": {
+ "enabled": true
+ },
+ "tasks": [
+ {
+ "run_job_task": {
+ "job_id": [NUMID]
+ },
+ "task_key": "run_bar"
+ }
+ ]
+ }
+}
+
+>>> [CLI] bundle summary
+Name: select-[UNIQUE_NAME]
+Target: default
+Workspace:
+ User: [USERNAME]
+ Path: /Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default
+Resources:
+ Jobs:
+ bar:
+ Name: bar-[UNIQUE_NAME]
+ URL: [DATABRICKS_URL]/jobs/[NUMID]?w=[NUMID]
+ baz:
+ Name: baz-[UNIQUE_NAME]
+ URL: (not deployed)
+ foo:
+ Name: foo-[UNIQUE_NAME]
+ URL: [DATABRICKS_URL]/jobs/[NUMID]?w=[NUMID]
+
+=== Full plan after partial deploy
+>>> [CLI] bundle plan
+create jobs.baz
+
+Plan: 1 to add, 0 to change, 0 to delete, 2 unchanged
+
+=== Full deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
+
+>>> print_requests.py --sort //jobs
+{
+ "method": "POST",
+ "path": "/api/2.2/jobs/create",
+ "body": {
+ "deployment": {
+ "kind": "BUNDLE",
+ "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/state/metadata.json"
+ },
+ "edit_mode": "UI_LOCKED",
+ "format": "MULTI_TASK",
+ "max_concurrent_runs": 1,
+ "name": "baz-[UNIQUE_NAME]",
+ "queue": {
+ "enabled": true
+ }
+ }
+}
+
+=== Full plan again
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 3 unchanged
+
+=== Destroy
+>>> [CLI] bundle destroy --auto-approve
+The following resources will be deleted:
+ delete resources.jobs.bar
+ delete resources.jobs.baz
+ delete resources.jobs.foo
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default
+
+Deleting files...
+Destroy complete!

acceptance/bundle/select/basic/script

@@ -0,0 +1,47 @@
+envsubst '$UNIQUE_NAME' < databricks.yml.tmpl > databricks.yml
+
+cleanup() {
+ title "Destroy"
+ trace $CLI bundle destroy --auto-approve
+ rm -f out.requests.txt
+}
+trap cleanup EXIT
+
+# --- selector resolution ---
+# Unqualified name, unique across resource types.
+trace $CLI bundle plan --select bar
+# Repeated --select.
+trace $CLI bundle plan --select jobs.bar --select jobs.baz
+# Comma-separated --select (equivalent to repeating the flag).
+trace $CLI bundle plan --select jobs.bar,jobs.baz
+# Qualified name; foo pulls in its dependency bar but not the independent baz.
+trace $CLI bundle plan --select jobs.foo
+
+# --- partial deploy ---
+# Serialize the filtered plan to a temp file (kept out of the recorded output: the
+# JSON embeds remote state that differs between local and cloud), then deploy it:
+# inline, or via --plan (READPLAN=1). The deploy is not traced because readplanarg
+# varies the command line between READPLAN variants, which must produce identical output.
+$CLI bundle plan --select jobs.foo -o json > plan.json
+title "bundle deploy --select jobs.foo\n"
+$CLI bundle deploy --select jobs.foo $(readplanarg plan.json)
+# The deploy reports that --select was used via telemetry.
+title "Telemetry:\n"
+print_telemetry_bool_values | grep '^select_used '
+# Only bar and foo were created, never baz.
+trace print_requests.py --sort //jobs
+# Summary after the partial deploy: foo and bar are deployed, baz is not.
+trace $CLI bundle summary
+
+# --- full plan/deploy after the partial deploy ---
+# foo and bar are already deployed, so only baz remains to create.
+title "Full plan after partial deploy"
+trace $CLI bundle plan
+$CLI bundle plan -o json > plan-full.json
+title "Full deploy\n"
+$CLI bundle deploy $(readplanarg plan-full.json)
+# Only baz is created this time.
+trace print_requests.py --sort //jobs
+# Everything is deployed now: no changes.
+title "Full plan again"
+trace $CLI bundle plan

acceptance/bundle/select/basic/test.toml

@@ -0,0 +1,11 @@
+Local = true
+Cloud = true
+RecordRequests = true
+# --select is only supported by the direct engine, so this happy-path test runs on
+# direct only (the rejection on terraform is covered by bundle/select/rejected).
+EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]
+# Also run a variant that deploys a serialized (already filtered) plan via --plan.
+EnvMatrix.READPLAN = ["", "1"]
+# databricks.yml and the serialized plans are generated at runtime; the plan JSON
+# is kept out of the recorded output because it embeds local/cloud-specific state.
+Ignore = [".databricks", ".gitignore", "databricks.yml", "plan.json", "plan-full.json"]

acceptance/bundle/select/missing/databricks.yml

@@ -0,0 +1,7 @@
+bundle:
+ name: select-errors
+
+resources:
+ jobs:
+ my_job:
+ name: my-job

acceptance/bundle/select/missing/output.txt

@@ -0,0 +1,12 @@
+
+>>> [CLI] bundle plan --select no_such_resource
+Error: no such resource: no_such_resource
+
+
+Exit code: 1
+
+>>> [CLI] bundle plan --select jobs.no_such_job
+Error: no such resource: jobs.no_such_job
+
+
+Exit code: 1

acceptance/bundle/select/missing/script

@@ -0,0 +1,8 @@
+# Selectors are resolved by the ResolveSelect mutator during initialize, before the
+# engine is known, so these errors are identical on both engines.
+
+# Unqualified name that matches no resource.
+errcode trace $CLI bundle plan --select no_such_resource
+
+# Qualified name whose resource does not exist.
+errcode trace $CLI bundle plan --select jobs.no_such_job

acceptance/bundle/select/rejected/databricks.yml

@@ -0,0 +1,7 @@
+bundle:
+ name: select-rejected
+
+resources:
+ jobs:
+ my_job:
+ name: my-job

acceptance/bundle/select/rejected/output.txt

@@ -0,0 +1,12 @@
+
+>>> [CLI] bundle plan --select my_job
+Error: --select is only supported with the direct engine. See https://docs.databricks.com/aws/en/dev-tools/bundles/direct
+
+
+Exit code: 1
+
+>>> [CLI] bundle deploy --select my_job
+Error: --select is only supported with the direct engine. See https://docs.databricks.com/aws/en/dev-tools/bundles/direct
+
+
+Exit code: 1

acceptance/bundle/select/rejected/script

@@ -0,0 +1,3 @@
+# --select is only supported by the direct engine; both plan and deploy reject it.
+errcode trace $CLI bundle plan --select my_job
+errcode trace $CLI bundle deploy --select my_job

acceptance/bundle/select/rejected/test.toml

@@ -0,0 +1,2 @@
+# --select is rejected on the terraform engine with an actionable error.
+EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["terraform"]

acceptance/bundle/select/test.toml

@@ -0,0 +1,3 @@
+Local = true
+Cloud = false
+Ignore = [".databricks", ".gitignore"]

bundle/bundle.go

@@ -145,6 +145,10 @@ type Bundle struct {
 	// files
 	AutoApprove bool
 
+	// Select contains resource selectors passed via --select flag.
+	// When non-empty, only the specified resources are included in deployment.
+	Select []string
+
 	// SkipLocalFileValidation makes path translation tolerant of missing local files.
 	// When set, TranslatePaths computes workspace paths without verifying files exist.
 	// Used by config-remote-sync: a user may modify resource paths remotely (e.g.,