Only show installable projects in 'databricks labs list' by janniklasrose · Pull Request #5560 · databricks/cli

janniklasrose · 2026-06-11T18:23:49Z

Changes

databricks labs list showed every non-archived, non-fork repository in the databrickslabs GitHub org — currently 39 — but only repositories that ship a labs.yml manifest can actually be installed (currently 8: blueprint, dlt-meta, dqx, lakebridge, lsql, pylint-plugin, sandbox, ucx). Picking anything else from the list failed databricks labs install with Error: remote: read labs.yml from GitHub: not found (error message improved separately in #5559), so the listing mostly advertised projects that cannot be installed.

With this change, the filter is narrowed to labs that are tagged as databricks-cli-installable, currently 3.

Output before (39 entries, abridged) / after (3 entries):

Name Description
blueprint Baseline for Databricks Labs projects written in P...
lakebridge Accelerates migrations to Databricks by automating...
ucx Automated migrations to Unity Catalog

Tests

Unit tests

This pull request and its description were written by Isaac, an AI coding agent.

'databricks labs list' showed every non-archived, non-fork repository in the databrickslabs GitHub org (currently 39), but only repositories that ship a labs.yml manifest at the root of their release tag can actually be installed (currently 8). Everything else failed 'databricks labs install' with a not-found error. Filter the listing to repositories that have a root labs.yml on their default branch, checked concurrently via raw.githubusercontent.com (not subject to the low unauthenticated GitHub API rate limit) and cached for 24 hours like the repository list itself. Co-authored-by: Isaac

Co-authored-by: Isaac

eng-dev-ecosystem-bot · 2026-06-11T19:29:58Z

Integration test report

Commit: 1ff24cb

Run: 28436899542

	Env	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🟨	aws linux	7		1	13	231	1037	5:03
🟨	aws windows	7		1	13	233	1035	5:39
💚	aws-ucws linux			8	13	315	955	4:25
💚	aws-ucws windows			8	13	317	953	3:34
🔄	azure linux		2	2	15	229	1036	5:38
🔄	azure windows		2	2	15	231	1034	4:28
💚	azure-ucws linux			2	15	317	952	4:21
💚	azure-ucws windows			2	15	319	950	3:20
💚	gcp linux			2	15	230	1038	3:27
💚	gcp windows			2	15	232	1036	2:32

23 interesting tests: 13 SKIP, 7 KNOWN, 2 flaky, 1 RECOVERED

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
🟨	TestAccept	🟨K	🟨K	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/permissions	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🙈	TestAccept/bundle/resources/postgres_branches/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/recreate	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/replace_existing	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/update_protected	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/without_branch_id	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_endpoints/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_projects/update_display_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/synced_database_tables/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
💚	TestFetchRepositoryInfoAPI_FromRepo	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
🔄	TestFetchRepositoryInfoAPI_FromRepo/root	✅p	✅p	✅p	✅p	🔄f	🔄f	✅p	✅p	✅p	✅p
🔄	TestFetchRepositoryInfoAPI_FromRepo/subdir	✅p	✅p	✅p	✅p	🔄f	🔄f	✅p	✅p	✅p	✅p

simonfaltum

Reviewed the full diff plus the supporting packages (localcache, cmd/labs/github, clear_cache.go), and ran an independent second-model pass over the same diff; both converged on the same two issues, so requesting changes for those (details inline):

labs clear-cache does not know about the new cache file.
An offline cold start writes an empty installable cache that then sticks for 24h, and (1) means clear-cache cannot fix it.

Both fixes are small. Two smaller notes inline (changelog wording given #5559 is still open, and a test nit).

Checked and found sound: the errgroup filter (writes to distinct slice elements, first-error semantics, limit 10, ctx propagation), preserved ordering and archived/fork semantics, graceful offline behavior when caches exist, the raw.githubusercontent choice and its failure mode (failing loudly beats caching a partial list for 24h), no stale-cache hazard on default_branch (it has been in ghRepo since #914, so old on-disk caches have it), and the test design (the blueprint fixture proof in TestListingWorks is a nice touch). Unit tests for cmd/labs, cmd/labs/github, and cmd/labs/localcache pass locally, including a -race run of the new test.

This review was written by Isaac, an AI coding agent, with an independent second pass by another model.

asnare

I agree that labs list should only list installable projects, and considered the problem back when I addressed the paging issue.

The problem with testing for labs.yml is the additional REST calls that are required:

Before this PR: 1 request (results cached for 24h)
After this request: 1 + N requests, where N is currently 39.

Although this implementation avoids the 60/IP/hour quota on the REST API by hitting the CDN directly, in terms of light-touch I think I'd prefer to filter projects based on repository "topics". I've tagged a few with databricks-cli-installable, for testing purposes. Filtering on this has a few benefits:

No additional HTTP requests necessary, repository topics are already included in the response we get.
Caching remains simple.
On the labs/maintainer side, things become opt-in. At 8/39 I think opt-in is preferable to opt-out.
On the labs/maintainer side, turning up on labs list becomes an admin operation.

Before reviewing the technical implementation I'd like to get consensus on this.

P.S. I also rejected using the GraphQL API as a solution to detect the presence of labs.yml: calls need to be authenticated, and the quota system would still make it costly.

janniklasrose · 2026-06-16T22:51:29Z

@asnare I like the databricks-cli-installable tag idea, it's simple (the diff now looks much cleaner). Any concern with tagging the other 5 installable repos before we proceed here? Do we need alignment from repo owners, or is the fact that these repos already support labs install enough to warrant the tag?

asnare

This looks fine to me.

@janniklasrose: the installable repositories all have the magic tag now.

## Release v1.6.0 ### CLI * `ssh connect` now accepts a `--base-environment` flag to run a serverless session on a custom base environment. It takes an `env.yaml` path, a `workspace-base-environments/...` resource ID, or a base environment display name, and is rejected together with `--environment-version` or `--cluster` ([#5706](#5706)). * `databricks aitools install` is now plugin-first: it installs the Databricks plugin through each agent's own CLI (Claude Code, Codex, GitHub Copilot) instead of copying raw skill files. Agents without a plugin (OpenCode, Antigravity) still get skill files, and Cursor prints the `/add-plugin databricks` step. Use `--skills-only` to force raw skill files for every agent, or `--path <dir>` to write skills to a directory ([#5738](#5738)). * `databricks labs list` now only shows projects that can be installed ([#5560](#5560)). ### Bundles * direct: Fixed persistent drift on `model_serving_endpoints` caused by the `traffic_config` field ([#5708](#5708)). * direct: Fix spurious update when `apply_policy_default_values: true` is set on job task, for-each-task, or job cluster new_cluster ([#5731](#5731)). Also fix spurious updates for for-each-task clusters due to missing backend defaults for `data_security_mode`, `node_type_id`, `driver_node_type_id`, `driver_instance_pool_id`, `enable_elastic_disk`, and `enable_local_disk_encryption`. * direct: Cluster resize now falls back to regular update if resize fails due to `INVALID_STATE` ([#5716](#5716)). * `bundle generate dashboard` now honors the `--key` flag when naming the generated resource, and rejects combining `--existing-path`, `--existing-id`, and `--resource` instead of silently ignoring all but one ([#5492](#5492)). * Fixed `bundle deployment migrate` failing on `model_serving_endpoints`/`database_instances` with permissions (regression since v1.5.0) ([#5775](#5775)). * After a terraform deploy, the CLI now dry-runs a migration to the direct engine (writing nothing locally or remotely) and reports the outcome via telemetry, warning if the migration could not be completed ([#5797](#5797)). ### Dependency updates * Bump `github.com/databricks/databricks-sdk-go` from v0.147.0 to v0.152.0 ([#5773](#5773)). * Bump Terraform provider from v1.118.0 to v1.120.0 ([#5792](#5792)).

janniklasrose temporarily deployed to test-trigger-is June 11, 2026 18:24 — with GitHub Actions Inactive

Add changelog entry
8ed1722
Co-authored-by: Isaac

janniklasrose temporarily deployed to test-trigger-is June 11, 2026 18:43 — with GitHub Actions Inactive

janniklasrose requested review from pietern and simonfaltum June 11, 2026 18:44

janniklasrose mentioned this pull request Jun 11, 2026
Improve error for labs projects that do not provide labs.yml #5559
Merged

pietern requested a review from asnare June 11, 2026 19:01

simonfaltum requested changes Jun 12, 2026
View reviewed changes

Comment threadcmd/labs/list.go Outdated
Comment threadcmd/labs/list.go Outdated
Comment threadNEXT_CHANGELOG.md Outdated
Comment threadcmd/labs/list_test.go Outdated

asnare requested changes Jun 12, 2026
View reviewed changes

janniklasroseand others added 2 commits June 17, 2026 00:41

Use databricks-cli-installable tag on repos
ffe4dcc

Merge branch 'main' into janniklasrose/labs-list-installable
14ce2cb

janniklasrose temporarily deployed to test-trigger-is June 16, 2026 22:44 — with GitHub Actions Inactive

janniklasrose added the do-not-merge label Jun 16, 2026

asnare approved these changes Jun 22, 2026
View reviewed changes

asnare requested a review from simonfaltum June 22, 2026 08:44

Merge branch 'main' into janniklasrose/labs-list-installable
1ff24cb

janniklasrose removed the do-not-merge label Jun 30, 2026

janniklasrose temporarily deployed to test-trigger-is June 30, 2026 10:13 — with GitHub Actions Inactive

janniklasrose enabled auto-merge June 30, 2026 10:13

simonfaltum approved these changes Jun 30, 2026
View reviewed changes

janniklasrose added this pull request to the merge queue Jun 30, 2026

Merged via the queue into main with commit 53dcecaJun 30, 2026
25 checks passed

janniklasrose deleted the janniklasrose/labs-list-installable branch June 30, 2026 11:25

Navigation Menu

Uh oh!

Only show installable projects in 'databricks labs list' - #5560

Only show installable projects in 'databricks labs list'#5560
janniklasrose merged 5 commits into
mainfrom
janniklasrose/labs-list-installable

janniklasrose commented Jun 11, 2026•
edited
Loading

Uh oh!

eng-dev-ecosystem-bot commented Jun 11, 2026•
edited
Loading

Uh oh!

simonfaltum left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asnare left a comment •
edited
Loading

Uh oh!

janniklasrose commented Jun 16, 2026

Uh oh!

asnare left a comment •
edited
Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

janniklasrose commented Jun 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

eng-dev-ecosystem-bot commented Jun 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

simonfaltum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asnare left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janniklasrose commented Jun 16, 2026

Uh oh!

asnare left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

janniklasrose commented Jun 11, 2026•
edited
Loading

eng-dev-ecosystem-bot commented Jun 11, 2026•
edited
Loading

asnare left a comment •
edited
Loading

asnare left a comment •
edited
Loading