Support serverless mode in default-python template by denik · Pull Request #2348 · databricks/cli

denik · 2025-02-12T21:48:00Z

Add 'serverless' setting to default-python template. Default is currently set "no" and the setting is not prompted for.

If set to "auto", it will call /settings-api/ to detect serverless support. It falls back to classic in case of errors on that API.

Depends on #2352.

Tests

Split default-python into two tests default-python/classic and default-python/serverless.
Add acceptance tests for 'auto' mode: returning yes/no/error.
Manually check that "bundle init default-python" does not prompt for serverless.
Manually check that "bundle init default-python" with serverless=auto can be deployed and run on dogfood.

This is needed for b.WorkspaceClient().CurrentWorkspaceID(ctx) which is used by initialize_urls.go mutator ("bundle summary") (#2316). It also also needed for to call serverless endpoint (#2348). Builds on top of #2338

This is needed for b.WorkspaceClient().CurrentWorkspaceID(ctx) which is used by initialize_urls.go mutator ("bundle summary") #2316 It also also needed for to call serverless detection endpoint #2348 Builds on top of #2338

diff.py is like "diff -r -U2" but it applies replacements first to the argument. This allows comparing different output files and directories but ignore differences that are going to be replaced by placeholders. This is useful for tests that record large amount of files, specifically "bundle init" with standard templates. In those tests, changing one parameter results in a small diff so recording the full directory is not helpful, because it's hard to see what changed there. I'm using it in implementation of serverless mode for templates that need it: #2348 Related small changes: add [TESTROOT] replacement for absolute path to acceptance directory in git repo. Add $TESTDIR env var for absolute path to a given test in git repo.

## Changes diff.py is like "diff -r -U2" but it applies replacements first to the argument. This allows comparing different output files and directories but ignore differences that are going to be replaced by placeholders. This is useful for tests that record large amount of files, specifically "bundle init" with standard templates. In those tests, changing one parameter results in a small diff so recording the full directory is not helpful, because it's hard to see what changed there. I'm using it in implementation of serverless mode for templates that need it: #2348 The serverless templates are slightly different from classic, capturing the diff helps to see exactly where. Related small changes: - Add [TESTROOT] replacement for absolute path to acceptance directory in git repo. - Add $TESTDIR env var for absolute path to a given test in git repo. ## Tests - New test acceptance/selftest/diff to test the helper. - Via #2348 which makes use of this feature.

It's not interesting since it just dumps what is in the repo. This is especially annoying with bundle/templates tests with a lot of files.

… Windows

denik · 2025-02-25T15:22:50Z

We should rely on the serverless_jobs_ws_nb_enable settings name, see the slack thread I sent you in a DM

After offline discussion, these APIs may have edge cases so for now the safe thing to do is explicit prompt: #2377

…2377) ## Changes - Add 'serverless' prompt to default-python template (default is currently set to "no"). - This is a simplified version of #2348 with 'auto' functionality removed. ## Tests - Split default-python into default-python/classic, default-python/serverless, default-python/serverless-customcatalog. - Manually check that "bundle init default-python" with serverless=yes can be deployed and run on dogfood and test env.

…atabricks#2377) ## Changes - Add 'serverless' prompt to default-python template (default is currently set to "no"). - This is a simplified version of databricks#2348 with 'auto' functionality removed. ## Tests - Split default-python into default-python/classic, default-python/serverless, default-python/serverless-customcatalog. - Manually check that "bundle init default-python" with serverless=yes can be deployed and run on dogfood and test env.

This is needed for b.WorkspaceClient().CurrentWorkspaceID(ctx) which is used by initialize_urls.go mutator ("bundle summary") #2316 It also also needed for to call serverless detection endpoint #2348 Builds on top of #2338

## Changes diff.py is like "diff -r -U2" but it applies replacements first to the argument. This allows comparing different output files and directories but ignore differences that are going to be replaced by placeholders. This is useful for tests that record large amount of files, specifically "bundle init" with standard templates. In those tests, changing one parameter results in a small diff so recording the full directory is not helpful, because it's hard to see what changed there. I'm using it in implementation of serverless mode for templates that need it: #2348 The serverless templates are slightly different from classic, capturing the diff helps to see exactly where. Related small changes: - Add [TESTROOT] replacement for absolute path to acceptance directory in git repo. - Add $TESTDIR env var for absolute path to a given test in git repo. ## Tests - New test acceptance/selftest/diff to test the helper. - Via #2348 which makes use of this feature.

…2377) ## Changes - Add 'serverless' prompt to default-python template (default is currently set to "no"). - This is a simplified version of #2348 with 'auto' functionality removed. ## Tests - Split default-python into default-python/classic, default-python/serverless, default-python/serverless-customcatalog. - Manually check that "bundle init default-python" with serverless=yes can be deployed and run on dogfood and test env.

denik temporarily deployed to test-trigger-is February 12, 2025 21:48 — with GitHub Actions Inactive

denik mentioned this pull request Feb 12, 2025
acc: Set X-Databricks-Org-Id on scim/v2/Me endpoint #2349
Merged

denik force-pushed the denik/serverless-template branch from 7a261c9 to 5ac7251Compare February 12, 2025 21:53

denik temporarily deployed to test-trigger-is February 12, 2025 21:53 — with GitHub Actions Inactive

denik changed the base branch from main to denik/acc-x-databricks-orgFebruary 12, 2025 21:53

denik temporarily deployed to test-trigger-is February 12, 2025 21:58 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 12, 2025 22:08 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 12, 2025 22:10 — with GitHub Actions Inactive

Base automatically changed from denik/acc-x-databricks-org to mainFebruary 13, 2025 07:31

denik mentioned this pull request Feb 13, 2025
acc: add a helper to diff with replacements #2352
Merged

denik force-pushed the denik/serverless-template branch from fda87dd to 7d35ea0Compare February 13, 2025 09:27

denik temporarily deployed to test-trigger-is February 13, 2025 09:27 — with GitHub Actions Inactive

denik changed the base branch from main to denik/acc-diffFebruary 13, 2025 09:29

denik force-pushed the denik/serverless-template branch from 7d35ea0 to d78d857Compare February 13, 2025 10:09

denik temporarily deployed to test-trigger-is February 13, 2025 10:09 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 13, 2025 10:23 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 13, 2025 10:28 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 13, 2025 12:23 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 14, 2025 09:38 — with GitHub Actions Inactive

denik force-pushed the denik/serverless-template branch from 7c272e9 to ca39ceaCompare February 14, 2025 10:16

denik temporarily deployed to test-trigger-is February 14, 2025 10:16 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 14, 2025 10:19 — with GitHub Actions Inactive

denik force-pushed the denik/serverless-template branch from 6a3a987 to 5a5b59cCompare February 14, 2025 10:47

denik temporarily deployed to test-trigger-is February 14, 2025 10:47 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 14, 2025 11:03 — with GitHub Actions Inactive

Base automatically changed from denik/acc-diff to mainFebruary 14, 2025 11:08

denik temporarily deployed to test-trigger-is February 24, 2025 09:23 — with GitHub Actions Inactive

denik temporarily deployed to test-trigger-is February 24, 2025 09:39 — with GitHub Actions Inactive

denik added 17 commits February 24, 2025 12:36

Support serverless in default-python
dcee3db
It's not interesting since it just dumps what is in the repo. This is especially annoying with bundle/templates tests with a lot of files.

tweak whitespace
6530657

update default-python-errorserverless
a9c6716

update with regard to libraries
44bdd35

reorg tests
2d65e71

change default to no
94b4700

use_serverless --> serverless
97caa1c

Fix tests; record classic output fully
734021c

add README
e88ee45

rebase & get rid of SaveRepls option
41ffe48

clean up
dca31a7

do not add 'trace' in front of diff.py to avoid slashes difference on…
406cd3f
… Windows

More diagnostics
5d29d84

add 'serverless: true' to pipeline
71b6394

adjust whitespace
654832e

rebase and update acceptance tests
06f0c52

update tests
5706d1e

denik force-pushed the denik/serverless-template branch from cdd4311 to 5706d1eCompare February 24, 2025 11:37

denik temporarily deployed to test-trigger-is February 24, 2025 11:38 — with GitHub Actions Inactive

denik mentioned this pull request Feb 25, 2025
Support serverless mode in default-python template (explicit prompt) #2377
Merged

denik closed this Feb 25, 2025

pietern deleted the denik/serverless-template branch March 3, 2025 09:53

Navigation Menu

Uh oh!

Support serverless mode in default-python template - #2348

Support serverless mode in default-python template#2348
denik wants to merge 17 commits into
mainfrom
denik/serverless-template

denik commented Feb 12, 2025•
edited
Loading

Uh oh!

denik commented Feb 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

denik commented Feb 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

Uh oh!

denik commented Feb 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

denik commented Feb 12, 2025•
edited
Loading