Add a foundation for built-in templates by lennartkats-db · Pull Request #685 · databricks/cli

lennartkats-db · 2023-08-21T07:55:40Z

Changes

This pull request extends the templating support in preparation of a new, default template (WIP, #686):

builtin templates that can be initialized using e.g. databricks bundle init default-python
builtin templates are embedded into the executable using go's embed functionality, making sure they're co-versioned with the CLI
new helpers to get the workspace name, current user name, etc. help craft a complete template
(not enabled yet) when the user types databricks bundle init they can interactively select the default-python template

And makes two tangentially related changes:

IsServicePrincipal now uses the "users" API rather than the "principals" API, since the latter is too slow for our purposes.
mode: prod no longer requires the 'target.prod.git' setting. It's hard to set that from a template. (Pieter is planning an overhaul of warnings support; this would be one of the first warnings we show.)

The actual default-python template is maintained in a separate PR: #686

Tests

Unit tests, manual testing

pietern · 2023-08-29T08:26:09Z

+// Determines whether a given user id is a service principal.
+// This function uses a heuristic: if no user exists with this id, we assume
+// it's a service principal. Unfortunately, the standard service principal API is too
+// slow for our purposes.


We should do something else. This can fail for other reasons and if it does will incorrectly assume the user is a service principal. If I look at the scim/me output (that we include in workspace.current_user), there are other heuristics that should be more reliable. Example of workspace.current_user for a service principal:
{ "active":true, "displayName":"sdk-1693297374999987000", "emails": [ { "primary":true, "type":"work", "value":"a1e691b7-a6df-4a6b-881e-d93a03a7ce52" } ], "id":"6079497891687986", "name": { "givenName":"sdk-1693297374999987000" }, "userName":"a1e691b7-a6df-4a6b-881e-d93a03a7ce52" }

Is that always the case though, that the username is an id like that? And will that be the case in the future?

Actually — this should have error handling code. Not sure what happened to it. Let me add it back in a cleanup PR: #707.

CLI: * Fixed --environment flag ([#705](#705)). Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)).

@pietern

## Changes Add some cleanup based on @pietern's comments on #685

Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Fixed --environment flag ([#705](#705)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)). * Test transform when no Python wheel tasks defined ([#714](#714)). * Pin Terraform binary version to 1.5.5 ([#715](#715)). * Cleanup after "Add a foundation for built-in templates" ([#707](#707)). * Filter down to Python wheel tasks only for trampoline ([#712](#712)). * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).

@pietern

Add some cleanup based on @pietern's comments on #685

Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Fixed --environment flag ([#705](#705)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)). * Test transform when no Python wheel tasks defined ([#714](#714)). * Pin Terraform binary version to 1.5.5 ([#715](#715)). * Cleanup after "Add a foundation for built-in templates" ([#707](#707)). * Filter down to Python wheel tasks only for trampoline ([#712](#712)). * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).

## Changes This adds a built-in "default-python" template to the CLI. This is based on the new default-template support of #685. The goal here is to offer an experience where customers can simply type `databricks bundle init` to get a default template: ``` $ databricks bundle init Template to use [default-python]: default-python Unique name for this project [my_project]: my_project ✨ Successfully initialized template ``` The present template: - [x] Works well with VS Code - [x] Works well with the workspace - [x] Works well with DB Connect - [x] Uses minimal stubs rather than boiler-plate-heavy examples I'll have a followup with tests + DLT support. --------- Co-authored-by: Andrew Nester <andrew.nester@databricks.com> Co-authored-by: PaulCornellDB <paul.cornell@databricks.com> Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

## Changes This pull request extends the templating support in preparation of a new, default template (WIP, databricks#686): * builtin templates that can be initialized using e.g. `databricks bundle init default-python` * builtin templates are embedded into the executable using go's `embed` functionality, making sure they're co-versioned with the CLI * new helpers to get the workspace name, current user name, etc. help craft a complete template * (not enabled yet) when the user types `databricks bundle init` they can interactively select the `default-python` template And makes two tangentially related changes: * IsServicePrincipal now uses the "users" API rather than the "principals" API, since the latter is too slow for our purposes. * mode: prod no longer requires the 'target.prod.git' setting. It's hard to set that from a template. (Pieter is planning an overhaul of warnings support; this would be one of the first warnings we show.) The actual `default-python` template is maintained in a separate PR: databricks#686 ## Tests Unit tests, manual testing Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

@pietern

## Changes Add some cleanup based on @pietern's comments on databricks#685 Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

Bundles: * Support cluster overrides with cluster_key and compute_key ([databricks#696](databricks#696)). * Allow referencing local Python wheels without artifacts section defined ([databricks#703](databricks#703)). * Fixed --environment flag ([databricks#705](databricks#705)). * Correctly identify local paths in libraries section ([databricks#702](databricks#702)). * Fixed path joining in FindFilesWithSuffixInPath ([databricks#704](databricks#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([databricks#635](databricks#635)). Internal: * Add a foundation for built-in templates ([databricks#685](databricks#685)). * Test transform when no Python wheel tasks defined ([databricks#714](databricks#714)). * Pin Terraform binary version to 1.5.5 ([databricks#715](databricks#715)). * Cleanup after "Add a foundation for built-in templates" ([databricks#707](databricks#707)). * Filter down to Python wheel tasks only for trampoline ([databricks#712](databricks#712)). * Update Terraform provider schema structs from 1.23.0 ([databricks#713](databricks#713)). Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

## Changes This adds a built-in "default-python" template to the CLI. This is based on the new default-template support of databricks#685. The goal here is to offer an experience where customers can simply type `databricks bundle init` to get a default template: ``` $ databricks bundle init Template to use [default-python]: default-python Unique name for this project [my_project]: my_project ✨ Successfully initialized template ``` The present template: - [x] Works well with VS Code - [x] Works well with the workspace - [x] Works well with DB Connect - [x] Uses minimal stubs rather than boiler-plate-heavy examples I'll have a followup with tests + DLT support. --------- Co-authored-by: Andrew Nester <andrew.nester@databricks.com> Co-authored-by: PaulCornellDB <paul.cornell@databricks.com> Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com> Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

## Changes This pull request extends the templating support in preparation of a new, default template (WIP, #686): * builtin templates that can be initialized using e.g. `databricks bundle init default-python` * builtin templates are embedded into the executable using go's `embed` functionality, making sure they're co-versioned with the CLI * new helpers to get the workspace name, current user name, etc. help craft a complete template * (not enabled yet) when the user types `databricks bundle init` they can interactively select the `default-python` template And makes two tangentially related changes: * IsServicePrincipal now uses the "users" API rather than the "principals" API, since the latter is too slow for our purposes. * mode: prod no longer requires the 'target.prod.git' setting. It's hard to set that from a template. (Pieter is planning an overhaul of warnings support; this would be one of the first warnings we show.) The actual `default-python` template is maintained in a separate PR: #686 ## Tests Unit tests, manual testing

@pietern

## Changes Add some cleanup based on @pietern's comments on #685

Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Fixed --environment flag ([#705](#705)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)). * Test transform when no Python wheel tasks defined ([#714](#714)). * Pin Terraform binary version to 1.5.5 ([#715](#715)). * Cleanup after "Add a foundation for built-in templates" ([#707](#707)). * Filter down to Python wheel tasks only for trampoline ([#712](#712)). * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).

## Changes This adds a built-in "default-python" template to the CLI. This is based on the new default-template support of #685. The goal here is to offer an experience where customers can simply type `databricks bundle init` to get a default template: ``` $ databricks bundle init Template to use [default-python]: default-python Unique name for this project [my_project]: my_project ✨ Successfully initialized template ``` The present template: - [x] Works well with VS Code - [x] Works well with the workspace - [x] Works well with DB Connect - [x] Uses minimal stubs rather than boiler-plate-heavy examples I'll have a followup with tests + DLT support. --------- Co-authored-by: Andrew Nester <andrew.nester@databricks.com> Co-authored-by: PaulCornellDB <paul.cornell@databricks.com> Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

lennartkats-db added 4 commits August 21, 2023 09:50

Add foundation for builtin templates
59c1c73

Avoid nil context
2b67262

Add missing file
737c612

Fix test in Windows
2955e84

lennartkats-db mentioned this pull request Aug 21, 2023
databricks bundle init template v1 #686
Merged
4 tasks

lennartkats-db added 3 commits August 21, 2023 13:45

Fix typos
6ae0279

Remove the hard requirement of a target.prod.git setting
568afe0

Merge remote-tracking branch 'databricks/main' into template-foundation
37de1e7

lennartkats-db changed the title ~~Add foundation for a builtin templates~~Add foundation for a built-in templatesAug 21, 2023

Remove old test
a7c6928

lennartkats-db requested a review from shreyas-goenka August 21, 2023 11:52

Fix typo
9435670

lennartkats-db requested a review from andrewnester August 22, 2023 15:44

lennartkats-db marked this pull request as ready for review August 22, 2023 15:45

lennartkats-db changed the title ~~Add foundation for a built-in templates~~Add foundation for built-in templatesAug 22, 2023

lennartkats-db changed the title ~~Add foundation for built-in templates~~Add a foundation for built-in templatesAug 22, 2023

andrewnester reviewed Aug 23, 2023
View reviewed changes

lennartkats-db added 2 commits August 23, 2023 22:04

Cleanup
cfa8a83

Add caching to is_service_principal
4340356

andrewnester approved these changes Aug 24, 2023
View reviewed changes

lennartkats-db added 2 commits August 24, 2023 14:03

Fix tests
9d41ccc

Improve error message
33d8f38

lennartkats-db added this pull request to the merge queue Aug 25, 2023

Merged via the queue into main with commit a5b8609Aug 25, 2023

lennartkats-db deleted the template-foundation branch August 25, 2023 09:10

This was referenced Aug 28, 2023
databricks bundle init template v2: optional stubs, DLT support #700
Merged
Migrate Mlops Stacks to use databricks asset templates for project creation databricks/mlops-stacks#96
Merged

pietern reviewed Aug 29, 2023
View reviewed changes

lennartkats-db mentioned this pull request Aug 29, 2023
Cleanup after "Add a foundation for built-in templates" #707
Merged

andrewnester mentioned this pull request Aug 30, 2023
Release v0.203.3 #711
Closed

github-merge-queueBot pushed a commit that referenced this pull request Aug 30, 2023
Cleanup after "Add a foundation for built-in templates" (#707)
707fd6f
## Changes Add some cleanup based on @pietern's comments on #685

andrewnester mentioned this pull request Aug 30, 2023
Release v0.203.3 #716
Merged

lennartkats-db added a commit that referenced this pull request Sep 1, 2023
Cleanup after "Add a foundation for built-in templates" (#707)
c44e15f
Add some cleanup based on @pietern's comments on #685

denik pushed a commit that referenced this pull request May 20, 2026
Cleanup after "Add a foundation for built-in templates" (#707)
3d6e49a
## Changes Add some cleanup based on @pietern's comments on #685

Navigation Menu

Uh oh!

Add a foundation for built-in templates - #685

Add a foundation for built-in templates#685
lennartkats-db merged 13 commits into
mainfrom
template-foundation

lennartkats-db commented Aug 21, 2023•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternAug 29, 2023

Uh oh!

lennartkats-dbAug 29, 2023

Uh oh!

lennartkats-dbAug 29, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

lennartkats-db commented Aug 21, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternAug 29, 2023

Choose a reason for hiding this comment

Uh oh!

lennartkats-dbAug 29, 2023

Choose a reason for hiding this comment

Uh oh!

lennartkats-dbAug 29, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lennartkats-db commented Aug 21, 2023•
edited
Loading