Add an experimental dbt-sql template by lennartkats-db · Pull Request #1059 · databricks/cli

lennartkats-db · 2023-12-12T15:01:58Z

Changes

This adds a new dbt-sql template. This work requires the new WorkspaceFS support for dbt tasks.

In this latest revision, I've hidden the new template from the list so we can merge it, iterate over it, and propertly release the template at the right time.

Blockers:

Tests

## Changes This enhances the template selection experience a bit as we add more and more built-in templates (like #1051 and #1059): ### New experience: <img width="661" alt="image" src="https://github.com/databricks/cli/assets/58432911/afe3b84d-8a77-47f3-b9c2-f827f7893cd7"> ### Current experience: <img width="265" alt="image" src="https://github.com/databricks/cli/assets/58432911/36f8d568-819f-4920-83b1-fb76109ea3d1"> --------- Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>

pietern · 2024-02-05T08:03:40Z

+ },
+ "http_path": {
+ "type": "string",
+ "pattern": "^/sql/.\\../warehouses/[a-z0-9]+$",


How does this pattern work without + or *?

I'm not sure I understand this question. There is a + in there?

Or maybe you're referring to the \.\\.. part? That matches a version, like 1.0.

CLI: * Add support for UC Volumes to the `databricks fs` commands ([#1209](#1209)). Bundles: * Use dynamic configuration model in bundles ([#1098](#1098)). * Allow use of variables references in primitive non-string fields ([#1219](#1219)). * Add an experimental default-sql template ([#1051](#1051)). * Add an experimental dbt-sql template ([#1059](#1059)). Internal: * Add fork-user to winget release workflow ([#1214](#1214)). * Use `any` as type for data sources and resources in `tf/schema` ([#1216](#1216)). * Avoid infinite recursion when normalizing a recursive type ([#1213](#1213)). * Fix issue where interpolating a new ref would rewrite unrelated fields ([#1217](#1217)). * Use `dyn.Value` as input to generating Terraform JSON ([#1218](#1218)). API Changes: * Changed `databricks lakehouse-monitors update` command with new required argument order. * Added `databricks online-tables` command group. OpenAPI commit cdd76a98a4fca7008572b3a94427566dd286c63b (2024-02-19) Dependency updates: * Bump Terraform provider to v1.36.2 ([#1215](#1215)). * Bump github.com/databricks/databricks-sdk-go from 0.32.0 to 0.33.0 ([#1222](#1222)).

## Changes This enhances the template selection experience a bit as we add more and more built-in templates (like #1051 and #1059): ### New experience: <img width="661" alt="image" src="https://github.com/databricks/cli/assets/58432911/afe3b84d-8a77-47f3-b9c2-f827f7893cd7"> ### Current experience: <img width="265" alt="image" src="https://github.com/databricks/cli/assets/58432911/36f8d568-819f-4920-83b1-fb76109ea3d1"> --------- Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>

## Changes This adds a new dbt-sql template. This work requires the new WorkspaceFS support for dbt tasks. In this latest revision, I've hidden the new template from the list so we can merge it, iterate over it, and propertly release the template at the right time. Blockers: - [x] WorkspaceFS support for dbt projects is in prod - [x] Move dbt files into a subdirectory - [ ] Wait until the next (>1.7.4) release of the dbt plugin which will have major improvements! - _Rather than wait, this template is hidden from the list of templates._ - [x] SQL extension is preconfigured based on extension settings (if possible) - MV / streaming tables: - [x] Add to template - [x] Fixdatabricks/dbt-databricks#535 (to be released with in 1.7.4) - [x] Merge databricks/dbt-databricks#338 (to be released with in 1.7.4) - [ ] Fix "too many 503 errors" issue (databricks/dbt-databricks#570, internal tracker: ES-1009215, ES-1014138) - [x] Support ANSI mode in the template - [ ] Streaming tables support is either ungated or the template provides instructions about signup - _Mitigation for now: this template is hidden from the list of templates._ - [x] Support non-workspace-admin deployment - [x] Make sure `data_security_mode: SINGLE_USER` works on non-UC workspaces (it's required to be explicitly specified on UC workspaces with single-node clusters) - [x] Support non-UC workspaces ## Tests - [x] Unit tests - [x] Manual testing - [x] More manual testing - [ ] Reviewer manual testing - _I'd like to do a small bug bash post-merging._ - [x] Unit tests

CLI: * Add support for UC Volumes to the `databricks fs` commands ([#1209](#1209)). Bundles: * Use dynamic configuration model in bundles ([#1098](#1098)). * Allow use of variables references in primitive non-string fields ([#1219](#1219)). * Add an experimental default-sql template ([#1051](#1051)). * Add an experimental dbt-sql template ([#1059](#1059)). Internal: * Add fork-user to winget release workflow ([#1214](#1214)). * Use `any` as type for data sources and resources in `tf/schema` ([#1216](#1216)). * Avoid infinite recursion when normalizing a recursive type ([#1213](#1213)). * Fix issue where interpolating a new ref would rewrite unrelated fields ([#1217](#1217)). * Use `dyn.Value` as input to generating Terraform JSON ([#1218](#1218)). API Changes: * Changed `databricks lakehouse-monitors update` command with new required argument order. * Added `databricks online-tables` command group. OpenAPI commit cdd76a98a4fca7008572b3a94427566dd286c63b (2024-02-19) Dependency updates: * Bump Terraform provider to v1.36.2 ([#1215](#1215)). * Bump github.com/databricks/databricks-sdk-go from 0.32.0 to 0.33.0 ([#1222](#1222)).

Add a dbt template
4fee665

lennartkats-db changed the title ~~[WIP] Add a dbt template~~[WIP] Add a dbt-sql templateDec 12, 2023

lennartkats-db mentioned this pull request Dec 12, 2023
Improve experience for multiple builtin templates #1052
Merged

Use a template for VS Code settings
b3a5ef8

pietern reviewed Dec 15, 2023
View reviewed changes

Comment threadlibs/template/templates/dbt-sql/databricks_template_schema.json
Comment threadlibs/template/templates/dbt-sql/template/{{.project_name}}/profile_template.yml.tmpl Outdated

lennartkats-db added 2 commits December 19, 2023 10:27

Tweak message
c81d139

Update
c900cff

lennartkats-db changed the title ~~[WIP] Add a dbt-sql template~~[WIP] [DONTMERGE] Add a dbt-sql templateDec 23, 2023

Add tests
16f26a3

lennartkats-db added 2 commits December 28, 2023 13:32

Merge remote-tracking branch 'databricks/main' into dbt-template
cd52c83

Fix test
d85d4c4

lennartkats-db marked this pull request as ready for review December 28, 2023 12:34

lennartkats-db changed the title ~~[WIP] [DONTMERGE] Add a dbt-sql template~~[DONTMERGE] Add a dbt-sql templateDec 28, 2023

lennartkats-db requested a review from pietern December 28, 2023 12:35

lennartkats-db added 7 commits January 8, 2024 21:16

Merge remote-tracking branch 'databricks/main' into dbt-template
419bd27

Add template
9030f56

Improve catalog handling
45ea8db

Minor tweaks
0268c88

Update template to use materialized views & streaming tables
94ebd9a

Add conditional
14bc1fa

Improve template
1501298

lennartkats-db force-pushed the dbt-template branch from 62a0b69 to 1501298Compare January 20, 2024 21:59

lennartkats-db added 4 commits January 22, 2024 09:23

Offer an option to use personal schemas
6fc5ed4

Merge remote-tracking branch 'databricks/main' into dbt-template
220a1ea

Fix ANSI mode
99f920e

Merge remote-tracking branch 'databricks/main' into dbt-template
af0dd6d

pietern added the do-not-merge label Jan 25, 2024

lennartkats-db added 2 commits January 25, 2024 21:17

Don't ask for a "production" schema, just assume "default"
1099eed

Explain mode: development
33c5e91

lennartkats-db added 3 commits January 28, 2024 09:11

Remove from list of templates for now
8e7c6a1

Update README.md
18c6b70

Merge remote-tracking branch 'databricks/main' into dbt-template
a660efa

lennartkats-db changed the title ~~[DONTMERGE] [WIP] Add a dbt-sql template~~Add a dbt-sql templateJan 28, 2024

lennartkats-db changed the title ~~Add a dbt-sql template~~Add an experimental dbt-sql templateJan 29, 2024

Mark as experimental
2f52ff1

lennartkats-db removed the do-not-merge label Feb 4, 2024

pietern approved these changes Feb 5, 2024
View reviewed changes

pietern mentioned this pull request Feb 5, 2024
Add an experimental default-sql template #1051
Merged
9 tasks

juliacrawf-db reviewed Feb 13, 2024
View reviewed changes

Comment threadlibs/template/templates/dbt-sql/databricks_template_schema.json

juliacrawf-db reviewed Feb 19, 2024
View reviewed changes

Comment threadlibs/template/templates/dbt-sql/template/{{.project_name}}/README.md.tmpl Outdated

juliacrawf-db reviewed Feb 19, 2024
View reviewed changes

Comment threadlibs/template/templates/dbt-sql/template/{{.project_name}}/README.md.tmpl Outdated

juliacrawf-db reviewed Feb 19, 2024
View reviewed changes

Comment threadlibs/template/templates/dbt-sql/template/{{.project_name}}/README.md.tmpl Outdated

lennartkats-db added 3 commits February 19, 2024 09:56

Restore sql-dbt template in hidden form
e041148

Merge remote-tracking branch 'databricks/main' into dbt-template
00bf2fe

Copy-editing
e5fb708

lennartkats-db force-pushed the dbt-template branch from aaec97d to e5fb708Compare February 19, 2024 09:07

lennartkats-db enabled auto-merge February 19, 2024 09:10

lennartkats-db removed the request for review from PaulCornellDB February 19, 2024 09:12

lennartkats-db added this pull request to the merge queue Feb 19, 2024

Merged via the queue into databricks:main with commit 1c68012Feb 19, 2024

lennartkats-db deleted the dbt-template branch February 19, 2024 09:20

lennartkats-db mentioned this pull request Feb 20, 2024
Add examples based on default-sql & dbt-sql templates databricks/bundle-examples#23
Merged

andrewnester mentioned this pull request Feb 20, 2024
Release v0.214.0 #1226
Merged

Navigation Menu

Uh oh!

Add an experimental dbt-sql template - #1059

Add an experimental dbt-sql template#1059
lennartkats-db merged 29 commits into
databricks:mainfrom
lennartkats-db:dbt-template

lennartkats-db commented Dec 12, 2023•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternFeb 5, 2024

Uh oh!

lennartkats-dbFeb 19, 2024

Uh oh!

lennartkats-dbFeb 19, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

lennartkats-db commented Dec 12, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternFeb 5, 2024

Choose a reason for hiding this comment

Uh oh!

lennartkats-dbFeb 19, 2024

Choose a reason for hiding this comment

Uh oh!

lennartkats-dbFeb 19, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lennartkats-db commented Dec 12, 2023•
edited
Loading