Skip to content

Add Lakeflow template - #2959

Merged
fjakobs merged 13 commits into
mainfrom
lakeflow-template
Jun 2, 2025
Merged

Add Lakeflow template#2959
fjakobs merged 13 commits into
mainfrom
lakeflow-template

Conversation

@fjakobs

@fjakobsfjakobs commented May 30, 2025

Copy link
Copy Markdown
Contributor

Changes

Add Lakeflow template based on the new Pipeline folder structure and leveraging the new glob and root_path properties.

Why

Tests

@fjakobs
fjakobstemporarily deployed to test-trigger-is May 30, 2025 09:14 — with GitHub Actions Inactive
@fjakobs
fjakobs requested review from lennartkats-db and removed request for andrewnester, anton-107 and denikMay 30, 2025 09:14
fjakobs
fjakobs commented May 30, 2025
View reviewed changes
Comment threadlibs/template/template.go Outdated
@fjakobs
fjakobsforce-pushed the lakeflow-template branch from 08a9131 to 5e6be26CompareMay 30, 2025 09:41
@fjakobs
fjakobstemporarily deployed to test-trigger-is May 30, 2025 09:41 — with GitHub Actions Inactive
lennartkats-db
lennartkats-db reviewed May 30, 2025
View reviewed changes

@lennartkats-dblennartkats-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! I added some quick initial comments

Comment threadlibs/template/template.go Outdated
Comment threadlibs/template/template.go Outdated
Comment threadlibs/template/templates/lakeflow/README.md Outdated
Comment threadlibs/template/templates/lakeflow/databricks_template_schema.json Outdated
Comment threadlibs/template/templates/lakeflow/databricks_template_schema.json Outdated
Comment threadlibs/template/templates/lakeflow/databricks_template_schema.json Outdated
@denik

denik commented May 30, 2025

Copy link
Copy Markdown
Contributor

Can you add acceptance test here? https://github.com/databricks/cli/tree/main/acceptance/bundle/templates

For some templates we only do init + validate in local mode, for other templates (default-python) we do init+validate+deploy against local fake server and against cloud env.

We also include materialized template there so you can see what you get.

@fjakobs
fjakobstemporarily deployed to test-trigger-is May 30, 2025 12:14 — with GitHub Actions Inactive
@fjakobs
fjakobstemporarily deployed to test-trigger-is May 30, 2025 13:00 — with GitHub Actions Inactive
@fjakobs
fjakobstemporarily deployed to test-trigger-is May 30, 2025 13:02 — with GitHub Actions Inactive
@fjakobs
fjakobsforce-pushed the lakeflow-template branch from 7f5ec1f to 2bcd816CompareMay 30, 2025 15:00
@fjakobs
fjakobstemporarily deployed to test-trigger-is May 30, 2025 15:00 — with GitHub Actions Inactive
@fjakobs
fjakobstemporarily deployed to test-trigger-is May 30, 2025 15:55 — with GitHub Actions Inactive
lennartkats-db
lennartkats-db requested changes May 30, 2025
View reviewed changes
Comment threadlibs/template/templates/lakeflow-pipelines/library/variables.tmpl Outdated
Comment threadlibs/template/templates/lakeflow-pipelines/template/__preamble.tmpl Outdated
Comment thread.../template/templates/lakeflow-pipelines/template/{{.project_name}}/.vscode/settings.json.tmpl Outdated
Comment thread.../template/templates/lakeflow-pipelines/template/{{.project_name}}/.vscode/settings.json.tmpl
"editor.defaultFormatter": "ms-python.black-formatter",
"editor.formatOnSave": true,
},
}

@lennartkats-dblennartkats-dbMay 30, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add "python.analysis.typeCheckingMode": "basic" here? Or, probably better, revisit that question later?

@fjakobsfjakobsMay 30, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to defer that decision

Comment threadlibs/template/templates/lakeflow-pipelines/template/{{.project_name}}/README.md.tmpl Outdated
Comment threadlibs/template/templates/lakeflow-pipelines/template/{{.project_name}}/databricks.yml.tmpl Outdated
Comment thread...low-pipelines/template/{{.project_name}}/resources/{{.project_name}}_pipeline/README.md.tmpl Outdated
Comment thread...low-pipelines/template/{{.project_name}}/resources/{{.project_name}}_pipeline/README.md.tmpl Outdated
Comment thread...resources/{{.project_name}}_pipeline/transformations/sample_trips_{{.project_name}}.sql.tmpl Outdated
Comment threadlibs/template/templates/lakeflow-pipelines/databricks_template_schema.json Outdated
@fjakobs
fjakobstemporarily deployed to test-trigger-is May 30, 2025 16:36 — with GitHub Actions Inactive
@fjakobs

fjakobs commented May 30, 2025

Copy link
Copy Markdown
ContributorAuthor

@denik I've added the acceptance tests

@fjakobs
fjakobstemporarily deployed to test-trigger-is June 2, 2025 08:08 — with GitHub Actions Inactive
pietern
pietern reviewed Jun 2, 2025
View reviewed changes
Comment threadacceptance/bundle/templates/lakeflow-pipelines/sql/input.json
Comment threadlibs/template/templates/experimental-jobs-as-code/template/__preamble.tmpl
# Preamble

This file only template directives; it is skipped for the actual output.
This file only contains template directives; it is skipped for the actual output.

@pieternpieternJun 2, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you file a separate PR for this?

Comment threadlibs/template/templates/lakeflow-pipelines/template/__preamble.tmpl
{{$isSQL := eq .language "sql"}}

{{if $isSQL}}
{{skip "{{.project_name}}/resources/{{.project_name}}_pipeline/utilities/utils.py"}}

@pieternpieternJun 2, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default pipeline name is clunky. Can this be static instead?

@fjakobsfjakobsJun 2, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a discussion about this last Friday. I'll send you the pointer.

pietern
pietern reviewed Jun 2, 2025
View reviewed changes
Comment thread...e/{{.project_name}}/resources/{{.project_name}}_pipeline/{{.project_name}}.pipeline.yml.tmpl Outdated
Comment thread...e/{{.project_name}}/resources/{{.project_name}}_pipeline/{{.project_name}}.pipeline.yml.tmpl
Comment thread...e/{{.project_name}}/resources/{{.project_name}}_pipeline/{{.project_name}}.pipeline.yml.tmpl Outdated
@fjakobs
fjakobstemporarily deployed to test-trigger-is June 2, 2025 08:32 — with GitHub Actions Inactive
@fjakobs
fjakobstemporarily deployed to test-trigger-is June 2, 2025 08:57 — with GitHub Actions Inactive
@fjakobs
fjakobs requested a review from pieternJune 2, 2025 11:24
@fjakobs
fjakobstemporarily deployed to test-trigger-is June 2, 2025 11:39 — with GitHub Actions Inactive
pietern
pietern reviewed Jun 2, 2025
View reviewed changes
Comment threadacceptance/bundle/templates/lakeflow-pipelines/sql/input.json
Comment thread...low-pipelines/template/{{.project_name}}/resources/{{.project_name}}_pipeline/README.md.tmpl Outdated
@fjakobs
fjakobstemporarily deployed to test-trigger-is June 2, 2025 12:00 — with GitHub Actions Inactive
@fjakobs
fjakobstemporarily deployed to test-trigger-is June 2, 2025 12:33 — with GitHub Actions Inactive
@fjakobs
fjakobstemporarily deployed to test-trigger-is June 2, 2025 13:48 — with GitHub Actions Inactive
@fjakobs
fjakobstemporarily deployed to test-trigger-is June 2, 2025 13:49 — with GitHub Actions Inactive
@fjakobs
fjakobs enabled auto-merge June 2, 2025 13:50
@fjakobs
fjakobs added this pull request to the merge queueJun 2, 2025
Merged via the queue into main with commit 6d5f2a8Jun 2, 2025
@fjakobs
fjakobs deleted the lakeflow-template branch June 2, 2025 14:31
deco-sdk-taggingBot added a commit that referenced this pull request Jun 3, 2025
@deco-sdk-tagging
[Release] Release v0.254.0
70bb289
## Release v0.254.0
### Bundles
* Added `experimental.skip_artifact_cleanup` flag ([#2980](#2980))
* Add an experimental project template for Lakeflow Declarative Pipelines ([#2959](#2959))
denik pushed a commit that referenced this pull request May 20, 2026
@fjakobs
Add Lakeflow template (#2959)
9328a16
## Changes
Add Lakeflow template based on the new Pipeline folder structure and
leveraging the new `glob` and `root_path` properties.
## Why
<!-- Why are these changes needed? Provide the context that the reviewer
might be missing.
For example, were there any decisions behind the change that are not
reflected in the code itself? -->
## Tests
<!-- How have you tested the changes? -->
<!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
denik pushed a commit that referenced this pull request May 20, 2026
@deco-sdk-tagging
[Release] Release v0.254.0
69a7abb
## Release v0.254.0
### Bundles
* Added `experimental.skip_artifact_cleanup` flag ([#2980](#2980))
* Add an experimental project template for Lakeflow Declarative Pipelines ([#2959](#2959))
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pieternpieternpietern approved these changes
@lennartkats-dblennartkats-dblennartkats-db approved these changes
@shreyas-goenkashreyas-goenkaAwaiting requested review from shreyas-goenka

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@fjakobs@denik@pietern@lennartkats-db