Skip to content

Update the lakeflow-pipelines template according to the latest Lakeflow conventions - #3558

Merged
lennartkats-db merged 10 commits into
mainfrom
lakeflow-default-python-v0
Sep 23, 2025
Merged

Update the lakeflow-pipelines template according to the latest Lakeflow conventions#3558
lennartkats-db merged 10 commits into
mainfrom
lakeflow-default-python-v0

Conversation

@lennartkats-db

@lennartkats-dblennartkats-db commented Sep 8, 2025

Copy link
Copy Markdown
Contributor

This adds a new version of the default-python template. To enable the first integration and evaluation of this template, this first pull request replaces the internal lakeflow-pipelines template. Replacing default-python will be the next milestone.

Changes

  • Update default-python to follow the Lakeflow conventions (per-pipeline sources go into resources/; shared sources go into (lib/)
  • Adapt this template so it can first be used in place of the lakeflow-pipelines template (this required disabling some of the template questions and components)

Why

Lakeflow comes with new conventions that emphasize modularity. This PR represents an initial milestone towards updating the DABs default-python template. default-sql will follow in a later milestone.

Tests

Existing tests were updated.

@lennartkats-dblennartkats-db changed the title Add a Lakeflow-friendly default-python templateAdd a Lakeflow-friendly DABs templateSep 8, 2025
lennartkats-db
lennartkats-db commented Sep 8, 2025
View reviewed changes
Comment threadlibs/template/templates/lakeflow-pipelines/databricks_template_schema.json
Comment threadlibs/template/templates/lakeflow-pipelines/library/versions.tmpl
Comment threadlibs/template/templates/lakeflow-pipelines/library/variables.tmpl
Comment thread...s/lakeflow-pipelines/template/{{.project_name}}/resources/sample_job/sample_job.job.yml.tmpl
Comment threadlibs/template/templates/lakeflow-pipelines/template/{{.project_name}}/tests/conftest.py
Comment threadlibs/template/templates/lakeflow-pipelines/template/{{.project_name}}/tests/conftest.py
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Sep 8, 2025•
edited
Loading

Copy link
Copy Markdown
Collaborator

Run: 17893981483

Env✅​pass🔄​flaky🙈​skip
✅​aws linux311528
✅​aws windows312527
🔄​aws-ucws linux4149426
🔄​aws-ucws windows4159425
✅​azure linux311527
✅​azure windows312526
🔄​azure-ucws linux4194425
🔄​azure-ucws windows4168424
✅​gcp linux310529
✅​gcp windows311528
10 failing tests:
Test Nameaws-ucws linuxaws-ucws windowsazure-ucws linuxazure-ucws windows
TestAccept✅​pass✅​pass🔄​flaky✅​pass
TestAccept/bundle/deploy/dashboard/detect-change🔄​flaky🔄​flaky✅​pass🔄​flaky
TestAccept/bundle/deploy/dashboard/generate_inplace🔄​flaky🔄​flaky✅​pass🔄​flaky
TestAccept/bundle/deploy/dashboard/nested-folders🔄​flaky🔄​flaky✅​pass🔄​flaky
TestAccept/bundle/deploy/dashboard/simple🔄​flaky🔄​flaky✅​pass🔄​flaky
TestAccept/bundle/deploy/dashboard/simple_outside_bundle_root🔄​flaky🔄​flaky✅​pass🔄​flaky
TestAccept/bundle/deploy/dashboard/simple_syncroot🔄​flaky🔄​flaky✅​pass🔄​flaky
TestAccept/bundle/deployment/bind/dashboard🔄​flaky🔄​flaky🔄​flaky🔄​flaky
TestAccept/bundle/deployment/bind/dashboard/recreation🔄​flaky🔄​flaky🔄​flaky✅​pass
TestDashboardAssumptions_WorkspaceImport🔄​flaky🔄​flaky🔄​flaky🔄​flaky

fjakobs
fjakobs reviewed Sep 8, 2025
View reviewed changes
Comment thread...undle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/.vscode/settings.json Outdated
Comment thread...t/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_etl.pipeline.yml Outdated
Comment thread...output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_job.job.yml
Comment thread...lines/resources/lakeflow_pipelines_etl/transformations/sample_trips_my_lakeflow_pipelines.py Outdated
Comment threadlibs/template/templates/lakeflow-pipelines/databricks_template_schema.json
Comment threadlibs/template/templates/lakeflow-pipelines/databricks_template_schema.vnext.json
pietern
pietern reviewed Sep 8, 2025
View reviewed changes

@pieternpietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't test this end-to-end yet because of the missing pyproject.toml.

Comment threadlibs/template/templates/lakeflow-pipelines/template/__preamble.tmpl
Comment threadlibs/template/templates/lakeflow-pipelines/databricks_template_schema.vnext.json
Comment threadlibs/template/templates/lakeflow-pipelines/databricks_template_schema.json
Comment thread...undle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/.vscode/settings.json
Comment thread...ines/python/output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/utilities/utils.py Outdated
@lennartkats-db
Apply repo formatting
d3acf61
Though this is unfortunately inconsistent with the one from Black
@lennartkats-dblennartkats-db mentioned this pull request Sep 15, 2025
Merged
@lennartkats-dblennartkats-db changed the title Add a Lakeflow-friendly DABs templateUpdate the lakeflow-pipelines template according to the latest Lakeflow conventionsSep 16, 2025
fjakobs
fjakobs approved these changes Sep 22, 2025
View reviewed changes

@fjakobsfjakobs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK with the proposed changes but let's make sure to address @andersrexdb 's comment.

@lennartkats-db
lennartkats-db added this pull request to the merge queueSep 23, 2025
Merged via the queue into main with commit a6398a3Sep 23, 2025
13 checks passed
@lennartkats-db
lennartkats-db deleted the lakeflow-default-python-v0 branch September 23, 2025 09:11
denik pushed a commit that referenced this pull request May 20, 2026
@lennartkats-db
Update the lakeflow-pipelines template according to the latest Lakefl…
dfb65d5
…ow conventions (#3558)
This adds a new version of the `default-python` template. To enable the
first integration and evaluation of this template, this first pull
request replaces the internal `lakeflow-pipelines` template. Replacing
`default-python` will be the next milestone.
## Changes
* Update `default-python` to follow the Lakeflow conventions
(per-pipeline sources go into `resources/`; shared sources go into
(`lib/`)
* Adapt this template so it can first be used in place of the
`lakeflow-pipelines` template (this required disabling some of the
template questions and components)
## Why
Lakeflow comes with new conventions that emphasize modularity. This PR
represents an initial milestone towards updating the DABs
`default-python` template. `default-sql` will follow in a later
milestone.
## Tests
Existing tests were updated.
<!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andersrexdbandersrexdbandersrexdb left review comments
@fjakobsfjakobsfjakobs approved these changes
@andrewnesterandrewnesterAwaiting requested review from andrewnester
@shreyas-goenkashreyas-goenkaAwaiting requested review from shreyas-goenka
@denikdenikAwaiting requested review from denik
@anton-107anton-107Awaiting requested review from anton-107
@pieternpieternAwaiting requested review from pietern
@kanterovkanterovAwaiting requested review from kanterov
@jefferycheng1jefferycheng1Awaiting requested review from jefferycheng1

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@lennartkats-db@eng-dev-ecosystem-bot@pietern@fjakobs@andrewnester@andersrexdb