Skip to content

Update templates to use serverless environment version 4 and matching Python version#3897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lennartkats-db merged 9 commits into from
Nov 14, 2025
Merged

Update templates to use serverless environment version 4 and matching Python version #3897

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
82f49e7
Update serverless environment version to 4 in templates
lennartkats-db Nov 7, 2025
7d245e9
Update bundle templates to serverless environment version 4 and Pytho…
lennartkats-db Nov 8, 2025
6e67551
Cleanup
lennartkats-db Nov 10, 2025
c5f1db8
Remove experimental-jobs-as-code Python version changes
lennartkats-db Nov 10, 2025
93dd6d5
Add NEXT_CHANGELOG entry for serverless environment version update
lennartkats-db Nov 10, 2025
bd3ead2
Add rationale for Python version spec in template
lennartkats-db Nov 10, 2025
c54d302
Merge branch 'main' into update-environment-version
lennartkats-db Nov 13, 2025
afcd213
Update NEXT_CHANGELOG.md
lennartkats-db Nov 13, 2025
f6ba2b8
Merge main into update-environment-version
lennartkats-db Nov 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions NEXT_CHANGELOG.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -9,6 +9,7 @@
### Dependency updates

### Bundles
* Update templates to use serverless environment version 4 and matching Python version ([#3897](https://github.com/databricks/cli/pull/3897))
* Add a language prompt to the `default-minimal` template ([#3918](https://github.com/databricks/cli/pull/3918))
* Add `default-scala` template for Scala projects with SBT build configuration and example code ([#3906](https://github.com/databricks/cli/pull/3906))

Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/dbt-sql/output/my_dbt_sql/resources/my_dbt_sql.job.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -29,6 +29,6 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "2"
environment_version: "4"
dependencies:
- dbt-databricks>=1.8.0,<2.0.0
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/classic/out.compare-vs-serverless.diff
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@
- environments:
- - environment_key: default
- spec:
- environment_version: "2"
- environment_version: "4"
- dependencies:
- # By default we just include the .whl file generated for the my_default_python package.
- # See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/default-python/classic/output/my_default_python/pyproject.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,7 +2,7 @@
name = "my_default_python"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<=3.13"
requires-python = ">=3.10,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand Down
4 changes: 2 additions & 2 deletions ...undle/templates/default-python/classic/output/my_default_python/src/sample_notebook.ipynb
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -72,7 +72,7 @@
"dependencies": [
"--editable .."
],
"environment_version": "2"
"environment_version": "4"
},
"language": "python",
"notebookMetadata": {
Expand DownExpand Up@@ -141,7 +141,7 @@
},
"language_info": {
"name": "python",
"version": "3.11.4"
"version": "3.12"
}
},
"nbformat": 4,
Expand Down
2 changes: 1 addition & 1 deletion ...ptance/bundle/templates/default-python/serverless/output/my_default_python/pyproject.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,7 +2,7 @@
name = "my_default_python"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<=3.13"
requires-python = ">=3.10,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand Down
2 changes: 1 addition & 1 deletion ...templates/default-python/serverless/output/my_default_python/resources/sample_job.job.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -46,7 +46,7 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "2"
environment_version: "4"
dependencies:
# By default we just include the .whl file generated for the my_default_python package.
# See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
Expand Down
4 changes: 2 additions & 2 deletions ...le/templates/default-python/serverless/output/my_default_python/src/sample_notebook.ipynb
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -72,7 +72,7 @@
"dependencies": [
"--editable .."
],
"environment_version": "2"
"environment_version": "4"
},
"language": "python",
"notebookMetadata": {
Expand DownExpand Up@@ -141,7 +141,7 @@
},
"language_info": {
"name": "python",
"version": "3.11.4"
"version": "3.12"
}
},
"nbformat": 4,
Expand Down
2 changes: 1 addition & 1 deletion ...ce/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/pyproject.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,7 +2,7 @@
name = "my_lakeflow_pipelines"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<=3.13"
requires-python = ">=3.10,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand Down
2 changes: 1 addition & 1 deletion ...lates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/sample_job.job.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -29,4 +29,4 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "2"
environment_version: "4"
2 changes: 1 addition & 1 deletion ...emplates/lakeflow-pipelines/sql/output/my_lakeflow_pipelines/resources/sample_job.job.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -29,4 +29,4 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "2"
environment_version: "4"
2 changes: 1 addition & 1 deletion acceptance/bundle/templates/pydabs/init-classic/output/my_pydabs/pyproject.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,7 +2,7 @@
name = "my_pydabs"
version = "0.0.1"
authors = [{ name = "[USERNAME]" }]
requires-python = ">=3.10,<=3.13"
requires-python = ">=3.10,<3.13"
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand Down
7 changes: 7 additions & 0 deletions libs/template/templates/dbt-sql/library/versions.tmpl
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -5,3 +5,10 @@
{{define "latest_lts_db_connect_version_spec" -}}
>=15.4,<15.5
{{- end}}

{{/* The serverless environment version to use.
* See https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/
*/}}
{{define "serverless_environment_version" -}}
4
{{- end}}
2 changes: 1 addition & 1 deletion ...ate/templates/dbt-sql/template/{{.project_name}}/resources/{{.project_name}}.job.yml.tmpl
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "2"
environment_version: "{{template "serverless_environment_version"}}"
dependencies:
- dbt-databricks>=1.8.0,<2.0.0
{{- end }}
32 changes: 28 additions & 4 deletions libs/template/templates/default/library/versions.tmpl
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
{{/* The latest LTS DBR version; this should be updated a few months after each LTS.
* See https://docs.databricks.com/release-notes/runtime/index.html for available versions.
*/}}
{{define "latest_lts_dbr_version" -}}
16.4.x-scala2.12
Expand All@@ -11,16 +12,39 @@
* only connect to compute of that same version and higher. If the version is
* deemed too old, customers can update the version themselves after initializing
* the template.
*
* See https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements
* for DB Connect release notes and version compatibility.
Comment thread
lennartkats-db marked this conversation as resolved.
*/}}
{{define "conservative_db_connect_version_spec" -}}
>=15.4,<15.5
{{- end}}

{{/* DB Connect 15 only supports versions up to 3.13 because of a limitation in
* pyarrow: https://arrow.apache.org/docs/python/install.html#python-compatibility
{{/* The serverless environment version to use.
* See https://docs.databricks.com/release-notes/runtime/index.html for available versions.
*/}}
{{define "serverless_environment_version" -}}
4
{{- end}}

{{/* Python version specification for the template.
* This needs to be compatible with the version of DB Connect and the serverless_environment_version.
*
* Serverless environment version 4 (and upcoming v5), DBR 16, and DBR 17 all use Python 3.12.
* DB Connect 15 supports all of these and is compatible with Python 3.10-3.12.
*
* We use >=3.10,<3.13 rather than pinning to 3.12 specifically because overly narrow pinning
* means users always need to use uv or similar tools to install the exact version. A broader
* range improves usability while maintaining compatibility with both DB Connect and serverless.
*/}}
{{define "python_version_spec" -}}
>=3.10,<3.13
Comment thread
lennartkats-db marked this conversation as resolved.
{{- end}}

{{/* The default Python version used in notebook metadata.
*/}}
{{define "conservative_db_connect_python_version_spec" -}}
>=3.10,<=3.13
{{define "default_python_version" -}}
3.12
{{- end}}

{{define "latest_databricks_bundles_version" -}}0.277.0{{- end}}
2 changes: 1 addition & 1 deletion libs/template/templates/default/template/{{.project_name}}/pyproject.toml.tmpl
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,7 +2,7 @@
name = "{{.project_name}}"
version = "0.0.1"
authors = [{ name = "{{user_name}}" }]
requires-python = "{{template "conservative_db_connect_python_version_spec"}}"
requires-python = "{{template "python_version_spec"}}"
Comment thread
lennartkats-db marked this conversation as resolved.
dependencies = [
# Any dependencies for jobs and pipelines in this project can be added here
# See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
Expand Down
2 changes: 1 addition & 1 deletion libs/template/templates/default/template/{{.project_name}}/resources/sample_job.job.yml.tmpl
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -94,7 +94,7 @@ resources:
environments:
- environment_key: default
spec:
environment_version: "2"
environment_version: "{{template "serverless_environment_version"}}"
{{- if $python_package}}
dependencies:
# By default we just include the .whl file generated for the {{.project_name}} package.
Expand Down
2 changes: 1 addition & 1 deletion libs/template/templates/default/template/{{.project_name}}/resources/sample_job.py.tmpl
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -116,7 +116,7 @@ sample_job = Job.from_dict(
{
"environment_key": "default",
"spec": {
"environment_version": "2",
"environment_version": "{{template "serverless_environment_version"}}",
{{- if $python_package}}
"dependencies": [
# By default we just include the .whl file generated for the {{.project_name}} package.
Expand Down
4 changes: 2 additions & 2 deletions libs/template/templates/default/template/{{.project_name}}/src/sample_notebook.ipynb.tmpl
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -77,7 +77,7 @@
{{- /* Make sure that this file always gets any library dependencies (when using serverless)*/}}
"--editable .."
],
"environment_version": "2"
"environment_version": "{{template "serverless_environment_version"}}"
},
"language": "python",
"notebookMetadata": {
Expand DownExpand Up@@ -146,7 +146,7 @@
},
"language_info": {
"name": "python",
"version": "3.11.4"
"version": "{{template "default_python_version"}}"
}
},
"nbformat": 4,
Expand Down