Skip to content

Update templates to use serverless environment version 4 and matching Python version - #3897

Merged
lennartkats-db merged 9 commits into
mainfrom
update-environment-version
Nov 14, 2025
Merged

Update templates to use serverless environment version 4 and matching Python version#3897
lennartkats-db merged 9 commits into
mainfrom
update-environment-version

Conversation

@lennartkats-db

Copy link
Copy Markdown
Contributor

Changes

This updates templates to serverless environment version 4 and associated Python versions

Some considerations:

  • Client version 4, client version 5 (upcoming), DBR 16, DBR 17 all use Python 3.12
  • DB Connect 15 supports all DBR versions above and supports Python 3.12
  • DB Connect 15 also supports DBR 15, which uses Python 3.11 at runtime, but that discrepancy already exists and seems okay
  • Pinning Python to a very narrow range in pyproject.toml means users always need to use uv to install the right version, which may not be great for usability. so I haven't change that approach either

Why

  • Environment version 4 provides better startup performance and new features

Tests

  • Standard acceptance testing

lennartkats-dband others added 4 commits November 7, 2025 08:58
Remove the Python version spec changes from experimental-jobs-as-code template
to minimize the diff. This template doesn't need the same Python version
constraints as the default template.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 10, 2025

Copy link
Copy Markdown
Collaborator

Run: 19373470510

Env❌​FAIL🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux7236061320:58
🟨​aws windows72235961221:58
🔄​aws-ucws linux36249650323:35
🔄​aws-ucws windows67249350222:22
🔄​azure linux111434961221:43
🔄​azure windows15434761120:46
🔄​azure-ucws linux161447850228:53
❌​azure-ucws windows291446650125:27
🔄​gcp linux4435361522:18
🔄​gcp windows21435561420:05
82 failing tests:
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
TestAccept🟨​K🟨​K🔄​f💚​R💚​R🔄​f💚​R🟨​K🔄​f💚​R
TestAccept/bundle/deploy/files/no-snapshot-sync✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/deployment/unbind/permissions✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/resources/clusters/deploy/simple✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/resources/clusters/deploy/update-after-create✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/resources/dashboards/simple_syncroot✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestAccept/bundle/resources/dashboards/simple_syncroot/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestAccept/bundle/resources/jobs/check-metadata✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/resources/jobs/check-metadata/DATABRICKS_BUNDLE_ENGINE=terraform✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/resources/model_serving_endpoints/basic🙈​s🙈​s✅​p✅​p🙈​s🙈​s🔄​f✅​p🙈​s🙈​s
TestAccept/bundle/resources/model_serving_endpoints/basic/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p🔄​f✅​p
TestAccept/bundle/resources/models/basic✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p
TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=direct✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f
TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=terraform✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p🔄​f
TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
TestAccept/bundle/resources/secret_scopes✅​p✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p
TestAccept/bundle/run/app-with-job🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
TestAccept/bundle/templates/default-python/combinations/classic✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=yes✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
TestAccept/bundle/templates/default-python/integration_classic✅​p✅​p🔄​f🔄​f🔄​f✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.12✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p✅​p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9✅​p✅​p✅​p🔄​f🔄​f✅​p✅​p✅​p✅​p✅​p
TestBundleInitHelpers✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestGenerateFromExistingPipelineAndDeploy✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFsCatOnADir✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFsCatOnADir/uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s✅​p❌​F🙈​s🙈​s
TestFsCatOnNonExistentFile✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFsCatOnNonExistentFile/uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s✅​p❌​F🙈​s🙈​s
TestFsCpDir✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFsCpDir/dbfs_to_dbfs✅​p✅​p✅​p✅​p✅​p🔄​f✅​p❌​F✅​p✅​p
TestFsCpDir/dbfs_to_local✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
TestFsCpDir/uc-volumes_to_uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s✅​p❌​F🙈​s🙈​s
TestFsCpDirToDirWithOverwriteFlag/dbfs_to_local✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p
TestFsCpDirToDirWithOverwriteFlag/local_to_dbfs✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p
TestFsCpFileToDir✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFsCpFileToDir/dbfs_to_dbfs✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFsCpFileToDir/dbfs_to_local✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFsCpFileToDir/local_to_uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s✅​p❌​F🙈​s🙈​s
TestFsCpFileToDir/uc-volumes_to_uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s✅​p❌​F🙈​s🙈​s
TestFsCpFileToFileFileNotOverwritten✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p🔄​f✅​p
TestFsCpFileToFileFileNotOverwritten/dbfs_to_local✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p🔄​f✅​p
TestFsCpFileToFileWithOverwriteFlag✅​p✅​p✅​p✅​p🔄​f🔄​f✅​p✅​p✅​p✅​p
TestFsCpFileToFileWithOverwriteFlag/dbfs_to_dbfs✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p
TestFsCpFileToFileWithOverwriteFlag/local_to_dbfs✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
TestFsCpSourceIsDirectoryButTargetIsFile✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p
TestFsCpSourceIsDirectoryButTargetIsFile/dbfs_to_dbfs✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p
TestFsLs✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p
TestFsLs/dbfs✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p
TestFsLsForNonexistingDir✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p
TestFsLsForNonexistingDir/uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s🔄​f✅​p🙈​s🙈​s
TestFsLsOnEmptyDir✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p
TestFsLsOnEmptyDir/uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s🔄​f✅​p🙈​s🙈​s
TestFsMkdirCreatesIntermediateDirectories✅​p✅​p✅​p✅​p✅​p🔄​f🔄​f✅​p✅​p✅​p
TestFsMkdirCreatesIntermediateDirectories/dbfs✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
TestFsMkdirCreatesIntermediateDirectories/uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s🔄​f✅​p🙈​s🙈​s
TestFsMkdirWhenFileExistsAtPath✅​p✅​p✅​p✅​p✅​p🔄​f🔄​f✅​p✅​p✅​p
TestFsMkdirWhenFileExistsAtPath/dbfs✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
TestFsMkdirWhenFileExistsAtPath/uc-volumes🙈​s🙈​s✅​p✅​p🙈​s🙈​s🔄​f✅​p🙈​s🙈​s
TestSecretsPutSecretStringValue✅​p✅​p🙈​s🙈​s✅​p✅​p✅​p❌​F✅​p✅​p
TestSyncNestedFolderDoesntFailOnNonEmptyDirectory✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestSyncNestedSpacePlusAndHashAreEscapedSync✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestImportDirDoesNotOverwrite✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFilerReadDir✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFilerReadDir/dbfs✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFilerReadWrite✅​p✅​p✅​p✅​p✅​p🔄​f✅​p❌​F✅​p✅​p
TestFilerReadWrite/dbfs✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFilerReadWrite/workspace_files_extensions✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
TestFilerRecursiveDelete✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestFilerRecursiveDelete/files🙈​s🙈​s✅​p✅​p🙈​s🙈​s✅​p❌​F🙈​s🙈​s
TestFilerRecursiveDelete/workspace_files✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
TestFilerRecursiveDelete/workspace_files_extensions✅​p✅​p✅​p✅​p✅​p🔄​f✅​p❌​F✅​p✅​p
TestWorkspaceFilesExtensions_ExportFormatIsPreserved✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_scala✅​p✅​p✅​p✅​p✅​p✅​p✅​p❌​F✅​p✅​p
Top 30 slowest tests (at least 2 minutes):
durationenvtestname
6:29gcp linuxTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
6:12azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
6:06gcp windowsTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
5:53aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:40gcp linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:38azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:33gcp windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:22aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:22gcp linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:21aws windowsTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
5:20aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:15gcp windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:14aws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:09aws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:08aws linuxTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
4:58aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
4:54aws-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:53azure windowsTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
4:45azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:01azure windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:50azure windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:49azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:39azure-ucws linuxTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:18azure-ucws linuxTestAccept/bundle/resources/synced_database_tables/basic
2:36aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:14azure-ucws windowsTestAccept/bundle/resources/synced_database_tables/basic
2:04azure-ucws windowsTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
2:03aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
2:02aws-ucws windowsTestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:01azure-ucws linuxTestAccept/ssh/connection

Comment threadlibs/template/templates/default/library/versions.tmpl
Expands the python_version_spec comment to explain why we use >=3.10,<3.13
rather than pinning to 3.12. The broader range improves usability by avoiding
the need for version managers while maintaining compatibility with DB Connect
15, serverless environment v4/v5, and DBR 16/17.
Addresses PR feedback: #3897 (review)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>

@kanterovkanterov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I've left a few suggestions

Comment threadlibs/template/templates/default/library/versions.tmpl
@lennartkats-db
lennartkats-db added this pull request to the merge queueNov 11, 2025
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Nov 11, 2025
@lennartkats-db
lennartkats-db added this pull request to the merge queueNov 13, 2025
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Nov 13, 2025
@lennartkats-db
lennartkats-db added this pull request to the merge queueNov 13, 2025
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Nov 13, 2025
- Resolved conflict in NEXT_CHANGELOG.md by including all changelog entries
- Removed pyproject.toml from SQL lakeflow template (deleted in main)
- Updated acceptance test outputs to match merged state
All tests pass locally (4797 tests).
@lennartkats-db
lennartkats-db added this pull request to the merge queueNov 14, 2025
Merged via the queue into main with commit e54dfe5Nov 14, 2025
13 checks passed
@lennartkats-db
lennartkats-db deleted the update-environment-version branch November 14, 2025 21:45
deco-sdk-taggingBot added a commit that referenced this pull request Nov 19, 2025
## Release v0.278.0
### Dependency updates
* Upgrade Go SDK to 0.91.0 ([#3948](#3948))
* Bump Alpine Linux to 3.22 in the Docker image ([#3942](#3942))
### Bundles
* Update templates to use serverless environment version 4 and matching Python version ([#3897](#3897))
* Add a language prompt to the `default-minimal` template ([#3918](#3918))
* Add `default-scala` template for Scala projects with SBT build configuration and example code ([#3906](#3906))
denik pushed a commit that referenced this pull request May 20, 2026
… Python version (#3897)
## Changes
This updates templates to serverless environment version 4 and
associated Python versions
Some considerations:
* Client version 4, client version 5 (upcoming), DBR 16, DBR 17 all use
Python 3.12
* DB Connect 15 supports all DBR versions above and supports Python 3.12
* DB Connect 15 also supports DBR 15, which uses Python 3.11 at runtime,
but that discrepancy already exists and seems okay
* Pinning Python to a very narrow range in pyproject.toml means users
always need to use uv to install the right version, which may not be
great for usability. so I haven't change that approach either
## Why
* Environment version 4 provides better startup performance and new
features
## Tests
* Standard acceptance testing
---------
Co-authored-by: Claude <noreply@anthropic.com>
denik pushed a commit that referenced this pull request May 20, 2026
## Release v0.278.0
### Dependency updates
* Upgrade Go SDK to 0.91.0 ([#3948](#3948))
* Bump Alpine Linux to 3.22 in the Docker image ([#3942](#3942))
### Bundles
* Update templates to use serverless environment version 4 and matching Python version ([#3897](#3897))
* Add a language prompt to the `default-minimal` template ([#3918](#3918))
* Add `default-scala` template for Scala projects with SBT build configuration and example code ([#3906](#3906))
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@lennartkats-db@eng-dev-ecosystem-bot@pietern@kanterov