Skip to content

Fixed spark version check for clusters defined in the same bundle - #2374

Merged
andrewnester merged 3 commits into
mainfrom
fix/python-warning-interactive-cluster
Feb 26, 2025
Merged

Fixed spark version check for clusters defined in the same bundle#2374
andrewnester merged 3 commits into
mainfrom
fix/python-warning-interactive-cluster

Conversation

@andrewnester

@andrewnesterandrewnester commented Feb 24, 2025

Copy link
Copy Markdown
Contributor

Changes

Previously using python wheel tasks in the tasks with compute referering to interactive cluster defied in the same bundle would produce a warning like below

GET /api/2.1/clusters/get?cluster_id=${resources.clusters.development_cluster.id}
< HTTP/2.0 400 Bad Request
< {
< "error_code": "INVALID_PARAMETER_VALUE",
< "message": "Cluster ${resources.clusters.development_cluster.id} does not exist"
< } pid=14465 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning sdk=true

This PR fixes it by making sure that we check spark version for such clusters based on its bundle configuration and don't make API calls

Tests

Added acceptance test

denik
denik reviewed Feb 25, 2025
View reviewed changes
Comment threadbundle/trampoline/python_dbr_warning.go Outdated
shreyas-goenka
shreyas-goenka reviewed Feb 25, 2025
View reviewed changes
Comment threadbundle/trampoline/python_dbr_warning.go
Comment threadbundle/trampoline/python_dbr_warning.go Outdated
Comment threadbundle/trampoline/python_dbr_warning.go
pietern
pietern reviewed Feb 26, 2025
View reviewed changes
Comment threadacceptance/bundle/trampoline/warning_message/databricks.yml Outdated
Comment threadacceptance/bundle/trampoline/warning_message/databricks.yml Outdated
Comment threadbundle/trampoline/python_dbr_warning.go Outdated
Comment threadbundle/trampoline/python_dbr_warning.go
shreyas-goenka
shreyas-goenka reviewed Feb 26, 2025
View reviewed changes
Comment threadbundle/trampoline/python_dbr_warning.go
// So we can get the version from the cluster definition.
// It's defined in a form of resources.clusters.<cluster_key>.spark_version while the value here is
// ${resources.clusters.<cluster_key>.id}
if strings.HasPrefix(task.ExistingClusterId, "${") {

@shreyas-goenkashreyas-goenkaFeb 26, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to have this check, you can directly use the ok value from dynvar.PureReferenceToPath

@andrewnesterandrewnesterFeb 26, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I know, just want to make the intention here a bit more clear when reading it

@andrewnester
andrewnester merged commit cdea775 into mainFeb 26, 2025
@andrewnester
andrewnester deleted the fix/python-warning-interactive-cluster branch February 26, 2025 12:04
andrewnester added a commit that referenced this pull request Feb 27, 2025
@andrewnester
[Release] Release v0.242.0
0818f2a
Notable changes:
Starting this version CLI does not load bundle auth information when CLI command is executed inside the bundle directory with explicitly provided via `-p` flag profile.
CLI:
* Do not load host from bundle for CLI commands when profile flag is used ([#2335](#2335)).
* Fixed accessing required path parameters in CLI generation when --json flag ([#2373](#2373)).
Bundles:
* Provide instructions for testing in the default-python template ([#2355](#2355)).
* Remove `run_as` from the built-in templates ([#2044](#2044)).
* Change warning about incomplete permissions section into a recommendation ([#2043](#2043)).
* Refine `mode: production` diagnostic output ([#2236](#2236)).
* Support serverless mode in default-python template (explicit prompt) ([#2377](#2377)).
* Set default data_security_mode to "SINGLE_USER" in bundle templates ([#2372](#2372)).
* Fixed spark version check for clusters defined in the same bundle ([#2374](#2374)).
API Changes:
* Added `databricks genie get-message-query-result-by-attachment` command.
OpenAPI commit 99f644e72261ef5ecf8d74db20f4b7a1e09723cc (2025-02-11)
@andrewnesterandrewnester mentioned this pull request Feb 27, 2025
Merged
github-merge-queueBot pushed a commit that referenced this pull request Feb 27, 2025
@andrewnester
[Release] Release v0.242.0 (#2393)
767c0a9
Notable changes:
Starting this version CLI does not load bundle auth information when CLI
command is executed inside the bundle directory with explicitly provided
via `-p` flag profile. For more details see the related GitHub issue
#1358
CLI:
* Do not load host from bundle for CLI commands when profile flag is
used ([#2335](#2335)).
* Fixed accessing required path parameters in CLI generation when --json
flag ([#2373](#2373)).
Bundles:
* Provide instructions for testing in the default-python template
([#2355](#2355)).
* Remove `run_as` from the built-in templates
([#2044](#2044)).
* Change warning about incomplete permissions section into a
recommendation ([#2043](#2043)).
* Refine `mode: production` diagnostic output
([#2236](#2236)).
* Support serverless mode in default-python template (explicit prompt)
([#2377](#2377)).
* Set default data_security_mode to "SINGLE_USER" in bundle templates
([#2372](#2372)).
* Fixed spark version check for clusters defined in the same bundle
([#2374](#2374)).
API Changes:
* Added `databricks genie get-message-query-result-by-attachment`
command.
OpenAPI commit 99f644e72261ef5ecf8d74db20f4b7a1e09723cc (2025-02-11)
HariGS-DB pushed a commit to HariGS-DB/cli that referenced this pull request Feb 28, 2025
@andrewnester@HariGS-DB
Fixed spark version check for clusters defined in the same bundle (da…
b50c6d5
…tabricks#2374)
## Changes
Previously using python wheel tasks in the tasks with compute referering
to interactive cluster defied in the same bundle would produce a warning
like below
```
GET /api/2.1/clusters/get?cluster_id=${resources.clusters.development_cluster.id}
< HTTP/2.0 400 Bad Request
< {
< "error_code": "INVALID_PARAMETER_VALUE",
< "message": "Cluster ${resources.clusters.development_cluster.id} does not exist"
< } pid=14465 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning sdk=true
```
This PR fixes it by making sure that we check spark version for such
clusters based on its bundle configuration and don't make API calls
## Tests
Added acceptance test
HariGS-DB pushed a commit to HariGS-DB/cli that referenced this pull request Feb 28, 2025
@andrewnester@HariGS-DB
[Release] Release v0.242.0 (databricks#2393)
b04fce4
Notable changes:
Starting this version CLI does not load bundle auth information when CLI
command is executed inside the bundle directory with explicitly provided
via `-p` flag profile. For more details see the related GitHub issue
databricks#1358
CLI:
* Do not load host from bundle for CLI commands when profile flag is
used ([databricks#2335](databricks#2335)).
* Fixed accessing required path parameters in CLI generation when --json
flag ([databricks#2373](databricks#2373)).
Bundles:
* Provide instructions for testing in the default-python template
([databricks#2355](databricks#2355)).
* Remove `run_as` from the built-in templates
([databricks#2044](databricks#2044)).
* Change warning about incomplete permissions section into a
recommendation ([databricks#2043](databricks#2043)).
* Refine `mode: production` diagnostic output
([databricks#2236](databricks#2236)).
* Support serverless mode in default-python template (explicit prompt)
([databricks#2377](databricks#2377)).
* Set default data_security_mode to "SINGLE_USER" in bundle templates
([databricks#2372](databricks#2372)).
* Fixed spark version check for clusters defined in the same bundle
([databricks#2374](databricks#2374)).
API Changes:
* Added `databricks genie get-message-query-result-by-attachment`
command.
OpenAPI commit 99f644e72261ef5ecf8d74db20f4b7a1e09723cc (2025-02-11)
denik pushed a commit that referenced this pull request May 20, 2026
@andrewnester
Fixed spark version check for clusters defined in the same bundle (#2374
af0232b
)
## Changes
Previously using python wheel tasks in the tasks with compute referering
to interactive cluster defied in the same bundle would produce a warning
like below
```
GET /api/2.1/clusters/get?cluster_id=${resources.clusters.development_cluster.id}
< HTTP/2.0 400 Bad Request
< {
< "error_code": "INVALID_PARAMETER_VALUE",
< "message": "Cluster ${resources.clusters.development_cluster.id} does not exist"
< } pid=14465 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning sdk=true
```
This PR fixes it by making sure that we check spark version for such
clusters based on its bundle configuration and don't make API calls
## Tests
Added acceptance test
denik pushed a commit that referenced this pull request May 20, 2026
@andrewnester
[Release] Release v0.242.0 (#2393)
996a0a7
Notable changes:
Starting this version CLI does not load bundle auth information when CLI
command is executed inside the bundle directory with explicitly provided
via `-p` flag profile. For more details see the related GitHub issue
#1358
CLI:
* Do not load host from bundle for CLI commands when profile flag is
used ([#2335](#2335)).
* Fixed accessing required path parameters in CLI generation when --json
flag ([#2373](#2373)).
Bundles:
* Provide instructions for testing in the default-python template
([#2355](#2355)).
* Remove `run_as` from the built-in templates
([#2044](#2044)).
* Change warning about incomplete permissions section into a
recommendation ([#2043](#2043)).
* Refine `mode: production` diagnostic output
([#2236](#2236)).
* Support serverless mode in default-python template (explicit prompt)
([#2377](#2377)).
* Set default data_security_mode to "SINGLE_USER" in bundle templates
([#2372](#2372)).
* Fixed spark version check for clusters defined in the same bundle
([#2374](#2374)).
API Changes:
* Added `databricks genie get-message-query-result-by-attachment`
command.
OpenAPI commit 99f644e72261ef5ecf8d74db20f4b7a1e09723cc (2025-02-11)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pieternpieternpietern approved these changes
@denikdenikAwaiting requested review from denik
@shreyas-goenkashreyas-goenkaAwaiting requested review from shreyas-goenka

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@andrewnester@pietern@denik@shreyas-goenka