Added test to submit and run various Python tasks on multiple DBR versions by andrewnester · Pull Request #806 · databricks/cli

andrewnester · 2023-09-26T14:04:08Z

Changes

These tests allow us to get information for execution context (PYTHONPATH, CWD) for various Python tasks and different cluster setups.

Note: this test won't be executed automatically as part of nightly builds since it requires RUN_PYTHON_TASKS_TEST env to be executed.

Tests

Integration test run successfully.

…sions

nfx · 2023-09-26T18:39:45Z

+var sparkVersions = []string{
+	"11.3.x-scala2.12",
+	"12.2.x-scala2.12",
+	"13.0.x-scala2.12",
+	"13.1.x-scala2.12",
+	"13.2.x-scala2.12",
+	"13.3.x-scala2.12",
+	"14.0.x-scala2.12",
+	"14.1.x-scala2.12",
+}


this list will get out of date in couple of months, can you use the w.Clusters.SparkVersions to pick the list of runtimes dynamically?
see https://github.com/databricks/databricks-sdk-py/blob/main/tests/integration/test_auth.py#L90-L124

I consider that but for now we want to keep it simple since we don't run them continuously yet. Once we do, we will get the list dynamically indeed

nfx · 2023-09-26T18:42:23Z

+ SparkVersion: sparkVersions[i],
+ NumWorkers: 1,
+ NodeTypeId: nodeTypeId,
+ DataSecurityMode: compute.DataSecurityModeUserIsolation,


this is only for Unity Catalog-enabled workspaces, but it doesn't check for .DataSecurityModeSingleUser and DataSecurityModeNone.
what about non-uc workspaces, like DataSecurityModeLegacySingleUser?

nfx · 2023-09-26T18:45:37Z

+
+func TestAccRunPythonTaskWorkspace(t *testing.T) {
+	// TODO: remove RUN_PYTHON_TASKS_TEST when ready to be executed as part of nightly
+	internal.GetEnvOrSkipTest(t, "RUN_PYTHON_TASKS_TEST")


there's ucws testing environment, which tests against UC-enabled workspace, use those env vars

atm we don't plan to execute it on continuously basis yet, just as a script to be executed manually to generate the data

yep, but you can still execute it with deco env shell and pick a *-ucws environment

pietern

General comment; since we don't run this as part of the regular suites, I think it should live in its own package. The helpers you introduce here are generic enough that they should be reusable from more testing packages.

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

pietern · 2023-10-02T09:04:39Z

Please include the source for this wheel file.

pietern

Stamp to unblock.

unblock merge

CLI: * Refactor change computation for sync ([#785](#785)). Bundles: * Allow digits in the generated short name ([#820](#820)). * Emit an error when incompatible all purpose cluster used with Python wheel tasks ([#823](#823)). * Use normalized short name for tag value in development mode ([#821](#821)). * Added `python.DetectInterpreters` and other utils ([#805](#805)). * Mark artifacts properties as optional ([#834](#834)). * Added support for glob patterns in pipeline libraries section ([#833](#833)). Internal: * Run tests to verify backend tag validation behavior ([#814](#814)). * Library to validate and normalize cloud specific tags ([#819](#819)). * Added test to submit and run various Python tasks on multiple DBR versions ([#806](#806)). * Create a release PR in setup-cli repo on tag push ([#827](#827)). API Changes: * Changed `databricks account metastore-assignments list` command to return . * Changed `databricks jobs cancel-all-runs` command with new required argument order. * Added `databricks account o-auth-published-apps` command group. * Changed `databricks serving-endpoints query` command . New request type is . * Added `databricks serving-endpoints patch` command. * Added `databricks credentials-manager` command group. * Added `databricks settings` command group. * Changed `databricks clean-rooms list` command to require request of . * Changed `databricks statement-execution execute-statement` command with new required argument order. OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02) Dependency updates: * Bump github.com/google/uuid from 1.3.0 to 1.3.1 ([#825](#825)). * Updated Go SDK to 0.22.0 ([#831](#831)).

…sions (#806) ## Changes These tests allow us to get information for execution context (PYTHONPATH, CWD) for various Python tasks and different cluster setups. Note: this test won't be executed automatically as part of nightly builds since it requires RUN_PYTHON_TASKS_TEST env to be executed. ## Tests Integration test run successfully. --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

CLI: * Refactor change computation for sync ([#785](#785)). Bundles: * Allow digits in the generated short name ([#820](#820)). * Emit an error when incompatible all purpose cluster used with Python wheel tasks ([#823](#823)). * Use normalized short name for tag value in development mode ([#821](#821)). * Added `python.DetectInterpreters` and other utils ([#805](#805)). * Mark artifacts properties as optional ([#834](#834)). * Added support for glob patterns in pipeline libraries section ([#833](#833)). Internal: * Run tests to verify backend tag validation behavior ([#814](#814)). * Library to validate and normalize cloud specific tags ([#819](#819)). * Added test to submit and run various Python tasks on multiple DBR versions ([#806](#806)). * Create a release PR in setup-cli repo on tag push ([#827](#827)). API Changes: * Changed `databricks account metastore-assignments list` command to return . * Changed `databricks jobs cancel-all-runs` command with new required argument order. * Added `databricks account o-auth-published-apps` command group. * Changed `databricks serving-endpoints query` command . New request type is . * Added `databricks serving-endpoints patch` command. * Added `databricks credentials-manager` command group. * Added `databricks settings` command group. * Changed `databricks clean-rooms list` command to require request of . * Changed `databricks statement-execution execute-statement` command with new required argument order. OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02) Dependency updates: * Bump github.com/google/uuid from 1.3.0 to 1.3.1 ([#825](#825)). * Updated Go SDK to 0.22.0 ([#831](#831)).

…sions (#806) ## Changes These tests allow us to get information for execution context (PYTHONPATH, CWD) for various Python tasks and different cluster setups. Note: this test won't be executed automatically as part of nightly builds since it requires RUN_PYTHON_TASKS_TEST env to be executed. ## Tests Integration test run successfully. --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

CLI: * Refactor change computation for sync ([#785](#785)). Bundles: * Allow digits in the generated short name ([#820](#820)). * Emit an error when incompatible all purpose cluster used with Python wheel tasks ([#823](#823)). * Use normalized short name for tag value in development mode ([#821](#821)). * Added `python.DetectInterpreters` and other utils ([#805](#805)). * Mark artifacts properties as optional ([#834](#834)). * Added support for glob patterns in pipeline libraries section ([#833](#833)). Internal: * Run tests to verify backend tag validation behavior ([#814](#814)). * Library to validate and normalize cloud specific tags ([#819](#819)). * Added test to submit and run various Python tasks on multiple DBR versions ([#806](#806)). * Create a release PR in setup-cli repo on tag push ([#827](#827)). API Changes: * Changed `databricks account metastore-assignments list` command to return . * Changed `databricks jobs cancel-all-runs` command with new required argument order. * Added `databricks account o-auth-published-apps` command group. * Changed `databricks serving-endpoints query` command . New request type is . * Added `databricks serving-endpoints patch` command. * Added `databricks credentials-manager` command group. * Added `databricks settings` command group. * Changed `databricks clean-rooms list` command to require request of . * Changed `databricks statement-execution execute-statement` command with new required argument order. OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02) Dependency updates: * Bump github.com/google/uuid from 1.3.0 to 1.3.1 ([#825](#825)). * Updated Go SDK to 0.22.0 ([#831](#831)).

Added test to submit and run various Python tasks on multiple DBR ver…
353e75a
…sions

andrewnester requested review from pietern and shreyas-goenka September 26, 2023 14:04

correctly skip files
e5b6dbd

nfx previously requested changes Sep 26, 2023
View reviewed changes

pietern reviewed Sep 27, 2023
View reviewed changes

Comment threadinternal/bundle/helpers.go Outdated
Comment threadinternal/bundle/helpers.go Outdated
Comment threadinternal/bundle/helpers.go Outdated

andrewnesterand others added 3 commits September 27, 2023 09:50

Update internal/bundle/helpers.go
0dfdd3e
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

Merge branch 'main' into run-python-task-everywhere
514b5c9

refactored
828a463

andrewnester requested a review from pietern September 27, 2023 10:09

use CLI repo as repo basis
aa5d241

pietern reviewed Oct 2, 2023
View reviewed changes

Comment threadinternal/python/testdata/my_test_code-0.0.1-py3-none-any.whl
pieternOct 2, 2023
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please include the source for this wheel file.

pietern approved these changes Oct 2, 2023
View reviewed changes

andrewnester enabled auto-merge October 3, 2023 11:11

Merge branch 'main' into run-python-task-everywhere
720b617

andrewnester requested a review from nfx October 3, 2023 11:11

andrewnester disabled auto-merge October 3, 2023 11:17

andrewnester added this pull request to the merge queue Oct 3, 2023

Merged via the queue into main with commit 79e271fOct 3, 2023

andrewnester deleted the run-python-task-everywhere branch October 3, 2023 11:23

andrewnester mentioned this pull request Oct 4, 2023
Release v0.207.0 #839
Merged

Navigation Menu

Uh oh!

Added test to submit and run various Python tasks on multiple DBR versions - #806

Added test to submit and run various Python tasks on multiple DBR versions#806
andrewnester merged 7 commits into
mainfrom
run-python-task-everywhere

andrewnester commented Sep 26, 2023

Uh oh!

nfxSep 26, 2023

Uh oh!

andrewnesterSep 27, 2023

Uh oh!

nfxSep 26, 2023

Uh oh!

Uh oh!

nfxSep 26, 2023

Uh oh!

andrewnesterSep 27, 2023

Uh oh!

nfxSep 27, 2023

Uh oh!

pietern left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternOct 2, 2023

Uh oh!

pietern left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

andrewnester commented Sep 26, 2023

Changes

Tests

Uh oh!

nfxSep 26, 2023

Choose a reason for hiding this comment

Uh oh!

andrewnesterSep 27, 2023

Choose a reason for hiding this comment

Uh oh!

nfxSep 26, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nfxSep 26, 2023

Choose a reason for hiding this comment

Uh oh!

andrewnesterSep 27, 2023

Choose a reason for hiding this comment

Uh oh!

nfxSep 27, 2023

Choose a reason for hiding this comment

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternOct 2, 2023

Choose a reason for hiding this comment

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants