Add ApplyPythonMutator by kanterov · Pull Request #1430 · databricks/cli

kanterov · 2024-05-14T12:33:42Z

Changes

Add ApplyPythonMutator, which will fork the Python subprocess and process pipe bundle configuration through it.

It's enabled through experimental section, for example:

experimental:
pydabs: enable: truevenv_path: .venv

For now, it's limited to two phases in the mutator pipeline:

load: adds new jobs
init: adds new jobs, or modifies existing ones

It's enforced that no jobs are modified in load and not jobs are deleted in load/init, because, otherwise, it will break existing assumptions.

Tests

Unit tests

codecov-commenter · 2024-05-14T12:37:22Z

Codecov Report

Attention: Patch coverage is 70.42254% with 42 lines in your changes missing coverage. Please review.

Project coverage is 54.96%. Comparing base (e22dd8a) to head (48d9328).
Report is 146 commits behind head on main.

Files	Patch %	Lines
...ndle/config/mutator/python/apply_python_mutator.go	83.18%	10 Missing and 9 partials ⚠️
libs/process/opts.go	0.00%	12 Missing ⚠️
bundle/config/mutator/python/log_writer.go	28.57%	10 Missing ⚠️
bundle/config/mutator/mutator.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #1430 +/- ##
==========================================
+ Coverage 52.25% 54.96% +2.71% 
==========================================
Files 317 353 +36 Lines 18004 16238 -1766 ==========================================
- Hits 9408 8926 -482 + Misses 7903 6499 -1404 - Partials 693 813 +120

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kanterov · 2024-05-17T14:24:45Z

@pietern can you please take a look, since the last time:

merge.Override has changed, so now we implement OverrideVisitor here, I added tests for it as well
left a bare minimum code to log stderr for the Python process, to be continued as a follow-up
addressed remaining comments

kanterov · 2024-06-19T13:32:14Z

@pietern I've addressed all comments, and also renamed "pre-init" to "load" as we discussed offline

pietern

One remaining minor thing. Approving to unblock. Thanks!

pietern · 2024-06-20T06:48:06Z

+
+	if job1, ok := b.Config.Resources.Jobs["job1"]; ok {
+ assert.Equal(t, "job_1", job1.Name)
+	}


Potential for follow-up: assert that the location information is persisted across the call.
As is, this would pass if the mutator uses only the process output.

Agree. There are tests for it in dyn.Override, but since this is important behaviour, we should have a simple test for ApplyPythonMutator as well.

I will add in a separate PR

kanterov · 2024-06-20T07:20:41Z

@pietern thanks for the review, can you please add to the merge queue?

CLI: * Add link to documentation for Homebrew installation to README ([#1505](#1505)). * Fix `databricks configure` to use `DATABRICKS_CONFIG_FILE` environment variable if exists as config file ([#1325](#1325)). Bundles: The Terraform upgrade to v1.48.0 includes a fix for library order not being respected. * Fix conditional in query in `default-sql` template ([#1479](#1479)). * Remove user credentials specified in the Git origin URL ([#1494](#1494)). * Serialize dynamic value for `bundle validate` output ([#1499](#1499)). * Override variables with lookup value even if values has default value set ([#1504](#1504)). * Pause quality monitors when "mode: development" is used ([#1481](#1481)). * Return `fs.ModeDir` for Git folders in the workspace ([#1521](#1521)). * Upgrade TF provider to 1.48.0 ([#1527](#1527)). * Added support for complex variables ([#1467](#1467)). Internal: * Add randIntn function ([#1475](#1475)). * Avoid multiple file tree traversals on bundle deploy ([#1493](#1493)). * Clean up unused code ([#1502](#1502)). * Use `dyn.InvalidValue` to indicate absence ([#1507](#1507)). * Add ApplyPythonMutator ([#1430](#1430)). * Set bool pointer to disable lock ([#1516](#1516)). * Allow the any type to be set to nil in `convert.FromTyped` ([#1518](#1518)). * Properly deal with nil values in `convert.FromTyped` ([#1511](#1511)). * Return `dyn.InvalidValue` instead of `dyn.NilValue` when errors happen ([#1514](#1514)). * PythonMutator: replace stdin/stdout with files ([#1512](#1512)). * Add context type and value to path rewriting ([#1525](#1525)). API Changes: * Added schedule CRUD commands to `databricks lakeview`. * Added subscription CRUD commands to `databricks lakeview`. * Added `databricks apps start` command. OpenAPI commit 7437dabb9dadee402c1fc060df4c1ce8cc5369f0 (2024-06-24) Dependency updates: * Bump golang.org/x/text from 0.15.0 to 0.16.0 ([#1482](#1482)). * Bump golang.org/x/term from 0.20.0 to 0.21.0 ([#1483](#1483)). * Bump golang.org/x/mod from 0.17.0 to 0.18.0 ([#1484](#1484)). * Bump golang.org/x/oauth2 from 0.20.0 to 0.21.0 ([#1485](#1485)). * Bump github.com/briandowns/spinner from 1.23.0 to 1.23.1 ([#1495](#1495)). * Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 ([#1496](#1496)). * Bump github.com/databricks/databricks-sdk-go from 0.42.0 to 0.43.0 ([#1522](#1522)).

## Changes Add ApplyPythonMutator, which will fork the Python subprocess and process pipe bundle configuration through it. It's enabled through `experimental` section, for example: ```yaml experimental: pydabs: enable: true venv_path: .venv ``` For now, it's limited to two phases in the mutator pipeline: - `load`: adds new jobs - `init`: adds new jobs, or modifies existing ones It's enforced that no jobs are modified in `load` and not jobs are deleted in `load/init`, because, otherwise, it will break existing assumptions. ## Tests Unit tests

CLI: * Add link to documentation for Homebrew installation to README ([#1505](#1505)). * Fix `databricks configure` to use `DATABRICKS_CONFIG_FILE` environment variable if exists as config file ([#1325](#1325)). Bundles: The Terraform upgrade to v1.48.0 includes a fix for library order not being respected. * Fix conditional in query in `default-sql` template ([#1479](#1479)). * Remove user credentials specified in the Git origin URL ([#1494](#1494)). * Serialize dynamic value for `bundle validate` output ([#1499](#1499)). * Override variables with lookup value even if values has default value set ([#1504](#1504)). * Pause quality monitors when "mode: development" is used ([#1481](#1481)). * Return `fs.ModeDir` for Git folders in the workspace ([#1521](#1521)). * Upgrade TF provider to 1.48.0 ([#1527](#1527)). * Added support for complex variables ([#1467](#1467)). Internal: * Add randIntn function ([#1475](#1475)). * Avoid multiple file tree traversals on bundle deploy ([#1493](#1493)). * Clean up unused code ([#1502](#1502)). * Use `dyn.InvalidValue` to indicate absence ([#1507](#1507)). * Add ApplyPythonMutator ([#1430](#1430)). * Set bool pointer to disable lock ([#1516](#1516)). * Allow the any type to be set to nil in `convert.FromTyped` ([#1518](#1518)). * Properly deal with nil values in `convert.FromTyped` ([#1511](#1511)). * Return `dyn.InvalidValue` instead of `dyn.NilValue` when errors happen ([#1514](#1514)). * PythonMutator: replace stdin/stdout with files ([#1512](#1512)). * Add context type and value to path rewriting ([#1525](#1525)). API Changes: * Added schedule CRUD commands to `databricks lakeview`. * Added subscription CRUD commands to `databricks lakeview`. * Added `databricks apps start` command. OpenAPI commit 7437dabb9dadee402c1fc060df4c1ce8cc5369f0 (2024-06-24) Dependency updates: * Bump golang.org/x/text from 0.15.0 to 0.16.0 ([#1482](#1482)). * Bump golang.org/x/term from 0.20.0 to 0.21.0 ([#1483](#1483)). * Bump golang.org/x/mod from 0.17.0 to 0.18.0 ([#1484](#1484)). * Bump golang.org/x/oauth2 from 0.20.0 to 0.21.0 ([#1485](#1485)). * Bump github.com/briandowns/spinner from 1.23.0 to 1.23.1 ([#1495](#1495)). * Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 ([#1496](#1496)). * Bump github.com/databricks/databricks-sdk-go from 0.42.0 to 0.43.0 ([#1522](#1522)).

kanterov force-pushed the python-mutator branch from df082e6 to 4641157Compare May 14, 2024 12:38

pietern reviewed May 15, 2024
View reviewed changes

kanterov force-pushed the python-mutator branch from 4641157 to 4c23dc4Compare May 17, 2024 12:21

kanterov requested a review from pietern May 17, 2024 14:23

lennartkats-db reviewed May 22, 2024
View reviewed changes

Comment threadbundle/config/experimental.go Outdated
Comment threadbundle/config/mutator/python/apply_python_mutator.go Outdated
Comment threadbundle/config/mutator/python/apply_python_mutator_test.go Outdated

pietern reviewed May 23, 2024
View reviewed changes

kanterov added 3 commits June 17, 2024 16:13

Add ApplyPythonMutator
463fbb6

Address CR comments
16fc994

Address CR comments
05c661d

kanterov force-pushed the python-mutator branch from d2986fd to 05c661dCompare June 17, 2024 14:13

kanterov requested review from lennartkats-db and pietern June 17, 2024 14:14

kanterov added 2 commits June 17, 2024 16:44

Fix tests on Windows
48d9328

Fix tests on Windows
ebca927

pietern reviewed Jun 19, 2024
View reviewed changes

Address CR comments
3a49549

kanterov added 3 commits June 19, 2024 15:37

Address CR comments
74a41f4

Fix fmt
c5de423

Fix build
ab50f40

kanterov requested a review from pietern June 19, 2024 13:58

pietern approved these changes Jun 20, 2024
View reviewed changes

kanterovand others added 2 commits June 20, 2024 09:17

Fix comment
8b31ad8

Merge branch 'main' into python-mutator
fc2ab9e

pietern added this pull request to the merge queue Jun 20, 2024

Merged via the queue into databricks:main with commit 57a5a65Jun 20, 2024

pietern mentioned this pull request Jun 26, 2024
Release v0.222.0 #1529
Merged

Uh oh!

Conversation

kanterov commented May 14, 2024• edited by pietern Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

codecov-commenter commented May 14, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kanterov commented May 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kanterov commented Jun 19, 2024

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pieternJun 20, 2024

Choose a reason for hiding this comment

Uh oh!

kanterovJun 20, 2024

Choose a reason for hiding this comment

Uh oh!

kanterovJun 20, 2024

Choose a reason for hiding this comment

Uh oh!

kanterov commented Jun 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kanterov commented May 14, 2024•
edited by pietern
Loading

codecov-commenter commented May 14, 2024•
edited
Loading