Add configurable presets for name prefixes, tags, etc. by lennartkats-db · Pull Request #1490 · databricks/cli

lennartkats-db · 2024-06-12T15:31:04Z

Changes

This adds configurable transformations based on the transformations currently seen in mode: development.

Example databricks.yml showcasing how some transformations:

bundle:
name: my_bundle
targets:
dev:
presets:
prefix: "myprefix_" # prefix all resource names with myprefix_
pipelines_development: true # set development to true by default for pipelines
trigger_pause_status: PAUSED # set pause_status to PAUSED by default for all triggers and schedules
jobs_max_concurrent_runs: 10 # set max_concurrent runs to 10 by default for all jobs
tags:
dev: true

Tests

Existing process_target_mode tests that were adapted to use this new code
Unit tests specific for the new mutator
Unit tests for config loading and merging
Manual e2e testing

codecov-commenter · 2024-06-14T12:37:19Z

Codecov Report

Attention: Patch coverage is 75.00000% with 48 lines in your changes missing coverage. Please review.

Project coverage is 53.69%. Comparing base (e22dd8a) to head (4dc5f41).
Report is 142 commits behind head on main.

Files	Patch %	Lines
bundle/config/mutator/process_target_mode.go	68.57%	16 Missing and 6 partials ⚠️
bundle/config/mutator/apply_transforms.go	84.82%	12 Missing and 5 partials ⚠️
bundle/config/root.go	16.66%	4 Missing and 1 partial ⚠️
bundle/config/transforms.go	0.00%	4 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #1490 +/- ##
==========================================
+ Coverage 52.25% 53.69% +1.43% 
==========================================
Files 317 353 +36 Lines 18004 20406 +2402 ==========================================
+ Hits 9408 10956 +1548 - Misses 7903 8642 +739 - Partials 693 808 +115

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pietern · 2024-06-18T06:53:33Z

Thanks, will be great to have this.

This cherry-picks from #1490 to address an issue that came up in #1511. The function `dyn.SetByPath` requires intermediate values to be present. If they are not, it returns an error that it cannot index a map. This is not an issue on main where the intermediate maps are always created, even if they are not present in the dynamic configuration tree. As of #1511 we'll no longer populate empty maps for empty structs if they are not explictly set (i.e. a non-nil pointer). We can write a bool pointer to avoid this issue altogether.

## Changes This cherry-picks from #1490 to address an issue that came up in #1511. The function `dyn.SetByPath` requires intermediate values to be present. If they are not, it returns an error that it cannot index a map. This is not an issue on main, where the intermediate maps are always created, even if they are not present in the dynamic configuration tree. As of #1511, we'll no longer populate empty maps for empty structs if they are not explicitly set (i.e., a non-nil pointer). This change writes a bool pointer to avoid this issue altogether. ## Tests Unit tests pass.

lennartkats-db · 2024-07-19T19:55:57Z

@pietern PTAL, I took another pass here

lennartkats-db · 2024-08-16T07:30:29Z

When ready, I'd like to first merge #1590. cc @pietern

## Changes In #1490 we regressed and started using the development mode prefix for UC schemas regardless of the mode of the bundle target. This PR fixes the regression and adds a regression test ## Tests Failing integration tests pass now.

CLI: * Added filtering flags for cluster list commands ([#1703](#1703)). Bundles: * Remove reference to "dbt" in the default-sql template ([#1696](#1696)). * Pause continuous pipelines when 'mode: development' is used ([#1590](#1590)). * Add configurable presets for name prefixes, tags, etc. ([#1490](#1490)). * Report all empty resources present in error diagnostic ([#1685](#1685)). * Improves detection of PyPI package names in environment dependencies ([#1699](#1699)). * [DAB] Add support for requirements libraries in Job Tasks ([#1543](#1543)). * Add paths field to bundle sync configuration ([#1694](#1694)). Internal: * Add `import` option for PyDABs ([#1693](#1693)). * Make fileset take optional list of paths to list ([#1684](#1684)). * Pass through paths argument to libs/sync ([#1689](#1689)). * Correctly mark package names with versions as remote libraries ([#1697](#1697)). * Share test initializer in common helper function ([#1695](#1695)). * Make `pydabs/venv_path` optional ([#1687](#1687)). * Use API mocks for duplicate path errors in workspace files extensions client ([#1690](#1690)). * Fix prefix preset used for UC schemas ([#1704](#1704)).

## Changes This cherry-picks from #1490 to address an issue that came up in #1511. The function `dyn.SetByPath` requires intermediate values to be present. If they are not, it returns an error that it cannot index a map. This is not an issue on main, where the intermediate maps are always created, even if they are not present in the dynamic configuration tree. As of #1511, we'll no longer populate empty maps for empty structs if they are not explicitly set (i.e., a non-nil pointer). This change writes a bool pointer to avoid this issue altogether. ## Tests Unit tests pass.

## Changes This adds configurable transformations based on the transformations currently seen in `mode: development`. Example databricks.yml showcasing how some transformations: ``` bundle: name: my_bundle targets: dev: presets: prefix: "myprefix_" # prefix all resource names with myprefix_ pipelines_development: true # set development to true by default for pipelines trigger_pause_status: PAUSED # set pause_status to PAUSED by default for all triggers and schedules jobs_max_concurrent_runs: 10 # set max_concurrent runs to 10 by default for all jobs tags: dev: true ``` ## Tests * Existing process_target_mode tests that were adapted to use this new code * Unit tests specific for the new mutator * Unit tests for config loading and merging * Manual e2e testing

## Changes In #1490 we regressed and started using the development mode prefix for UC schemas regardless of the mode of the bundle target. This PR fixes the regression and adds a regression test ## Tests Failing integration tests pass now.

CLI: * Added filtering flags for cluster list commands ([#1703](#1703)). Bundles: * Remove reference to "dbt" in the default-sql template ([#1696](#1696)). * Pause continuous pipelines when 'mode: development' is used ([#1590](#1590)). * Add configurable presets for name prefixes, tags, etc. ([#1490](#1490)). * Report all empty resources present in error diagnostic ([#1685](#1685)). * Improves detection of PyPI package names in environment dependencies ([#1699](#1699)). * [DAB] Add support for requirements libraries in Job Tasks ([#1543](#1543)). * Add paths field to bundle sync configuration ([#1694](#1694)). Internal: * Add `import` option for PyDABs ([#1693](#1693)). * Make fileset take optional list of paths to list ([#1684](#1684)). * Pass through paths argument to libs/sync ([#1689](#1689)). * Correctly mark package names with versions as remote libraries ([#1697](#1697)). * Share test initializer in common helper function ([#1695](#1695)). * Make `pydabs/venv_path` optional ([#1687](#1687)). * Use API mocks for duplicate path errors in workspace files extensions client ([#1690](#1690)). * Fix prefix preset used for UC schemas ([#1704](#1704)).

lennartkats-db added 10 commits June 8, 2024 18:40

Initial draft of customizable transformation
e5ac74d

Refactor to one function per transformer
6eaee84

WIP
601c32b

Remove enabled fields
b16c18c

Add config merging & test
901097a

e2e fixes
a815f30

Cleanup
13630dd

Use PAUSED/UNPAUSED instead of a boolean
e3b0435

Rename to transform for now
10a1ffc

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings
405e202

lennartkats-db changed the title ~~[WIP] Add configurable transformations for name prefixes, tags, etc.~~Add configurable transformations for name prefixes, tags, etc.Jun 12, 2024

lennartkats-db marked this pull request as ready for review June 12, 2024 16:16

lennartkats-db requested review from andrewnester, pietern and shreyas-goenka June 12, 2024 16:17

lennartkats-db mentioned this pull request Jun 12, 2024
Pause quality monitors when "mode: development" is used #1481
Merged
1 task

Cleanup
7323d02

Add stricter validations
4dc5f41

pietern reviewed Jun 18, 2024
View reviewed changes

lennartkats-db mentioned this pull request Jun 18, 2024
Customize the prefix assigned to resources deployed in development mode #1084
Closed

Cleanup
82e1d49

lennartkats-db force-pushed the cp-mutator-settings branch from 6fd240d to 82e1d49Compare June 19, 2024 08:33

Simply mutations, no need for dyn here
6d75e84

pietern reviewed Jun 20, 2024
View reviewed changes

andrewnester mentioned this pull request Jun 20, 2024
[BUG] Bundle in dev mode uses service principal client ID in the Job Name #1509
Closed

pietern mentioned this pull request Jun 21, 2024
Set bool pointer to disable lock #1516
Merged

andrewnester mentioned this pull request Aug 1, 2024
Alter bundle asset development mode tags for jobs and pipelines #773
Closed

pietern reviewed Aug 12, 2024
View reviewed changes

Comment thread.vscode/settings.json
Comment threadbundle/config/mutator/apply_presets_test.go Outdated
Comment threadbundle/config/mutator/process_target_mode.go
Comment threadbundle/config/mutator/process_target_mode.go
Comment threadbundle/tests/presets_test.go

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings
6159c3c

lennartkats-db force-pushed the cp-mutator-settings branch from 61704c8 to 6159c3cCompare August 14, 2024 14:53

Use bundle.Apply() for tests
b4564f2

lennartkats-db requested a review from pietern August 14, 2024 15:00

lennartkats-db added 2 commits August 16, 2024 09:27

Add assertion
70d8988

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings
a073f84

lennartkats-db enabled auto-merge August 16, 2024 07:29

lennartkats-db disabled auto-merge August 16, 2024 07:30

pietern approved these changes Aug 19, 2024
View reviewed changes

andrewnester approved these changes Aug 19, 2024
View reviewed changes

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings
b9e3278

lennartkats-db enabled auto-merge August 19, 2024 18:17

lennartkats-db added this pull request to the merge queue Aug 19, 2024

Merged via the queue into databricks:main with commit 78d0ac5Aug 19, 2024

lennartkats-db deleted the cp-mutator-settings branch August 19, 2024 18:26

shreyas-goenka mentioned this pull request Aug 21, 2024
Fix prefix preset used for UC schemas #1704
Merged

andrewnester mentioned this pull request Aug 21, 2024
[Release] Release v0.227.0 #1705
Merged

denik mentioned this pull request Apr 25, 2025
Make apply_presets.go more resilient and consistent #2777
Merged

Uh oh!

Conversation

lennartkats-db commented Jun 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

codecov-commenter commented Jun 14, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pietern commented Jun 18, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lennartkats-db commented Jul 19, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lennartkats-db commented Aug 16, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lennartkats-db commented Jun 12, 2024•
edited
Loading

codecov-commenter commented Jun 14, 2024•
edited
Loading

lennartkats-db commented Aug 16, 2024•
edited
Loading