Skip to content

Extend deployment mode support - #577

Merged
lennartkats-db merged 8 commits into
mainfrom
extend-deployment-modes-production
Jul 30, 2023
Merged

Extend deployment mode support#577
lennartkats-db merged 8 commits into
mainfrom
extend-deployment-modes-production

Conversation

@lennartkats-db

@lennartkats-dblennartkats-db commented Jul 17, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

This adds mode: production option. This mode doesn't do any transformations but verifies that an environment is configured correctly for production:

environments:
prod:
mode: production
# paths should not be scoped to a user (unless a service principal is used)
root_path: /Shared/non_user_path/...
# run_as and permissions should be set at the resource level (or at the top level when that is implemented)
run_as:
user_name: Alice
permissions:
- level: CAN_MANAGE
user_name: Alice

Additionally, this extends the existing mode: development option,

  • now prefixing deployed assets with [dev your.user] instead of just [dev]
  • validating that development deployments are scoped to a user

Related

https://github.com/databricks/cli/pull/578/files (in draft)

Tests

Manual testing to validate the experience, error messages, and functionality with all resource types. Automated unit tests.

@lennartkats-dblennartkats-db mentioned this pull request Jul 17, 2023
Merged
@lennartkats-db
lennartkats-db marked this pull request as ready for review July 17, 2023 07:31
fjakobs
fjakobs reviewed Jul 20, 2023
View reviewed changes

@fjakobsfjakobs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes make sense to me. Would be great to also expose these validations separate from the mode in the config.

Comment threadbundle/config/environment.go Outdated
Comment threadbundle/config/mutator/process_environment_mode.go
Comment threadbundle/config/mutator/process_environment_mode.go
pietern
pietern reviewed Jul 21, 2023
View reviewed changes

@pieternpietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel we need to fine tune the policies listed here. The approach and test coverage looks good.

Comment threadbundle/config/mutator/populate_current_user.go
Comment threadbundle/config/mutator/populate_current_user_test.go
Comment threadbundle/config/mutator/process_environment_mode.go
Comment threadlibs/sync/path.go
me, err = wsc.CurrentUser.Me(ctx)
if err != nil {
return err
}

@pieternpieternJul 21, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed? The additional nillable arg is a smell.

@lennartkats-dblennartkats-dbJul 24, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to keep it unless we can do something better here :( This API is extremely slow and makes deployments noticeably slower. I wasn't sure at what other level in the code to add some caching for CurrentUser.Me. Ideally, that would be done in the SDK itself.

@lennartkats-dblennartkats-dbJul 24, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can track adding caching to the SDK as a ticket? I added an inline TODO here for now.

@lennartkats-db
lennartkats-dbforce-pushed the extend-deployment-modes-production branch from e0dc4e0 to 6e6ba3dCompareJuly 24, 2023 08:17
pietern
pietern approved these changes Jul 27, 2023
View reviewed changes

@pieternpietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining comment.

Comment threadbundle/config/mutator/process_environment_mode.go
@lennartkats-db
lennartkats-dbforce-pushed the extend-deployment-modes-production branch from 1628ea3 to 1cb7774CompareJuly 30, 2023 07:15
@lennartkats-db
lennartkats-db added this pull request to the merge queueJul 30, 2023
Merged via the queue into main with commit d55652bJul 30, 2023
@lennartkats-db
lennartkats-db deleted the extend-deployment-modes-production branch July 30, 2023 07:25
github-merge-queueBot pushed a commit that referenced this pull request Jul 30, 2023
@lennartkats-db@fjakobs
Add validation for Git settings in bundles (#578)
433f401
## Changes
This checks whether the Git settings are consistent with the actual Git
state of a source directory.
(This PR adds to #577.) Previously, we would silently let users configure their Git branch to
e.g. `main` and deploy with that metadata even if they were actually on
a different branch.
With these changes, the following config would result in an error when
deployed from any other branch than `main`:
```
bundle:
name: example
workspace:
git:
branch: main
environments:
...
```
> not on the right Git branch:
> expected according to configuration: main
> actual: my-feature-branch
It's not very useful to set the same branch for all environments,
though. For development, it's better to just let the CLI auto-detect the
right branch. Therefore, it's now possible to set the branch just for a
single environment:
```
bundle:
name: example 2
environments:
development:
default: true
production:
# production can only be deployed from the 'main' branch
git:
branch: main
```
Adding to that, the `mode: production` option actually checks that users
explicitly set the Git branch as seen above. Setting that branch helps
avoid mistakes, where someone accidentally deploys to production from
the wrong branch. (I could see us offering an escape hatch for that in
the future.)
# Testing
Manual testing to validate the experience and error messages. Automated
unit tests.
---------
Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
pietern added a commit that referenced this pull request Aug 10, 2023
@pietern
Release v0.203.0
1c00d4f
CLI:
* Infer host from profile during `auth login` ([#629](#629)).
Bundles:
* Extend deployment mode support ([#577](#577)).
* Add validation for Git settings in bundles ([#578](#578)).
* Only treat files with .tmpl extension as templates ([#594](#594)).
* Add JSON schema validation for input template parameters ([#598](#598)).
* Add DATABRICKS_BUNDLE_INCLUDE_PATHS to specify include paths through env vars ([#591](#591)).
* Initialise a empty default bundle if BUNDLE_ROOT and DATABRICKS_BUNDLE_INCLUDES env vars are present ([#604](#604)).
* Regenerate bundle resource structs from latest Terraform provider ([#633](#633)).
* Fixed processing jobs libraries with remote path ([#638](#638)).
* Add unit test for file name execution during rendering ([#640](#640)).
* Add bundle init command and support for prompting user for input values ([#631](#631)).
* Fix bundle git branch validation ([#645](#645)).
Internal:
* Fix mkdir integration test on GCP ([#620](#620)).
* Fix git clone integration test for non-existing repo ([#610](#610)).
* Remove push to main trigger for build workflow ([#621](#621)).
* Remove workflow to publish binaries to S3 ([#622](#622)).
* Fix failing fs mkdir test on azure ([#627](#627)).
* Print y/n options when displaying prompts using cmdio.Ask ([#650](#650)).
API Changes:
* Changed `databricks account metastore-assignments create` command to not return anything.
* Added `databricks account network-policy` command group.
OpenAPI commit 7b57ba3a53f4de3d049b6a24391fe5474212daf8 (2023-07-28)
Dependency updates:
* Bump OpenAPI specification & Go SDK Version ([#624](#624)).
* Bump golang.org/x/term from 0.10.0 to 0.11.0 ([#643](#643)).
* Bump golang.org/x/text from 0.11.0 to 0.12.0 ([#642](#642)).
* Bump golang.org/x/oauth2 from 0.10.0 to 0.11.0 ([#641](#641)).
@pieternpietern mentioned this pull request Aug 10, 2023
Merged
github-merge-queueBot pushed a commit that referenced this pull request Aug 10, 2023
@pietern
Release v0.203.0 (#653)
979b680
CLI:
* Infer host from profile during `auth login`
([#629](#629)).
Bundles:
* Extend deployment mode support
([#577](#577)).
* Add validation for Git settings in bundles
([#578](#578)).
* Only treat files with .tmpl extension as templates
([#594](#594)).
* Add JSON schema validation for input template parameters
([#598](#598)).
* Add DATABRICKS_BUNDLE_INCLUDE_PATHS to specify include paths through
env vars ([#591](#591)).
* Initialise a empty default bundle if BUNDLE_ROOT and
DATABRICKS_BUNDLE_INCLUDES env vars are present
([#604](#604)).
* Regenerate bundle resource structs from latest Terraform provider
([#633](#633)).
* Fixed processing jobs libraries with remote path
([#638](#638)).
* Add unit test for file name execution during rendering
([#640](#640)).
* Add bundle init command and support for prompting user for input
values ([#631](#631)).
* Fix bundle git branch validation
([#645](#645)).
Internal:
* Fix mkdir integration test on GCP
([#620](#620)).
* Fix git clone integration test for non-existing repo
([#610](#610)).
* Remove push to main trigger for build workflow
([#621](#621)).
* Remove workflow to publish binaries to S3
([#622](#622)).
* Fix failing fs mkdir test on azure
([#627](#627)).
* Print y/n options when displaying prompts using cmdio.Ask
([#650](#650)).
API Changes:
* Changed `databricks account metastore-assignments create` command to
not return anything.
* Added `databricks account network-policy` command group.
OpenAPI commit 7b57ba3a53f4de3d049b6a24391fe5474212daf8 (2023-07-28)
Dependency updates:
* Bump OpenAPI specification & Go SDK Version
([#624](#624)).
* Bump golang.org/x/term from 0.10.0 to 0.11.0
([#643](#643)).
* Bump golang.org/x/text from 0.11.0 to 0.12.0
([#642](#642)).
* Bump golang.org/x/oauth2 from 0.10.0 to 0.11.0
([#641](#641)).
denik pushed a commit that referenced this pull request May 20, 2026
@lennartkats-db@fjakobs
Extend deployment mode support (#577)
dbfe711
## Changes
This adds `mode: production` option. This mode doesn't do any
transformations but verifies that an environment is configured correctly
for production:
```
environments:
prod:
mode: production
# paths should not be scoped to a user (unless a service principal is used)
root_path: /Shared/non_user_path/...
# run_as and permissions should be set at the resource level (or at the top level when that is implemented)
run_as:
user_name: Alice
permissions:
- level: CAN_MANAGE
user_name: Alice
```
Additionally, this extends the existing `mode: development` option,
* now prefixing deployed assets with `[dev your.user]` instead of just
`[dev`]
* validating that development deployments _are_ scoped to a user
## Related
https://github.com/databricks/cli/pull/578/files (in draft)
## Tests
Manual testing to validate the experience, error messages, and
functionality with all resource types. Automated unit tests.
---------
Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
denik pushed a commit that referenced this pull request May 20, 2026
@lennartkats-db@fjakobs
Add validation for Git settings in bundles (#578)
6626b9c
## Changes
This checks whether the Git settings are consistent with the actual Git
state of a source directory.
(This PR adds to #577.) Previously, we would silently let users configure their Git branch to
e.g. `main` and deploy with that metadata even if they were actually on
a different branch.
With these changes, the following config would result in an error when
deployed from any other branch than `main`:
```
bundle:
name: example
workspace:
git:
branch: main
environments:
...
```
> not on the right Git branch:
> expected according to configuration: main
> actual: my-feature-branch
It's not very useful to set the same branch for all environments,
though. For development, it's better to just let the CLI auto-detect the
right branch. Therefore, it's now possible to set the branch just for a
single environment:
```
bundle:
name: example 2
environments:
development:
default: true
production:
# production can only be deployed from the 'main' branch
git:
branch: main
```
Adding to that, the `mode: production` option actually checks that users
explicitly set the Git branch as seen above. Setting that branch helps
avoid mistakes, where someone accidentally deploys to production from
the wrong branch. (I could see us offering an escape hatch for that in
the future.)
# Testing
Manual testing to validate the experience and error messages. Automated
unit tests.
---------
Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
denik pushed a commit that referenced this pull request May 20, 2026
@pietern
Release v0.203.0 (#653)
11ab78b
CLI:
* Infer host from profile during `auth login`
([#629](#629)).
Bundles:
* Extend deployment mode support
([#577](#577)).
* Add validation for Git settings in bundles
([#578](#578)).
* Only treat files with .tmpl extension as templates
([#594](#594)).
* Add JSON schema validation for input template parameters
([#598](#598)).
* Add DATABRICKS_BUNDLE_INCLUDE_PATHS to specify include paths through
env vars ([#591](#591)).
* Initialise a empty default bundle if BUNDLE_ROOT and
DATABRICKS_BUNDLE_INCLUDES env vars are present
([#604](#604)).
* Regenerate bundle resource structs from latest Terraform provider
([#633](#633)).
* Fixed processing jobs libraries with remote path
([#638](#638)).
* Add unit test for file name execution during rendering
([#640](#640)).
* Add bundle init command and support for prompting user for input
values ([#631](#631)).
* Fix bundle git branch validation
([#645](#645)).
Internal:
* Fix mkdir integration test on GCP
([#620](#620)).
* Fix git clone integration test for non-existing repo
([#610](#610)).
* Remove push to main trigger for build workflow
([#621](#621)).
* Remove workflow to publish binaries to S3
([#622](#622)).
* Fix failing fs mkdir test on azure
([#627](#627)).
* Print y/n options when displaying prompts using cmdio.Ask
([#650](#650)).
API Changes:
* Changed `databricks account metastore-assignments create` command to
not return anything.
* Added `databricks account network-policy` command group.
OpenAPI commit 7b57ba3a53f4de3d049b6a24391fe5474212daf8 (2023-07-28)
Dependency updates:
* Bump OpenAPI specification & Go SDK Version
([#624](#624)).
* Bump golang.org/x/term from 0.10.0 to 0.11.0
([#643](#643)).
* Bump golang.org/x/text from 0.11.0 to 0.12.0
([#642](#642)).
* Bump golang.org/x/oauth2 from 0.10.0 to 0.11.0
([#641](#641)).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pieternpieternpietern approved these changes
@fjakobsfjakobsAwaiting requested review from fjakobs

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lennartkats-db@pietern@fjakobs