Add bundle init command and support for prompting user for input values by shreyas-goenka · Pull Request #631 · databricks/cli

shreyas-goenka · 2023-08-02T14:30:11Z

Changes

This PR adds two features:

The bundle init command
Support for prompting for input values

In order to do this, this PR also introduces a new config struct which handles reading config files, prompting users and all validation steps before we materialize the template

With this PR users can start authoring custom templates, based on go text templates, for their projects / orgs.

Tests

Unit tests, both existing and new

shreyas-goenka · 2023-08-02T14:43:24Z

How the prompting logic looks like. Here 1.1 is the default value for A:

Screen.Recording.2023-08-02.at.4.42.17.PM.mov

shreyas-goenka · 2023-08-03T09:51:12Z

End to end flow works, with downloading the repo from github. tested with this repo: https://github.com/databricks/bundle-template-samples

CLI: * Infer host from profile during `auth login` ([#629](#629)). Bundles: * Extend deployment mode support ([#577](#577)). * Add validation for Git settings in bundles ([#578](#578)). * Only treat files with .tmpl extension as templates ([#594](#594)). * Add JSON schema validation for input template parameters ([#598](#598)). * Add DATABRICKS_BUNDLE_INCLUDE_PATHS to specify include paths through env vars ([#591](#591)). * Initialise a empty default bundle if BUNDLE_ROOT and DATABRICKS_BUNDLE_INCLUDES env vars are present ([#604](#604)). * Regenerate bundle resource structs from latest Terraform provider ([#633](#633)). * Fixed processing jobs libraries with remote path ([#638](#638)). * Add unit test for file name execution during rendering ([#640](#640)). * Add bundle init command and support for prompting user for input values ([#631](#631)). * Fix bundle git branch validation ([#645](#645)). Internal: * Fix mkdir integration test on GCP ([#620](#620)). * Fix git clone integration test for non-existing repo ([#610](#610)). * Remove push to main trigger for build workflow ([#621](#621)). * Remove workflow to publish binaries to S3 ([#622](#622)). * Fix failing fs mkdir test on azure ([#627](#627)). * Print y/n options when displaying prompts using cmdio.Ask ([#650](#650)). API Changes: * Changed `databricks account metastore-assignments create` command to not return anything. * Added `databricks account network-policy` command group. OpenAPI commit 7b57ba3a53f4de3d049b6a24391fe5474212daf8 (2023-07-28) Dependency updates: * Bump OpenAPI specification & Go SDK Version ([#624](#624)). * Bump golang.org/x/term from 0.10.0 to 0.11.0 ([#643](#643)). * Bump golang.org/x/text from 0.11.0 to 0.12.0 ([#642](#642)). * Bump golang.org/x/oauth2 from 0.10.0 to 0.11.0 ([#641](#641)).

pietern · 2023-08-10T12:20:33Z

+	var projectDir string
+	cmd.Flags().StringVar(&configFile, "config-file", "", "File containing input parameters for template initialization.")
+	cmd.Flags().StringVar(&projectDir, "project-dir", "", "The project will be initialized in this directory.")
+	cmd.MarkFlagRequired("project-dir")


I thought we would use the current working directory by default.

I wanted to start out with a minimal implementation. We can add the default as a followup. Agreed though that the current directory makes sense.

pietern · 2023-08-10T12:24:37Z

+ // Download the template in a temporary directory
+ tmpDir := os.TempDir()
+ templateURL := templatePath
+ templateDir := filepath.Join(tmpDir, repoName(templateURL))


Why does this matter if it is a temporary directory?

Just an implementation detail. I figured temp dir would be the best place to download the repo, and then delete it once its served its purpose for now.

pietern · 2023-08-10T12:26:45Z

+ return fmt.Errorf("failed to cast value %v of property %s from file %s to an integer: %w", floatVal, name, path, err)
+ }
+ configFromFile[name] = v
+	}


This transformation would need to be done for any structure for which we have a JSON schema right?

Maybe. If a custom json unmarshaller we implement treats integer valued numbers as an int when type is ambigious, then we would not need this. Ofcourse doing this would have other side effects.

pietern · 2023-08-10T12:27:25Z

+	}
+
+	// Write configs from the file to the input map, not overwriting any existing
+	// configurations.


What is the rationale for not overwriting existing values? Why would we have existing values?

In the future we will add a --parameters flag which has a higher priority than the config-file. Those values would the existing ones mentioned here.

…es (#631) ## Changes This PR adds two features: 1. The bundle init command 2. Support for prompting for input values In order to do this, this PR also introduces a new `config` struct which handles reading config files, prompting users and all validation steps before we materialize the template With this PR users can start authoring custom templates, based on go text templates, for their projects / orgs. ## Tests Unit tests, both existing and new

CLI: * Infer host from profile during `auth login` ([#629](#629)). Bundles: * Extend deployment mode support ([#577](#577)). * Add validation for Git settings in bundles ([#578](#578)). * Only treat files with .tmpl extension as templates ([#594](#594)). * Add JSON schema validation for input template parameters ([#598](#598)). * Add DATABRICKS_BUNDLE_INCLUDE_PATHS to specify include paths through env vars ([#591](#591)). * Initialise a empty default bundle if BUNDLE_ROOT and DATABRICKS_BUNDLE_INCLUDES env vars are present ([#604](#604)). * Regenerate bundle resource structs from latest Terraform provider ([#633](#633)). * Fixed processing jobs libraries with remote path ([#638](#638)). * Add unit test for file name execution during rendering ([#640](#640)). * Add bundle init command and support for prompting user for input values ([#631](#631)). * Fix bundle git branch validation ([#645](#645)). Internal: * Fix mkdir integration test on GCP ([#620](#620)). * Fix git clone integration test for non-existing repo ([#610](#610)). * Remove push to main trigger for build workflow ([#621](#621)). * Remove workflow to publish binaries to S3 ([#622](#622)). * Fix failing fs mkdir test on azure ([#627](#627)). * Print y/n options when displaying prompts using cmdio.Ask ([#650](#650)). API Changes: * Changed `databricks account metastore-assignments create` command to not return anything. * Added `databricks account network-policy` command group. OpenAPI commit 7b57ba3a53f4de3d049b6a24391fe5474212daf8 (2023-07-28) Dependency updates: * Bump OpenAPI specification & Go SDK Version ([#624](#624)). * Bump golang.org/x/term from 0.10.0 to 0.11.0 ([#643](#643)). * Bump golang.org/x/text from 0.11.0 to 0.12.0 ([#642](#642)). * Bump golang.org/x/oauth2 from 0.10.0 to 0.11.0 ([#641](#641)).

shreyas-goenka added 2 commits August 2, 2023 16:28

Add bundle init command and support for prompting user for input values
ca4a68a

Merge remote-tracking branch 'origin' into add-prompt-and-init-cmd
52c749e

shreyas-goenka changed the title ~~Add bundle init command and support for prompting user for input values~~[DRAFT] Add bundle init command and support for prompting user for input valuesAug 2, 2023

shreyas-goenka requested a review from andrewnester August 2, 2023 14:39

shreyas-goenka added 4 commits August 2, 2023 18:21

added some comments and todos
418ab6c

added unit tests for utils
3173c1f

lint
c3d4684

Merge remote-tracking branch 'origin' into add-prompt-and-init-cmd
c8734a5

cleared todos and added some coverage for the validation function
0824d04

shreyas-goenka changed the title ~~[DRAFT] Add bundle init command and support for prompting user for input values~~Add bundle init command and support for prompting user for input valuesAug 3, 2023

config-file is not required
1d6156a

andrewnester reviewed Aug 3, 2023
View reviewed changes

shreyas-goenka added 11 commits August 3, 2023 15:33

use repo name as the dir name for temp download of tmeplate
e03cc74

use int64
ab80b65

add test for large integers
bbdb4df

remove integers as valid values for numbers
e1a7f62

added test cases
49a0062

Merge remote-tracking branch 'origin' into add-prompt-and-init-cmd
03553f7

make tests more cross platform by lowering the int number sizes
4e934cb

-
94c10c8

final bits of cleanup and testing
d6779d5

-
67827df

-
59647d0

shreyas-goenka requested a review from andrewnester August 3, 2023 20:03

cosmetic
db149ee

shreyas-goenka requested a review from pietern August 7, 2023 08:40

andrewnester reviewed Aug 7, 2023
View reviewed changes

Comment threadcmd/bundle/init.go
Comment threadcmd/bundle/init_test.go

added test for invalid URLs
eed1566

-
b101ec2

andrewnester approved these changes Aug 7, 2023
View reviewed changes

shreyas-goenka added this pull request to the merge queue Aug 7, 2023

Merged via the queue into main with commit 81ee031Aug 7, 2023

shreyas-goenka deleted the add-prompt-and-init-cmd branch August 7, 2023 13:23

pietern mentioned this pull request Aug 10, 2023
Release v0.203.0 #653
Merged

pietern reviewed Aug 10, 2023
View reviewed changes

Uh oh!

Conversation

shreyas-goenka commented Aug 2, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

shreyas-goenka commented Aug 2, 2023

Uh oh!

shreyas-goenka commented Aug 3, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternAug 10, 2023

Choose a reason for hiding this comment

Uh oh!

shreyas-goenkaAug 10, 2023

Choose a reason for hiding this comment

Uh oh!

pieternAug 10, 2023

Choose a reason for hiding this comment

Uh oh!

shreyas-goenkaAug 10, 2023

Choose a reason for hiding this comment

Uh oh!

pieternAug 10, 2023

Choose a reason for hiding this comment

Uh oh!

shreyas-goenkaAug 10, 2023

Choose a reason for hiding this comment

Uh oh!

pieternAug 10, 2023

Choose a reason for hiding this comment

Uh oh!

shreyas-goenkaAug 10, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shreyas-goenka commented Aug 2, 2023•
edited
Loading

shreyas-goenka commented Aug 3, 2023•
edited
Loading

shreyas-goenkaAug 10, 2023•
edited
Loading