Add support for cloning repositories by shreyas-goenka · Pull Request #544 · databricks/cli

shreyas-goenka · 2023-07-04T16:04:34Z

Changes

Adds support for cloning public and private github repositories for databricks templates

Tests

Integration tests

shreyas-goenka · 2023-07-04T16:08:18Z

This PR is a replacement for #534

shreyas-goenka · 2023-07-04T16:09:18Z

Please leave any feedback you have, if this looks good, then I'll proceed to add some unit tests for the arg parsing and command construction

pietern · 2023-07-18T12:00:08Z

+	// case: invalid repository references
+	assert.Equal(t, "https://github.com/databricks/", expandUrl(""))
+	assert.Equal(t, "https://github.com//", expandUrl("/"))
+	assert.Equal(t, "https://github.com//abc/def", expandUrl("/abc/def"))


These should return errors.

removed the expand url function

It's complex to have a simple and robust way to parse these URLs, across different protocols, git providers and operating systems. I decided to just support the repository name and the full URL to keep it simpler, with a simple rubrik, that ^[\w-\.]+$ without any / will be resolved against databricks
(see: https://stackoverflow.com/questions/59081778/rules-for-special-characters-in-github-repository-name)

Note ^[\w-\.]+/[\w-\.]+$ is likely a valid regex that can be used as a capturing org name and repository name, but that can be added in as a followup

pietern · 2023-07-25T11:18:33Z

+	return args
+}
+
+func Clone(ctx context.Context, url, reference, targetPath string) error {


The argument suggests reference but the function above and test suggest it can only be a branch.
Which one applies?

The git clone --branch flag can take in both tags and branches.
source: https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---branchltnamegt

Breaking Change: * Require include glob patterns to be explicitly defined ([#602](#602)). Bundles: * Add support for more SDK config options ([#587](#587)). * Add template renderer for Databricks templates ([#589](#589)). * Fix formatting in renderer.go ([#593](#593)). * Fixed python wheel test ([#608](#608)). * Auto detect Python wheel packages and infer build command ([#603](#603)). * Added support for artifacts building for bundles ([#583](#583)). * Add support for cloning repositories ([#544](#544)). * Add regexp compile helper function for templates ([#601](#601)). * Add unit test that raw strings are printed as is ([#599](#599)). Internal: * Fix tests under ./cmd/configure if DATABRICKS_TOKEN is set ([#605](#605)). * Remove dependency on global state in generated commands ([#595](#595)). * Remove dependency on global state for the root command ([#606](#606)). * Add merge_group trigger for build ([#612](#612)). * Added support for build command chaining and error on missing wheel ([#607](#607)). * Add TestAcc prefix to filer test and fix any failing tests ([#611](#611)). * Add url parse helper function for templates ([#600](#600)). * Remove dependency on global state for remaining commands ([#613](#613)). * Update CHANGELOG template ([#588](#588)).

## Changes Adds support for cloning public and private github repositories for databricks templates ## Tests Integration tests

Breaking Change: * Require include glob patterns to be explicitly defined ([#602](#602)). Bundles: * Add support for more SDK config options ([#587](#587)). * Add template renderer for Databricks templates ([#589](#589)). * Fix formatting in renderer.go ([#593](#593)). * Fixed python wheel test ([#608](#608)). * Auto detect Python wheel packages and infer build command ([#603](#603)). * Added support for artifacts building for bundles ([#583](#583)). * Add support for cloning repositories ([#544](#544)). * Add regexp compile helper function for templates ([#601](#601)). * Add unit test that raw strings are printed as is ([#599](#599)). Internal: * Fix tests under ./cmd/configure if DATABRICKS_TOKEN is set ([#605](#605)). * Remove dependency on global state in generated commands ([#595](#595)). * Remove dependency on global state for the root command ([#606](#606)). * Add merge_group trigger for build ([#612](#612)). * Added support for build command chaining and error on missing wheel ([#607](#607)). * Add TestAcc prefix to filer test and fix any failing tests ([#611](#611)). * Add url parse helper function for templates ([#600](#600)). * Remove dependency on global state for remaining commands ([#613](#613)). * Update CHANGELOG template ([#588](#588)).

shreyas-goenka added 2 commits July 4, 2023 18:03

Add support for cloning repositories
d55cba2

-
53bf15f

shreyas-goenka mentioned this pull request Jul 4, 2023
Add support for cloning repositories from github #534
Closed

shreyas-goenka requested review from andrewnester and pietern July 4, 2023 16:07

shreyas-goenka changed the title ~~Add support for cloning repositories~~[DRAFT] Add support for cloning repositoriesJul 4, 2023

pietern reviewed Jul 5, 2023
View reviewed changes

Comment threadinternal/git_clone_test.go Outdated
Comment threadlibs/git/clone.go Outdated
Comment threadlibs/cmdio/io.go Outdated
Comment threadlibs/git/clone.go Outdated
Comment threadlibs/git/clone.go Outdated
Comment threadlibs/git/clone.go Outdated
Comment threadinternal/git_clone_test.go

shreyas-goenka added 13 commits July 6, 2023 15:49

removed redirection for stdin and stdout
cf59210

updated tests to use smaller repos
4957cda

-
ad37e18

fix err
32b1eec

make variables privatE
5552c7d

added unit tests for git url construction
38b3de5

added test for git cli not existing
fdf30e7

added test for repository does not exist
a3357e2

-
cf71e65

Merge remote-tracking branch 'origin' into git-clone-clone
2142030

Merge remote-tracking branch 'origin' into git-clone-clone
feb065a

removed @ syntax support
77da327

Merge remote-tracking branch 'origin' into git-clone-clone
d765198

shreyas-goenka changed the title ~~[DRAFT] Add support for cloning repositories~~Add support for cloning repositoriesJul 12, 2023

shreyas-goenka requested a review from pietern July 12, 2023 15:51

andrewnester reviewed Jul 13, 2023
View reviewed changes

Comment threadlibs/git/clone.go Outdated
Comment threadlibs/git/clone.go
Comment threadinternal/git_clone_test.go Outdated

Added test for databricks internal clone too
28b3079

pietern reviewed Jul 18, 2023
View reviewed changes

shreyas-goenka added 3 commits July 23, 2023 19:20

Merge remote-tracking branch 'origin' into git-clone-clone
0fceb22

remove support for org/repo
4f9d162

add regexp check
38ee5b0

removed unnecessary / check
ca1cf7e

shreyas-goenka requested review from andrewnester and pietern July 23, 2023 21:08

andrewnester approved these changes Jul 25, 2023
View reviewed changes

pietern approved these changes Jul 25, 2023
View reviewed changes

shreyas-goenka added 2 commits July 25, 2023 13:25

Merge branch 'main' into git-clone-clone
2e2b705

Merge branch 'main' into git-clone-clone
c547425

shreyas-goenka merged commit 8fdc0fe into mainJul 25, 2023

shreyas-goenka deleted the git-clone-clone branch July 25, 2023 13:36

mgyucht mentioned this pull request Jul 27, 2023
Release v0.202.0 #619
Merged

denik pushed a commit that referenced this pull request May 20, 2026
Add support for cloning repositories (#544)
0026cf0
## Changes Adds support for cloning public and private github repositories for databricks templates ## Tests Integration tests

Uh oh!

Conversation

shreyas-goenka commented Jul 4, 2023

Changes

Tests

Uh oh!

shreyas-goenka commented Jul 4, 2023

Uh oh!

shreyas-goenka commented Jul 4, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pieternJul 18, 2023

Choose a reason for hiding this comment

Uh oh!

shreyas-goenkaJul 23, 2023

Choose a reason for hiding this comment

Uh oh!

shreyas-goenkaJul 23, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenkaJul 23, 2023

Choose a reason for hiding this comment

Uh oh!

pieternJul 25, 2023

Choose a reason for hiding this comment

Uh oh!

shreyas-goenkaJul 25, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shreyas-goenkaJul 23, 2023•
edited
Loading

shreyas-goenkaJul 25, 2023•
edited
Loading