Add workspace import_dir command to the CLI by shreyas-goenka · Pull Request #418 · databricks/cli

shreyas-goenka · 2023-06-01T09:24:03Z

TODO: add integration tests for functionality and progress events

shreyas-goenka · 2023-06-01T09:24:59Z

outputs:

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace import_dir . /Users/shreyas.goenka@databricks.com/new-vandaley
Import started
Error: Request failed for POST /Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/c.py: Path (/Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/c) already exists.

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace import_dir . /Users/shreyas.goenka@databricks.com/new-vandaley --overwrite
Import started
Uploaded foo/c.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/foo/c
Uploaded foo/b.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/foo/b
Uploaded exported/foo/b.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/b
Uploaded exported/foo/c.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/c
Uploaded myDir/b.txt -> /Users/shreyas.goenka@databricks.com/new-vandaley/myDir/b.txt
Uploaded exported/my-notebook.py.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/exported/my-notebook.py
Uploaded .gitignore -> /Users/shreyas.goenka@databricks.com/new-vandaley/.gitignore
Uploaded myDir/a.txt -> /Users/shreyas.goenka@databricks.com/new-vandaley/myDir/a.txt
Import completed

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace import_dir . /Users/shreyas.goenka@databricks.com/new-vandaley --overwrite --output json
{
"source_path":".",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley",
"type":"IMPORT_STARTED"
}
{
"source_path":"foo/b.py",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/foo/b",
"type":"UPLOAD_COMPLETE"
}
{
"source_path":"exported/my-notebook.py.py",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/exported/my-notebook.py",
"type":"UPLOAD_COMPLETE"
}
{
"source_path":"exported/foo/c.py",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/c",
"type":"UPLOAD_COMPLETE"
}
{
"source_path":"exported/foo/b.py",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/b",
"type":"UPLOAD_COMPLETE"
}
{
"source_path":"myDir/b.txt",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/myDir/b.txt",
"type":"UPLOAD_COMPLETE"
}
{
"source_path":"sync-snapshots/0bf79f861fa32456.json",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/sync-snapshots/0bf79f861fa32456.json",
"type":"UPLOAD_COMPLETE"
}
{
"source_path":".gitignore",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/.gitignore",
"type":"UPLOAD_COMPLETE"
}
{
"source_path":"foo/c.py",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/foo/c",
"type":"UPLOAD_COMPLETE"
}
{
"source_path":"myDir/a.txt",
"target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/myDir/a.txt",
"type":"UPLOAD_COMPLETE"
}
{
"type":"IMPORT_COMPLETE"
}

pietern · 2023-06-01T09:59:36Z

+
+ // Initialize error wait group, and spawn the progress event emitter inside
+ // the error wait group
+ group, ctx := errgroup.WithContext(ctx)


What's the behavior of the returned context?

The returned context is cancelled if there's an error in any of the subroutines. see: https://pkg.go.dev/golang.org/x/sync/errgroup#WithContext

pietern · 2023-06-01T10:10:30Z

+	if err != nil {
+ return err
+	}
+	_, err = w.Write([]byte("\n"))


Can you split this into a separate PR? Might affect other functionality.

Done in this PR: #443

pietern · 2023-06-01T10:12:12Z

+ return "", fmt.Errorf("could not find remote path for %s", localPath)
+	}
+	return s.repoFiles.RemotePath(relativePath)
+}


Can you add a comment here why this is a public function? Also this needs a mutex because the snapshot code is accessing it from a separate goroutine.

A better approach is to include the remote name in the events, then you don't need this function or the mutex.

Yeah, having this in the events is then you have to use reflection to read the remote name values, which is OK maybe

The type switch is there already right?

Ah right, using type assertions should be enough here

shreyas-goenka · 2023-06-06T14:50:16Z

Closing because we decided to move forward with just a simple walk dir

Add workspace import_dir command to the CLI
9f4bf12

shreyas-goenka requested review from andrewnester, nfx and pietern June 1, 2023 09:25

Merge remote-tracking branch 'origin' into import_dir
1244b3a

pietern reviewed Jun 1, 2023
View reviewed changes

shreyas-goenka added 23 commits June 2, 2023 00:04

sync options redefine better
2303f20

-
0382c74

-
289dd0d

helper wsfs function
fb5ea16

add check for isdir in readdir
931fae1

added some intgegration test for repofiles and moved to filer
57a9578

repofiles integration test
dc0ff89

wip import-dir acc tests
2be6b85

testdata
c1ccf82

-
efc72b5

Merge remote-tracking branch 'origin' into import_dir
7099874

remove underscore
0f1cb67

set template in annotation
49f7969

undo remote path public
298fed2

address some comments
6cc2571

-
bb17546

-
ae09abf

-
14b63d0

-
5fdd8a2

-
9629032

removes not a directory error
e9a309e

-
74cb87f

Merge remote-tracking branch 'origin' into import_dir
59081f7

shreyas-goenka added 6 commits June 6, 2023 01:01

some cleanup
58acf64

Merge remote-tracking branch 'origin' into import_dir
f1e99d3

Merge remote-tracking branch 'origin' into import_dir
764d9e9

revert changes to reposfile
8166aed

-
f0f75ff

made events work and moved validation outside sync object
43aadba

shreyas-goenka closed this Jun 6, 2023

Uh oh!

Conversation

shreyas-goenka commented Jun 1, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shreyas-goenka commented Jun 1, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka commented Jun 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shreyas-goenka commented Jun 1, 2023•
edited
Loading