Add workspace export-dir command

[shreyas-goenka]

Changes

This PR:

1. Adds the export-dir command
2. Changes filer.Read to return an error if a user tries to read a directory
3. Adds returning internal file structures from filer.Stat().Sys()

Tests

Integration tests and manually

[shreyas-goenka]

Output:

case 1: command is run, directory foo does not exist

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace export-dir /Users/me@example.com/december foo
Export started. Download files from /Users/me@example.com/december
Downloaded /Users/me@example.com/december/.gitignore -> foo/.gitignore
Downloaded /Users/me@example.com/december/apple -> foo/apple
Downloaded /Users/me@example.com/december/hello_world -> foo/hello_world.py
Downloaded /Users/me@example.com/december/mango -> foo/mango
Downloaded /Users/me@example.com/december/veggies/tomatoes -> foo/veggies/tomatoes
Export complete. Files can be found at foo

case 2: foo exists, with a couple of files in it.

shreyas.goenka@THW32HFW6T vandaley-industries % rm foo/apple foo/mango
shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace export-dir /Users/me@example.com/december foo
Export started. Download files from /Users/me@example.com/december
Skipping /Users/me@example.com/december/.gitignore because foo/.gitignore already exists
Downloaded /Users/me@example.com/december/apple -> foo/apple
Skipping /Users/me@example.com/december/hello_world because foo/hello_world.py already exists
Downloaded /Users/me@example.com/december/mango -> foo/mango
Skipping /Users/me@example.com/december/veggies/tomatoes because foo/veggies/tomatoes already exists
Export complete. Files can be found at foo

case 3: foo exists, with a couple of files in it. We use the overwrite flag

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace export-dir /Users/me@example.com/december foo --overwrite
Export started. Download files from /Users/me@example.com/december
Downloaded /Users/me@example.com/december/.gitignore -> foo/.gitignore
Downloaded /Users/me@example.com/december/apple -> foo/apple
Downloaded /Users/me@example.com/december/hello_world -> foo/hello_world.py
Downloaded /Users/me@example.com/december/mango -> foo/mango
Downloaded /Users/me@example.com/december/veggies/tomatoes -> foo/veggies/tomatoes
Export complete. Files can be found at foo

case 4: output with the json mode enabled

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace export-dir /Users/me@example.com/december foo --overwrite --output=json
{
"source_path":"/Users/me@example.com/december",
"type":"EXPORT_STARTED"
}{
"source_path":"/Users/me@example.com/december/.gitignore",
"target_path":"foo/.gitignore",
"type":"DOWNLOAD_COMPLETE"
}{
"source_path":"/Users/me@example.com/december/apple",
"target_path":"foo/apple",
"type":"DOWNLOAD_COMPLETE"
}{
"source_path":"/Users/me@example.com/december/hello_world",
"target_path":"foo/hello_world.py",
"type":"DOWNLOAD_COMPLETE"
}{
"source_path":"/Users/me@example.com/december/mango",
"target_path":"foo/mango",
"type":"DOWNLOAD_COMPLETE"
}{
"source_path":"/Users/me@example.com/december/veggies/tomatoes",
"target_path":"foo/veggies/tomatoes",
"type":"DOWNLOAD_COMPLETE"
}{
"target_path":"foo",
"type":"EXPORT_COMPLETED"
}%

[shreyas-goenka]

github is experiencing an outage, so my latest commits are not in, but they are lint fixes, todo cleanups and making the test prefixes TestAcc

[pietern]

Fixed in databricks/databricks-sdk-go#415

I would prefer not adding a stat call for every read if we can avoid it.

[shreyas-goenka]

I have added a todo with a follow-up issue filed. I would rather not block on a go SDK release or remove test coverage

[pietern]

This means doing 2 stat calls for every object as is.

[shreyas-goenka]

Well, I think this is OK if a few assumptions hold true:

1. Stat API calls do not overload webapp
2. Latency added is reasonable

The reason I am fine with it, is this should be solved when we have the Files API since it is supposed to be a "DUMB" filesystem (no fancy extension stripping, service throws an error when we try to read a directory)

Right now it's us handing backend problems on the client side causing excessive API calls

[pietern]

This looks great -- comments are minor.