Add fs ls command for dbfs by shreyas-goenka · Pull Request #429 · databricks/cli

shreyas-goenka · 2023-06-02T15:24:16Z

Changes

Adds fs ls command
Adds ability to define multiple templates

Tests

Manually and integration tests

shreyas-goenka · 2023-06-04T23:15:07Z

shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore -l --output=json
[
{
"name":"babynames.csv",
"is_directory":false,
"size":278154,
"last_modified":"2023-03-05T16:27:11+01:00"
},
{
"name":"jars",
"is_directory":true,
"size":0,
"last_modified":"2022-10-03T08:18:02+02:00"
},
{
"name":"tables",
"is_directory":true,
"size":0,
"last_modified":"2022-06-23T15:45:57+02:00"
},
{
"name":"terraform-test",
"is_directory":true,
"size":0,
"last_modified":"2023-03-26T20:18:06+02:00"
}
]%

shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore -l
FILE 278154 2023-03-05T15:27:11Z babynames.csv
DIRECTORY 0 2022-10-03T06:18:02Z jars
DIRECTORY 0 2022-06-23T13:45:57Z tables
DIRECTORY 0 2023-03-26T18:18:06Z terraform-test

shreyas.goenka@THW32HFW6T bricks % cli fs ls dbfs:/FileStore
babynames.csv
jars
tables
terraform-test

shreyas-goenka · 2023-06-04T23:22:57Z

tests run: https://github.com/databricks/eng-dev-ecosystem/actions/runs/5171723068

andrewnester · 2023-06-05T07:56:13Z

+	ModTime time.Time `json:"last_modified"`
+}
+
+func toLsOutput(f fs.DirEntry) (*lsOutput, error) {


I would make it a method of fs.DirEntry instead, for better readability

We cannot define methods on structs defined in external packages right?

pietern

The change modifying cmdio to use a map of templates doesn't seem necessary. Please revert that and stick to calling RenderWithTemplate with an actual template.

pietern · 2023-06-05T12:45:42Z

+ {{end}}
+ `))
+ }
+ return cmdio.Render(ctx, jsonDirEntries)


Please specify the template here as well. Now this command it taking two different approaches for rendering a template based on the mode which is weird to the uninitiated reader.

Is there a significance for storing templates in the annotations in the first place?
cc: @nfx

It has to be done for autogenerated commands because assignment of the annotations field happens in a different file from where the command is defined. Means a stable way to associate metadata with the command instance.

pietern

LGTM, final comment.

Thanks!

pietern · 2023-06-05T14:17:23Z

@@ -0,0 +1,104 @@
+package internal


Could you name this file fs_ls_test.go to disambiguate from (potential) other ls tests?

What are the other potential ls tests possible? did you have anything specific in mind?

workspace fs

shreyas-goenka · 2023-06-05T15:21:53Z

integration test run: https://github.com/databricks/eng-dev-ecosystem/actions/runs/5178919118

## Changes CLI: * Add directory tracking to sync ([#425](#425)). * Add fs cat command for dbfs files ([#430](#430)). * Add fs ls command for dbfs ([#429](#429)). * Add fs mkdirs command for dbfs ([#432](#432)). * Add fs rm command for dbfs ([#433](#433)). * Add installation instructions ([#458](#458)). * Add new line to cmdio JSON rendering ([#443](#443)). * Add profile on `databricks auth login` ([#423](#423)). * Add readable console logger ([#370](#370)). * Add workspace export-dir command ([#449](#449)). * Added secrets input prompt for secrets put-secret command ([#413](#413)). * Added spinner when loading command prompts ([#420](#420)). * Better error message if can not load prompts ([#437](#437)). * Changed service template to correctly handle required positional arguments ([#405](#405)). * Do not generate prompts for certain commands ([#438](#438)). * Do not prompt for List methods ([#411](#411)). * Do not use FgWhite and FgBlack for terminal output ([#435](#435)). * Skip path translation of job task for jobs with a Git source ([#404](#404)). * Tweak profile prompt ([#454](#454)). * Update with the latest Go SDK ([#457](#457)). * Use cmdio in version command for `--output` flag ([#419](#419)). Bundles: * Check for nil environment before accessing it ([#453](#453)). Dependencies: * Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0 ([#459](#459)). * Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19 ([#412](#412)). Internal: * Add Mkdir and ReadDir functions to filer.Filer interface ([#414](#414)). * Add Stat function to filer.Filer interface ([#421](#421)). * Add check for path is a directory in filer.ReadDir ([#426](#426)). * Add fs.FS adapter for the filer interface ([#422](#422)). * Add implementation of filer.Filer for local filesystem ([#460](#460)). * Allow equivalence checking of filer errors to fs errors ([#416](#416)). * Fix locker integration test ([#417](#417)). * Implement DBFS filer ([#139](#139)). * Include recursive deletion in filer interface ([#442](#442)). * Make filer.Filer return fs.DirEntry from ReadDir ([#415](#415)). * Speed up sync integration tests ([#428](#428)).

## Changes 1. Adds fs ls command 2. Adds ability to define multiple templates ## Tests Manually and integration tests

## Changes CLI: * Add directory tracking to sync ([#425](#425)). * Add fs cat command for dbfs files ([#430](#430)). * Add fs ls command for dbfs ([#429](#429)). * Add fs mkdirs command for dbfs ([#432](#432)). * Add fs rm command for dbfs ([#433](#433)). * Add installation instructions ([#458](#458)). * Add new line to cmdio JSON rendering ([#443](#443)). * Add profile on `databricks auth login` ([#423](#423)). * Add readable console logger ([#370](#370)). * Add workspace export-dir command ([#449](#449)). * Added secrets input prompt for secrets put-secret command ([#413](#413)). * Added spinner when loading command prompts ([#420](#420)). * Better error message if can not load prompts ([#437](#437)). * Changed service template to correctly handle required positional arguments ([#405](#405)). * Do not generate prompts for certain commands ([#438](#438)). * Do not prompt for List methods ([#411](#411)). * Do not use FgWhite and FgBlack for terminal output ([#435](#435)). * Skip path translation of job task for jobs with a Git source ([#404](#404)). * Tweak profile prompt ([#454](#454)). * Update with the latest Go SDK ([#457](#457)). * Use cmdio in version command for `--output` flag ([#419](#419)). Bundles: * Check for nil environment before accessing it ([#453](#453)). Dependencies: * Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0 ([#459](#459)). * Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19 ([#412](#412)). Internal: * Add Mkdir and ReadDir functions to filer.Filer interface ([#414](#414)). * Add Stat function to filer.Filer interface ([#421](#421)). * Add check for path is a directory in filer.ReadDir ([#426](#426)). * Add fs.FS adapter for the filer interface ([#422](#422)). * Add implementation of filer.Filer for local filesystem ([#460](#460)). * Allow equivalence checking of filer errors to fs errors ([#416](#416)). * Fix locker integration test ([#417](#417)). * Implement DBFS filer ([#139](#139)). * Include recursive deletion in filer interface ([#442](#442)). * Make filer.Filer return fs.DirEntry from ReadDir ([#415](#415)). * Speed up sync integration tests ([#428](#428)).

shreyas-goenka added 3 commits June 2, 2023 17:24

Add fs ls command for dbfs
b5f5f11

added integration tests
a4879c0

lint
2706beb

shreyas-goenka requested review from andrewnester and pietern June 4, 2023 23:16

shreyas-goenka added 2 commits June 5, 2023 01:20

added test for ls on flie
6b667b5

Merge remote-tracking branch 'origin' into fs-ls
db953ac

shreyas-goenka added 2 commits June 5, 2023 01:34

added resolvedbfs path func
492382c

manually check for dbfs prefix
3dc4770

andrewnester reviewed Jun 5, 2023
View reviewed changes

Comment threadcmd/fs/ls.go Outdated

andrewnester reviewed Jun 5, 2023
View reviewed changes

Comment threadcmd/fs/ls.go Outdated

andrewnester reviewed Jun 5, 2023
View reviewed changes

Comment threadcmd/fs/ls.go Outdated

andrewnester requested changes Jun 5, 2023
View reviewed changes

pietern reviewed Jun 5, 2023
View reviewed changes

Comment threadlibs/cmdio/render.go Outdated
Comment threadcmd/fs/ls.go Outdated
Comment threadlibs/filer/dbfs_client.go Outdated
Comment threadcmd/fs/ls_output.go Outdated

pietern mentioned this pull request Jun 5, 2023
Add databricks fs ls command for WSFS #408
Closed

shreyas-goenka added 4 commits June 5, 2023 13:49

comments
19a2051

move sort outside
8050dc5

-
fc7fbbe

address comments
b048c86

pietern reviewed Jun 5, 2023
View reviewed changes

shreyas-goenka added 2 commits June 5, 2023 15:07

address comments 2
265e517

added preallocation of size
0882609

shreyas-goenka requested review from andrewnester and pietern June 5, 2023 13:10

initialize dbfs client at root
9160a6e

pietern approved these changes Jun 5, 2023
View reviewed changes

shreyas-goenka added 2 commits June 5, 2023 17:18

Merge remote-tracking branch 'origin' into fs-ls
503f727

-
993a545

Merge branch 'main' into fs-ls
e68aa27

shreyas-goenka merged commit 6ff0012 into mainJun 5, 2023

shreyas-goenka deleted the fs-ls branch June 5, 2023 15:41

pietern mentioned this pull request Jun 12, 2023
Release v0.100.3 #461
Merged

pietern mentioned this pull request Jun 15, 2023
Add unit test for bricks fs ls #50
Closed

andrewnester mentioned this pull request Jul 13, 2023
Add unit test for fs ls command #20
Closed

denik pushed a commit that referenced this pull request May 20, 2026
Add fs ls command for dbfs (#429)
2bfbbe3
## Changes 1. Adds fs ls command 2. Adds ability to define multiple templates ## Tests Manually and integration tests

Uh oh!

Conversation

shreyas-goenka commented Jun 2, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

shreyas-goenka commented Jun 4, 2023

Uh oh!

shreyas-goenka commented Jun 4, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka commented Jun 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shreyas-goenka commented Jun 2, 2023•
edited
Loading