Skip to content

Add tests for shell execution on artifacts build#2884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shreyas-goenka merged 20 commits into from
Jun 6, 2025
Merged

Add tests for shell execution on artifacts build #2884

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e985d53
Add test for build command execution
shreyas-goenka May 14, 2025
491fa71
fix tests
shreyas-goenka May 14, 2025
a4a6257
new tests
shreyas-goenka May 14, 2025
cb1ecc3
-
shreyas-goenka May 14, 2025
8f42e5c
fix test
shreyas-goenka May 14, 2025
cd8b5c8
fix typo
shreyas-goenka May 14, 2025
a69b146
fix test
shreyas-goenka May 14, 2025
6cfd936
fix tests
shreyas-goenka May 14, 2025
30dc3ef
-
shreyas-goenka May 14, 2025
fc25366
-
shreyas-goenka May 14, 2025
a090604
-
shreyas-goenka May 14, 2025
11708fb
-
shreyas-goenka May 14, 2025
c08f532
-
shreyas-goenka May 14, 2025
3b23f3f
rephrase comment
shreyas-goenka May 22, 2025
edf9420
Merge remote-tracking branch 'origin' into add-shell-test
shreyas-goenka May 22, 2025
cc9990a
Merge remote-tracking branch 'origin' into add-shell-test
shreyas-goenka May 23, 2025
d47de30
Merge branch 'main' into add-shell-test
denik Jun 5, 2025
ce1fe94
Merge remote-tracking branch 'origin' into add-shell-test
shreyas-goenka Jun 6, 2025
ca7936b
fix tests
shreyas-goenka Jun 6, 2025
e24ef11
fix windows test
shreyas-goenka Jun 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions acceptance/bundle/artifacts/shell/bash/databricks.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
bundle:
name: shell-bash

artifacts:
my_artifact:
executable: bash
# echo in bash does not treat \n as a newline.
build: |
echo "\n\n\n should not see new lines" > out.shell.txt
echo "multiline scripts should work" >> out.shell.txt
pwd >> out.shell.txt
3 changes: 3 additions & 0 deletions acceptance/bundle/artifacts/shell/bash/out.shell.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
\n\n\n should not see new lines
multiline scripts should work
[TEST_TMP_DIR]
6 changes: 6 additions & 0 deletions acceptance/bundle/artifacts/shell/bash/output.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@

>>> [CLI] bundle deploy
Building my_artifact...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/shell-bash/default/files...
Deploying resources...
Deployment complete!
1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/bash/script
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
trace $CLI bundle deploy
5 changes: 5 additions & 0 deletions acceptance/bundle/artifacts/shell/bash/test.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
# Line feed (\n) characters are rendered differently by bash vs sh on Unix based systems.
# However that does not hold true for Windows WSL. Since the goal of this test is to assert
# the chosen shell is being used, we skip this test on Windows.
[GOOS]
windows = false
8 changes: 8 additions & 0 deletions acceptance/bundle/artifacts/shell/basic/databricks.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
bundle:
name: shell-basic

artifacts:
my_artifact:
build: |
echo "Hello, World!" > out.shell.txt
echo "Bye, World!" >> out.shell.txt
2 changes: 2 additions & 0 deletions acceptance/bundle/artifacts/shell/basic/out.shell.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
Hello, World!
Bye, World!
6 changes: 6 additions & 0 deletions acceptance/bundle/artifacts/shell/basic/output.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@

>>> [CLI] bundle deploy
Building my_artifact...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/shell-basic/default/files...
Deploying resources...
Deployment complete!
1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/basic/script
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
trace $CLI bundle deploy
12 changes: 12 additions & 0 deletions acceptance/bundle/artifacts/shell/cmd/databricks.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
bundle:
name: shell-cmd

artifacts:
my_artifact:
executable: cmd
build: |
if defined CMDCMDLINE (
Comment thread
shreyas-goenka marked this conversation as resolved.
echo shell is cmd.exe> out.shell.txt
)
echo multiline scripts should work>> out.shell.txt
echo %CD%>> out.shell.txt
3 changes: 3 additions & 0 deletions acceptance/bundle/artifacts/shell/cmd/out.shell.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
shell is cmd.exe
multiline scripts should work
[TEST_TMP_DIR]
6 changes: 6 additions & 0 deletions acceptance/bundle/artifacts/shell/cmd/output.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@

>>> [CLI] bundle deploy
Building my_artifact...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/shell-cmd/default/files...
Deploying resources...
Deployment complete!
1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/cmd/script
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
trace $CLI bundle deploy
4 changes: 4 additions & 0 deletions acceptance/bundle/artifacts/shell/cmd/test.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
# cmd.exe is only available on Windows.
[GOOS]
darwin = false
linux = false
10 changes: 10 additions & 0 deletions acceptance/bundle/artifacts/shell/default/databricks.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
bundle:
name: shell-bash

artifacts:
my_artifact:
# Default shell is bash. echo in bash does not treat \n as a newline.
build: |
echo "\n\n\n should not see new lines" > out.shell.txt
echo "multiline scripts should work" >> out.shell.txt
pwd >> out.shell.txt
3 changes: 3 additions & 0 deletions acceptance/bundle/artifacts/shell/default/out.shell.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
\n\n\n should not see new lines
multiline scripts should work
[TEST_TMP_DIR]
6 changes: 6 additions & 0 deletions acceptance/bundle/artifacts/shell/default/output.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@

>>> [CLI] bundle deploy
Building my_artifact...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/shell-bash/default/files...
Deploying resources...
Deployment complete!
1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/default/script
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
trace $CLI bundle deploy
5 changes: 5 additions & 0 deletions acceptance/bundle/artifacts/shell/default/test.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
# Line feed (\n) characters are rendered differently by bash vs sh on Unix based systems.
# However that does not hold true for Windows WSL. Since the goal of this test is to assert
# the chosen shell is being used, we skip this test on Windows.
[GOOS]
windows = false
7 changes: 7 additions & 0 deletions acceptance/bundle/artifacts/shell/invalid/databricks.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
bundle:
name: shell-invalid

artifacts:
my_artifact:
executable: invalid
build: foobar
7 changes: 7 additions & 0 deletions acceptance/bundle/artifacts/shell/invalid/output.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@

>>> [CLI] bundle deploy
Building my_artifact...
Error: invalid is not supported as an artifact executable, options are: bash, sh or cmd


Exit code: 1
1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/invalid/script
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
trace $CLI bundle deploy
11 changes: 11 additions & 0 deletions acceptance/bundle/artifacts/shell/sh/databricks.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
bundle:
name: shell-sh

artifacts:
my_artifact:
executable: sh
# echo in sh treats \n as a newline.
build: |
echo "\n\n\nshould see new lines" > out.shell.txt
echo "multiline scripts should work" >> out.shell.txt
pwd >> out.shell.txt
6 changes: 6 additions & 0 deletions acceptance/bundle/artifacts/shell/sh/out.shell.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@



should see new lines
multiline scripts should work
[TEST_TMP_DIR]
6 changes: 6 additions & 0 deletions acceptance/bundle/artifacts/shell/sh/output.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@

>>> [CLI] bundle deploy
Building my_artifact...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/shell-sh/default/files...
Deploying resources...
Deployment complete!
1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/sh/script
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
trace $CLI bundle deploy
5 changes: 5 additions & 0 deletions acceptance/bundle/artifacts/shell/sh/test.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
# Line feed (\n) characters are rendered differently by bash vs sh on Unix based systems.
# However that does not hold true for Windows WSL. Since the goal of this test is to assert
# the chosen shell is being used, we skip this test on Windows.
[GOOS]
windows = false
3 changes: 3 additions & 0 deletions acceptance/bundle/artifacts/shell/test.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
Local = true
Cloud = false
RecordRequests = false
2 changes: 1 addition & 1 deletion acceptance/selftest/basic/test.toml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,7 @@ Ignore = ['ignore*']
#windows = false

# Disable on Mac
#mac = false
#darwin = false

@shreyas-goenkashreyas-goenkaMay 14, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes small typo, Mac OS is "darwin", not "mac" for GOOS. You need to set darwin = false for this to work.


# Disable on Linux
#linux = false
Expand Down
4 changes: 2 additions & 2 deletions libs/exec/shell_execv.go
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -23,7 +23,7 @@ func shellExecvOpts(content, dir string, env []string) (ExecvOptions, error) {

// Variant of [Execv] that runs the given script through a shell
func ShellExecv(content, dir string, env []string) error {
newOpts, err := shellExecvOpts(content, dir, env)
opts, err := shellExecvOpts(content, dir, env)

@shreyas-goenkashreyas-goenkaMay 14, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small typo fix based on: #2862 (comment)

if err != nil {
return err
}
Expand All@@ -38,5 +38,5 @@ func ShellExecv(content, dir string, env []string) error {
//
// For windows the temp files are not automatically cleaned up. Automatic cleanup
// is a opt-in behavior.
return Execv(newOpts)
return Execv(opts)
}