Define language independent tests for your command line scripts and programs in simple yaml files.
- It runs on
windows,osxandlinux - It can validate local machines, ssh hosts and docker containers
- It is a self-contained binary - no need to install a heavy lib or language
- It is easy and fast to write
For more information take a look at the quick start, the examples or the integration tests.
Probably the easiest way to install commander is by using go get to download and install it in one simple command:
go install github.com/commander-cli/commander/v2/cmd/commander@latestThis works on any OS, as long as go is installed. If go is not installed on your system, follow one of the methods below.
Visit the release page to get the binary for you system.
curl -L https://github.com/commander-cli/commander/releases/download/v2.5.0/commander-linux-amd64 -o commander
chmod +x commander- Download the current release
- Add the path to your path environment variable
- Test it:
commander --version
A commander test suite consists of a config and tests root element. To start quickly you can use
the following examples.
# You can even let commander add tests for you!
$ ./commander add --stdout --file=/tmp/commander.yaml echo hello
tests:
echo hello:
exit-code: 0
stdout: hello
written to /tmp/commander.yaml
# ... and execute!
$ ./commander test /tmp/commander.yaml
Starting test file /tmp/commander.yaml...
✓ echo hello
Duration: 0.002s
Count: 1, Failed: 0Here you can see an example with all features for a quick reference
nodes:
ssh-host1:
type: sshaddr: 192.168.0.1:22user: rootpass: passssh-host2:
type: sshaddr: 192.168.0.1:22user: rootidentity-file: /home/user/id_rsa.pubdocker-host1:
type: dockerimage: alpine:2.4docker-host2:
type: dockerinstance: alpine_instance_1config: # Config for all executed testsdir: /tmp #Set working directoryenv: # Environment variablesKEY: globaltimeout: 50s# Define a timeout for a command under testretries: 2# Define retries for each testnodes:
- ssh-host1 # define default hoststests:
echo hello: # Define command as titlestdout: hello # Default is to check if it contains the given charactersexit-code: 0# Assert exit-codeit should skip:
command: echo "I should be skipped"stdout: I should be skippedskip: trueit should fail:
command: invalidstderr:
contains: - invalid # Assert only contain worknot-contains:
- not in there # Validate that a string does not occur in stdoutexactly: "/bin/sh: 1: invalid: not found"line-count: 1# Assert amount of lineslines: # Assert specific lines1: "/bin/sh: 1: invalid: not found"json:
object.attr: hello # Make assertions on json objectsxml:
"//book//auhtor": Steven King # Make assertions on xml documentsfile: correct-output.txtexit-code: 127skip: falseit has configs:
command: echo hellostdout:
contains: - hello #See test "it should fail"exactly: helloline-count: 1config:
inherit-env: true # You can inherit the parent shells env variablesdir: /home/user # Overwrite working direnv:
KEY: local # Overwrite env variableANOTHER: yeah # Add another env variabletimeout: 1s# Overwrite timeoutretries: 5nodes: # overwrite default nodes
- docker-host1
- docker-host2exit-code: 0# Execute file commander.yaml in current directory
$ ./commander test# Execute a specific suite
$ ./commander test /tmp/test.yaml
# Execute a single test
$ ./commander test /tmp/test.yaml --filter "my test"# Execute suite from stdin
$ cat /tmp/test.yaml | ./commander test -
# Execute suite from url
$ ./commander test https://your-url/commander_test.yaml
# Execute suites within a test directory
$ ./commander test --dir /tmp
# Execute suites in a different working directory
$ ./commander test --workdir /examples minimal_test.yamlYou can use the add argument if you want to commander to create your tests.
# Add a test to the default commander.yaml
$ ./commander add echo hello
written to /tmp/commander.yaml
# Write to a given file
$ ./commander add --file=test.yaml echo hello
written to test.yaml
# Write to stdout and file
$ ./commander add --stdout echo hello
tests:
echo hello:
exit-code: 0
stdout: hello
written to /tmp/commander.yaml
# Only to stdout
$ ./commander add --stdout --no-file echo hello
tests:
echo hello:
exit-code: 0
stdout: helloNAME:
Commander - CLI app testing
USAGE:
commander [global options] command [command options] [arguments...]
COMMANDS:
test Execute the test suite
add Automatically add a test to your test suite
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--help, -h show help
--version, -v print the version
Tests are defined in the tests root element. Every test consists of a command and an expected result,
i.e. an exit-code.
tests: # root elementecho test: # test case - can either be the command or a given titlestdout: testexit-code: 0A test is a map which configures the test.
The key (echo test in the example above) of the test can either be the command itself or the title of the test which will be displayed in the test execution.
If the same command is tested multiple times it is useful to set the title of the test manually and use the command property.
Further the title can be useful to describe tests better. See the commander test suite as an example.
- name:
title or command under test - type:
map - default:
{}
Examples:
tests:
echo test: # command and title will be the samestdout: testexit-code: 0my title: # custom titlecommand: exit 1 # set command manuallyexit-code: 1command is a string containing the command to be tested. Further the command property is automatically parsed from
the key if no command property was given.
- name:
command - type:
string - default:
can't be empty - notes: Will be parsed from the
keyif nocommandproperty was provided and used as the title too
echo test: # use command as key and titleexit-code: 0it should print hello world: # use a more descriptive title...command: echo hello world # ... and set the command in the property manuallystdout: hello worldexit-code: 0config sets configuration for the test. config can overwrite global configurations.
- name:
config - type:
map - default:
{} - notes:
- for more information look at config
echo test:
config:
timeout: 5sexit-code is an int type and compares the given code to the exit-code of the given command.
- name:
exit-code - type:
int - default:
0
exit 1: # will passexit-code: 1exit 0: # will failexit-code: 1stdout and stderr allow to make assertions on the output of the command.
The type can either be a string or a map of different assertions.
If only a string is provided it will check if the given string is contained in the output.
- name:
stdout - type:
stringormap - default:
- notes: stderr works the same way
echo test:
stdout: test # make a contains assertionecho hello world:
stdout:
line-count: 1# assert the amount of lines and use stdout as a mapcontains is an array or string. It checks if a string is contained in the output.
It is the default if a string is directly assigned to stdout or stderr.
- name:
contains - type:
stringorarray - default:
[] - notes: default assertion if directly assigned to
stdoutorstderr
echo hello world:
stdout: hello # Default is a contains assertionecho more output:
stdout:
contains:
- more
- outputexactly is a string type which matches the exact output.
- name:
exactly - type:
string - default:
echo test:
stdout:
exactly: testjson is a map type and allows to parse json documents with a given GJSON syntax to query for specific data.
The key represents the query, the value the expected value.
- name:
json - type:
map - default:
{} - notes: Syntax taken from GJSON
cat some.json: # print json file to stdoutname.last: Anderson # assert on name.last, see document belowsome.json file:
{
"name": {"first": "Tom", "last": "Anderson"},
"age":37,
"children": ["Sara","Alex","Jack"],
"fav.movie": "Deer Hunter",
"friends": [
{"first": "Dale", "last": "Murphy", "age": 44, "nets": ["ig", "fb", "tw"]},
{"first": "Roger", "last": "Craig", "age": 68, "nets": ["fb", "tw"]},
{"first": "Jane", "last": "Murphy", "age": 47, "nets": ["ig", "tw"]}
]
}More examples queries:
"name.last" >> "Anderson"
"age" >> 37
"children" >> ["Sara","Alex","Jack"]
"children.#" >> 3
"children.1" >> "Alex"
"child*.2" >> "Jack"
"c?ildren.0" >> "Sara"
"fav\.movie" >> "Deer Hunter"
"friends.#.first" >> ["Dale","Roger","Jane"]
"friends.1.last" >> "Craig"
lines is a map which makes exact assertions on a given line by line number.
- name:
lines - type:
map - default:
{} - note: starts counting at
1;-)
echo test\nline 2:
stdout:
lines:
2: line 2 # asserts only the second lineline-count asserts the amount of lines printed to the output. If set to 0 this property is ignored.
- name:
line-count - type:
int - default:
0
echo test\nline 2:
stdout:
line-count: 2not-contains is a array of elements which are not allowed to be contained in the output.
It is the inversion of contains.
- name:
not-contains - type:
array - default:
[]
echo hello:
stdout:
not-contains: bonjour # test passes because bonjour does not occur in the outputecho bonjour:
stdout:
not-contains: bonjour # test fails because bonjour occurs in the outputxml is a map which allows to query xml documents viá xpath queries.
Like the [json][#json] assertion this uses the key of the map as the query parameter to, the value is the expected value.
- name:
xml - type:
map - default:
{} - notes: Used library xmlquery
cat some.xml:
stdout:
xml:
//book//author: J. R. R. Tolkiensome.xml file:
<book>
<author>J. R. R. Tolkien</author>
</book>file is a file path, relative to the working directory that will have
its entire contents matched against the command output. Other than
reading from a file this works the same as exactly.
The example below will always pass.
output should match file:
command: cat output.txtstdout:
file: output.txtSee stdout for more information.
- name:
stderr - type:
stringormap - default:
- notes: is identical to stdout
# >&2 echos directly to stderr">&2 echo error": stderr: errorexit-code: 0">&2 echo more errors":
stderr:
line-count: 1skip is a boolean type, setting this field to true will skip the test case.
- name:
skip - type:
bool - default:
false
echo test:
stdout: testskip: trueYou can add configs which will be applied to all tests within a file or just for a specific test case, i.e.:
config:
dir: /home/root # Set working directory for all teststests:
echo hello:
config: # Define test specific configs which overwrite global configstimeout: 5sexit-code: 0You also have the option to define a separate config file that will
be applied globally to all test cases being ran using the --config flag.
The following will set the working directory for all tests that do not
explicitly set config.dir in the file config or the
test case config. In short, the lowest level config values takes precednce.
./commander test --config foo/bar/my-conifg.yaml foo/bar/test-suite.yaml# foo/bar/my-conifg.yaml`config:
dir: /home/root # Set working directory for all tests# foo/bar/test-suite.yamltests:
echo hello:
config: # Define test specific configs which overwrite global configstimeout: 5sexit-code: 0dir is a string which sets the current working directory for the command under test.
The test will fail if the given directory does not exist.
- name:
dir - type:
string - default:
current working dir
dir: /home/rootenv is a hash-map which is used to set custom env variables. The key represents the variable name and the value setting the value of the env variable.
- name:
env - type:
hash-map - default:
{} - notes:
- read env variables with
${PATH} - overwrites inherited variables, see #inherit-env
- read env variables with
env:
VAR_NAME: my value # Set custom env varCURRENT_USER: ${USER} # Set env var and read from current envinherit-env is a boolean type which allows you to inherit all environment variables from your active shell.
- name:
inherit-env - type:
bool - default:
false - notes: If this config is set to
truein the global configuration it will be applied for all tests and ignores local test configs.
inherit-env: trueinterval is a string type and sets the interval between retries.
- name:
interval - type:
string - default:
0ns - notes:
- valid time units: ns, us, µs, ms, s, m, h
- time string will be evaluated by golang's
timepackage, further reading time/#ParseDuration
interval: 5s# Waits 5 seconds until the next try after a failed test is startedretries is an int type and configures how often a test is allowed to fail until it will be marked as failed for the whole test run.
- name:
retries - type:
int - default:
0 - notes: interval can be defined between retry executions
retries: 3# Test will be executed 3 times or until it succeedstimeout is a string type and sets the time a test is allowed to run.
The time is parsed from a duration string like 300ms.
If a tests exceeds the given timeout the test will fail.
- name:
timeout - type:
string - default:
no limit - notes:
- valid time units: ns, us, µs, ms, s, m, h
- time string will be evaluated by golang's
timepackage, further reading time/#ParseDuration
timeout: 600sCommander has the option to execute tests against other hosts, i.e. via ssh.
Available node types are currently:
local, execute tests locallyssh, execute tests viá sshdocker, execute tests inside a docker container
nodes: # define nodes in the node sectionssh-host:
type: ssh # define the type of the connection user: root # set the user which is used by the connectionpass: password # set password for authenticationaddr: 192.168.0.100:2222 # target host addressidentity-file: ~/.ssh/id_rsa # auth with private keytests:
echo hello:
config:
nodes: # define on which host the test should be executed
- ssh-hoststdout: helloexit-code: 0You can identify on which node a test failed by inspecting the test output.
The [local] and [ssh-host] represent the node name on which the test were executed.
✗ [local] it should test ssh host
✗ [ssh-host] it should fail if env could not be set
The local node is the default execution and will be applied if nothing else was configured.
It is always pre-configured and available, i.e. if you want to execute tests on a node and locally.
nodes:
ssh-host:
addr: 192.168.1.100user: ...tests:
echo hello:
config:
nodes: # will be executed on local and ssh-host
- ssh-host
- localexit-code: 0The ssh node type will execute tests against a configured node using ssh.
Limitations:
- The
inhereit-envconfig is disabled for ssh hosts, nevertheless it is possible to set env variables - Private registries are not supported at the moment
nodes: # define nodes in the node sectionssh-host:
type: ssh # define the type of the connection user: root # set the user which is used by the connectionpass: password # set password for authenticationaddr: 192.168.0.100:2222 # target host addressidentity-file: ~/.ssh/id_rsa # auth with private keytests:
echo hello:
config:
nodes: # define on which host the test should be executed
- ssh-hoststdout: helloexit-code: 0The docker node type executes the given command inside a docker container.
Notes: If the default docker registry should be used prefix the container with the registry docker.io/library/
nodes:
docker-host:
type: docker
image: docker.io/library/alpine:3.11.3
docker-exec-user: 1000 # define the owner of the executed command
user: user # registry user
pass: password # registry password, it is recommended to use env variables like $REGISTRY_PASS
config:
nodes:
- docker-host
tests:
"id -u":
stdout: "1001"
See the documentation at development.md
Heavily inspired by goss.
Similar projects: