Automated testing

Amber is the automation driver used to coordinate product and tool validation. She runs command-line test steps, captures the evidence produced by those steps, and writes structured LaTeX fragments that autodoc can assemble into a validation report.

Amber is a complete product checkout when cloned under ~/soup/amber. She is self-validating: the same Amber workflow used by downstream products is used to exercise Amber's own command-line behavior and build her validation report. Amber borrows from a Ruby-on-Rails concept of convention over configuration. In this regard, Amber consumes YAML documents placed beneath a factory directory root and writes generated evidence beneath test-output.

Review Amber's report directory to see the automation and report-generation pattern used by the related ~/soup products. example.pdf has been provided to demonstrate the automation framework components amber, autodoc, docbld, and tlc-article.

TL;DR

Released version

1.6.4.415

Working with source

Copy the text below and paste into a shell.

git clone https://github.com/traap/amber.git

Build and install Amber

cd amber && rake build:amber

Validate Amber

cd amber && rake validate:amber

The validation task fails when required environment variables are missing, when Amber cannot generate her evidence, or when docbld cannot rebuild and deploy the report.

Supported Systems

amber is targeted to work seamlessly with Linux distributions.

Prerequisites

1. Ruby
2. Bundler
3. Rake

Optional components are only needed if you want to validate Amber and

build Amber's documentation

1. TexLive
2. autodoc
3. docbld
4. tlc-article

Input Factory

A YAML Plan

plan:
name: A plan namepurpose: The purpose of this test plan.includes:
- suite:
- name: a-suite-name

A YAML Suite

suite:
name: A Suite Namepurpose: The purpose of this test suite.requirement: 1, 2, and 3.includes:
- case:
- name: a-test-case
- name: another-test-case
- name: yet-another-test-case

A YAML Case

case:
name: First Test Casepurpose: Demonstrate requirements are met.requirement: 1, 2, and 3steps:
- confirm: Program echo has been installed.expectation: echo installation location is displayed.sudo: truecommand: whichargument: echoevidence: Starts on next line.
- confirm: Program date has been installed.expectation: date installation location is displayed.sudo: falsecommand: whichargument: dateevidence: Starts on next line.
- confirm: Program man has been installed.expectation: man installation location is displayed.sudo: falsecommand: whichargument: manevidence: Starts on next line.

Amber directory convention

../factory
../factory/config
../factory/config/app_config.yaml
../factory
../factory/plan
../factory/plan/a-plan-name/a-plan-name.yaml
../factory/suite
../factory/suite/a-suite-name/a-suite-name.yaml
../factory/case
../factory/case/a-test-case/a-test-case.yaml
../factory/case/another-test-case/another-test-case.yaml
../factory/case/yet-another-test-case/yet-another-test-case.yaml

Output Factory

Generated by amber

../test-output/factory
../test-output/factory/plan
../test-output/factory/plan/a-plan-name/a-plan-name.tex
../test-output/factory/suite
../test-output/factory/suite/a-suite-name/a-suite-name.tex
../test-output/factory/case
../test-output/factory/case/a-test-case/a-test-case.tex
../test-output/factory/case/a-test-case/a-test-case-step-001.tex
../test-output/factory/case/a-test-case/a-test-case-step-step-001-log.tex
../test-output/factory/case/a-test-case/a-test-case-step-step-001-status.tex
../test-output/factory/case/a-test-case/a-test-case-step-step-002-log.tex
../test-output/factory/case/a-test-case/a-test-case-step-step-002-status.tex
../test-output/factory/case/a-test-case/a-test-case-step-step-003-log.tex
../test-output/factory/case/a-test-case/a-test-case-step-step-003-status.tex

LaTeX test-results output

\tpo{full/path/to/test-output/factory/plan/a-plan-name/a-plan-nam}
\tso{full/path/to/test-output/factory/suite/a-suite-name/a-suite-nam}
\tco{full/path/to/test-output/factory/case/t001/t001}
\tco{full/path/to/test-output/factory/case/t002/t002}
\tco{full/path/to/test-output/factory/case/t003/t003}

LaTeX macros

test plan output (tpo), test suite output (tso), and test case output (tco) are LaTeX macros autodoc uses to assemble output from amber into a report.

Refer to autodoc to learn how an external program interacts with amber. amber must create an output factory containing files before they can be automatically consumed.

Generated by amber when --language and --browser are used

../test-output/factory
../test-output/factory/Chrome/fr-eu/plan
../test-output/factory/Chrome/fr-eu/plan/a-plan-name/a-plan-name.tex
../test-output/factory/Chrome/fr-eu/suite
../test-output/factory/Chrome/fr-eu/suite/a-suite-name/a-suite-name.tex
../test-output/factory/Chrome/fr-eu/case
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case.tex
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case-step-001-log.tex
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case-step-001-status.tex
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case-step-002-log.tex
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case-step-002-status.tex
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case-step-002-log.tex
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case-step-002-status.tex
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case-step-003-log.tex
../test-output/factory/Chrome/fr-eu/case/a-test-case/a-test-case-step-003-status.tex

LaTeX test-results output

\tpoC{Chrome}{fr-eu}{factory/plan/a-plan-name/a-plan-name}
\tsoC{Chrome}{fr-eu}{factory/suite/a-suite-name/a-suite-nam}
\tcoC{Chrome}{fr-eu}{factory/case/t001/t001}
\tcoC{Chrome}{fr-eu}{factory/case/t002/t002}
\tcoC{Chrome}{fr-eu}{factory/case/t003/t003}

LaTeX macros

test plan output (tpoC), test suite output (tsoC), and test case output (tcoC) are LaTeX macros autodoc uses to assemble output from amber into a report.

Refer to autodoc to learn how an external program interacts with amber. Amber must create an output factory containing files before they can be automatically consumed.

Console output

amber captures status, system out, and system error and records the results to the a-test-case.tex file.

Step files

amber records PASS or FAIL for each Test Step. The result is determined by the program that amber runs.

PNG files

Any custom program amber invokes can take a screen capture and record it as test-output/factory/case/a-test-case/a-test-case-001.png. Multiple PNG files are supported.

CSV files

Any custom program amber invokes can record CSV files as test-output/factory/case/a-test-case/a-test-case-001.csv. Multiple CSV files are supported.

amber command line

amber --help
Usage: amber [options]
Specific options:
-b, --browser BROWSER Select Browser
["None", "Chrome", "Edge", "Firefox", "IE",
"Opera"]
-n, --nodryrun No Dryrun
-e, --log-environment List environment
-f, --file x,y,x File name
-l, --language LANGUAGE Select language
["zz", "cs", "da", "de", "en", "es", "fr-ca",
"fr-eu", "it", "ne", "no", "pl", "ro", "sv"]
-v, --verbose Verbose
-L, --log-command Log Command
-r, --log-requirement Log Requirement
-S, --simulate Simulate run to create Test Output
directory.
-O, --obliterate Obliterate Test Output directory before Test
Execution.
-w, --writer WRITER Select writer
["Ascii", "LaTeX"]
-p, --plan x,y,x Plan name
-s, --suite x,y,x Suite name
-c, --case x,y,x Case name
-h, --help Show this message
--version Show version
-d, --dump Dump options (must be last).

General

--help

Show this message.

1.6.0.367 is the current version.

--verbose

Log the commands that will be run when --nodryrun is used. NOTE: This option was disabled because Amber writes the commands to the Test Output Factory. This occurs because Amber is capturing standard input, standard output, and standard error. Amber conditionally writes either standard output or standard error to the Test Output Factory based on the exit status of the program run.

Input Factory

--plan

A comma-separated list of test plan names amber is to process. The following directory and YAML file name convention is mandatory: factory/plan/a-plan/a-plan.yaml.

--suite

A comma-separated list of test suite names amber is to process. The following directory and YAML file name convention is mandatory: factory/suite/a-suite/a-suite.yaml.

--case

A comma-separated list of test case names amber is to process. The following directory and YAML file name convention is mandatory: factory/case/suite/a-case/a-case.yaml.

--file

A comma-separated list of file names amber is to process.

Substitution

--browser

amber uses Chrome by default. Your Test Plan, Test Suite, and Test Case must be written to reference a program that uses a Web test driver.

--language

amber uses English (en) by default. Your Test Plan, Test Suite, and Test Case must be written to reference a program that requires internationalization and localization.

Output Factory

--log-command

Amber recursively calls Amber. test-output/commands.log shows each Amber invocation.

--log-requirement

test-output/requirements.log is used to record each requirement and YAML file that references a requirement.

--nodryrun

By default, amber does not have side effects when run. You must explicitly use the --nodryrun option to cause side effects. The commands that would have been executed are echoed to system out.

--simulate

The simulate option is used to create a Test Output directory so that you can design your report. Each command defined in your Test Case YAML file is echoed to the Test Output directory Test Case file.

--obliterate

The Test Output directory is obliterated before running any Tests.

--environment

A list of environment variables amber records in the output factory. See environment.rb for a complete listing of files.

Add these functions to .bashrc

AMBERPATH=${HOME}/git/amber
export AMBERPATH
functionnewfactoryitem() {
${AMBERPATH}/bin/newfactoryitem $@
}
functioncheck-test-output() {
echo grep -rw --include=\*step*.* test-output/ -e $1
grep -rw --include=\*step*.* test-output/ -e $1
}
functionpass() {
check-test-output PASS
}
functionfail() {
check-test-output FAIL
}
functionrequirements() {
echo grep -rw --include=\*.yaml factory/ -e 'requirement:'
grep -rw --include=\*.yaml factory/ -e 'requirement:'
}

Amber demonstration

Amber has a Test Input Factory that is used to validate Amber. The following commands will demonstrate producing amber.pdf.

1. -n, --nodryrun
2. -e, --environment
3. -L, --log-command
4. -r, --log-requirement
5. -O, --obliterate
6. -p, --plan

cd git/amber/report
rake validate:amber

Amber demonstration output

2. git/amber/report/test-output/commands.log
3. git/amber/report/test-output/environment.tex
4. git/amber/report/test-output/requirements.csv
5. git/amber/report/test-output/test-results.tex
6. git/amber/report/_build/amber.pdf