Specimen

Yaml-based data-driven testing

Specimen is a data-driven testing library as well as a yaml data format. It enforces separation between the feature being tested and the data used for testing. Specimen enable the creation of language-agnostic actionable specifications.

So far, Specimen has been implemented in four languages: Rust, Golang, Python and JS (TypeScript acutally). The implementations are backed by the common Specimen specification.

It supports using the FOCUS and PENDING flags in the data tree to run only parts of the test data.

Overview

• A Test Box is a user-defined function passed to specimen.run. It serves as an adaptator between Specimen and the user code being tested. As such, it prepares the data for testing, runs the code being tested and performs the checks on the code result once it has finished.
• A Slab is a leaf of the yaml files data tree that Specimen processes.
• A Tile is a chunk of data to be loaded into a test box. When test matrices are used, a slab will produce multiple tiles.

Getting started with Specimen in Golang

To get started, create a directory it/ and the three files it.goit_test.go and it_testdata.yaml. For each file, copy the content of linked section. Install the dependencies and finally run go test in the it/ directory:

mkdir it
cd it
touch it.go it_test.go it_testdata.yaml

Fill each of the three files with the code found in this README:

• For it.go see Example package code
• For it_test.go see Code box
• For it_testdata.yml see Yaml Data

Finally, run:

go mod init it
go mod tidy
go test

You should get an output similar to this one:

TestIt:
Ran 4 tiles in 0s
SUCCESS -- 4 Passed | 0 Failed | 0 Aborted | 0 Panicked
PASS
ok it 0.646s

/!\ If go mod tidy fails to find the specimen module, you may need to clean the go cache: go clean -cache, go clean -modcache.

Yaml Data

The yaml data file looks like this:

box: zoocontent:
- content:
- name: horseanimal: horseexpected_result: horse
- flag: PENDINGname: parasprite # this slab will be ignoredanimal: parasprite
- name: zebraanimal: zebraexpected_result: horse zebra
- name: animal matrixanimal: [mouse, cat, dog]
- name: matrix checkanimal: wolfexpected_result: horse zebra mouse cat dog wolf

Test box

A test box is an adapter between the parsed data and the library code being tested. It takes as input the testing context s and the input map. A test box looks like this:

package it_test
import (
"testing"
specimen "github.com/ditrit/specimen/go""github.com/ditrit/specimen/test/zoo"
)
funcTestIt(t*testing.T) {
specimen.Run(
t,
func(s*specimen.S, input specimen.Dict) {
animal:=input["animal"]
expected:=input["expected_result"]
iflen(animal) >0 {
output:=zoo.AddAnimal(animal)
iflen(expected) >0 {
s.ExpectEqual(output, expected, "result comparison")
}
}
},
[]specimen.File{
specimen.ReadLocalFile("it_testdata.yaml"),
},
)
}

Example package code

package it
import"strings"typeZoo []stringvarzooZoofuncAddAnimal(animalstring) string {
zoo=append(zoo, animal)
returnstrings.Join(zoo, " ")
}

Running the tests

# rustcd rust
cargo test# golang
go test ./test/counter ./test/novel ./test/zoo
# or
go test test/counter/counter_test.go
go test test/novel/novel_test.go
go test test/zoo/zoo_test.go
# python (v3.10 or a future version)cd python
python -m poetry install
cd ..
(cd test/counter && python counter_test.py)
(cd test/novel && python novel_test.py)
(cd test/zoo && python zoo_test.py)
# jscd js
yarn install
yarn build
cd ..
node test/counter/counter_test.js
(cd test/novel && yarn install && node novel_test.js)
node test/zoo/zoo_test.js

Yaml Schema

The content of a yaml test data file must match the main rule of the lidy schema below:

main: nodule# A tip can be a string or a list of strings. In the case of a list, all the# combination of values taken from this list and lists of other parameters will# be generated and run. This produces the effect of a test matrix.tip:
_oneOf:
- string
- _listOf: stringnodule:
_mapFacultative:
# `content` contains all the children of the current nodule. Nodes which# contain a content entry are seen as tree nodes, while nodes which do# not contain it are seen as leavescontent:
- _listOf: nodule# The flag can start with "PENDING" or "FOCUS" or not be specified.# The PENDING flag tells the engine to skip the node and all its decendants.# The FOCUS flag tells the engine to skip all the OTHER nodes; the one which# do not have the flag "FOCUS".flag: string# `about` can contain any data: it will not be checked by the parser, and it# will not appear in the data passed to the box functionabout: any# all the entries of the mapping will be added to the descendant slabs of# this nodule and then passed to the code box, except for the `content`,# `flag` and `about` entries_mapOf: { string: tip }# Besides all the keys that are found in the yaml, the test box will be passed# an argument "filepath" which contains the path to the yaml file, as specified# to Lidy.