IT Runner - Integration Tests executor

IT Runner is a tool to automate the execution of integration tests across multiple versions of dependencies. It helps to build compatibiliy matrix by executing the same suite of tests against different setups of components.

DISCLAIMER: IT Runner is still in its early beta stages. Please mind that breaking changes in its APIs and configuration files format can be made until a first 1.X version is stabilised.

• IT Runner - Integration Tests executor
  • Usage
    • Suite configuration
    • Templating
    • Dependencies configuration
    • Invoking the tool from GO code
    • Using the config library in go tests
  • Roadmap

Usage

ITRunner relies on two configuration files to bootstrap the components to be used in the integration tests (dependencies configuration file) and to execute the tests themselves (suite configuration file).

Suite configuration

suite:
testGroups:
- groupName: postgres # descriptive name of the group of testspackages:
- ./tests/persistence/... # GO packages containing test filesversions:
- versionName: 9.6.16 # descriptive name of the version of the dependency(ies) being testeddependsOn:
- id: postgres_9_6_16 # list of dependency ids that will be requested before the test executiontestConfig:
templatePath: ./path/to/config.tpl # golang template of custom config used by the tests. It is rendered with data from dependenciesinputDataFrom:
dependencies:
- id: postgres_9_6_16 # Id of the dependencytemplateVar: Postgres # Variable to host the dependency data when templatingoutputPath: ./configs/rendered.yml # output of the result config file after rendering the template
- versionName: 10.12dependsOn:
- id: postgres_10_12testConfig:
templatePath: ./paht/to/another/config.tplinputDataFrom:
dependencies:
- id: postgres_10_12templateVar: PostgresoutputPath: ./configs/another-render.yml

Templating

When using templates to render configuration for tests, IT Runner will inject the following information for each dependency specified in testConfig:

• Host
• Port

Dependencies configuration

See the dependencymanager.ContainerRunConfig struct for the allowed fields of the container config section.

dependencies:
- id: postgres_9_6_16 # Id of the dependency, used in the suite config filecontainer: # Specs to run a Docker containerrepository: "postgres"tag: "9.6.16"env:
- "POSTGRES_PASSWORD="
- "POSTGRES_DB="
- id: postgres_10_12container:
repository: "postgres"tag: "10.12"env:
- "POSTGRES_PASSWORD="exposedPorts:
- "5432:5432"

Invoking the tool from GO code

Please check this example on how to invoke IT Runner from a GO script:

package main
import (
"fmt""os""strings""github.com/Adhara-Tech/itrunner/pkg/itrunner""github.com/Adhara-Tech/itrunner/cmd/integrationtestrunner"
)
funcmain() {
opt:= integrationtestrunner.RunnerOptions{
CompatibilityMatrixConfigFilePath: "path/to/itrunner-suite.yaml",
CompatibilityMatrixDependenciesFilePath: "path/to/itrunner-dependencies.yaml",
InDocker: false, # settotruewhenyouareexecutingthetestfrominsideadockercontainer
}
result, err:=integrationtestrunner.Run(opt)
iferr!=nil {
os.Exit(1) // an error happened during the test execution (for instance, docker daemon down or similar non-test related issues)
}
for_, testResult:=rangeresult.AllTestResults {
for_, versionResult:=rangetestResult.VersionExecutionResults {
ifversionResult.Result!=itrunner.TestSuccess {
os.Exit(1) // in this example: return non-zero exit code if one of the test failed
}
}
}
}

Using the config library in go tests

IT Runner provides a github.com/Adhara-Tech/itrunner/cmd/exportedtypes package with a ReadTestEnvExecutionData(data interface{}) function that allows you to load the configuration rendered from testConfig settings into the data structure.

package foostorage_test
import (
"testing"
"github.com/Adhara-Tech/itrunner/cmd/exportedtypes"
"github.com/your/coolproject/foostorage"
"github.com/stretchr/testify/assert"
)
type ExampleTestConfig struct {
ApplicationHost string `mapstructure:"applicationHost" yaml:"applicationHost"`
ApplicationPort string `mapstructure:"applicationPort" yaml:"applicationPort"`
}
func TestDoFoo(t *testing.T) {
var config ExampleTestConfig
err := exportedtypes.ReadTestEnvExecutionData(&config)
assert.NoError(t, err)
err = foostorage.DoFoo(config)
assert.NoError(t, err)
}

Roadmap

• Multiple ports exposed as part of a dependency (containers and pre provisioned services)
• Support for pre provisioned services
• Fine-grained control to shut down dependencies (now only after a group). Options can be after a version or after a suite
• Redefine test executable (support for gotestsum)
• Labeling to discriminate tests to be executed
• Redefine env variable used to inject data to tests
• Support to inject several variables

   testVars:
  - name: configPath
  value: ./tmp-configs/databases/postgres_config.yml
  - name: configValue
  type: [raw,yaml,json]
  fromFile: ./tmp-configs/databases/postgres_config.yml #reads the file
  

• Support more config formats other than YAML in exportedtypes.ReadTestEnvExecutionData(data interface{})