Scan Test Action

A GitHub Action for running automated tests when scanners are modified in the scanner registry.

Overview

This action detects which scanners have been modified in a pull request and runs their associated tests across CI/CD providers. It enables scanner authors to validate their changes before merging.

Usage

GitHub Action

- uses: boostsecurityio/scanner-registry-testing/test-action@mainwith:
provider: github-actionsprovider-config: | { "token": "${{ secrets.TEST_RUNNER_TOKEN }}", "owner": "boostsecurityio", "repo": "test-runner-github", "workflow_id": "scanner-test.yml" }registry-path: "."registry-repo: ${{ github.event.pull_request.head.repo.full_name }}registry-ref: ${{ github.event.pull_request.head.sha }}base-ref: ${{ github.event.pull_request.base.sha }}

Action Inputs

Input	Required	Description
`provider`	Yes	CI/CD provider key (e.g., `github-actions`, `gitlab-ci`, `azure-devops`, `bitbucket`)
`provider-config`	Yes	JSON configuration for the provider (see provider sections below)
`registry-path`	No	Path to registry root (default: `.`)
`registry-repo`	Yes	Registry repository in org/repo format. For fork PRs, use the fork repo.
`registry-ref`	Yes	Git ref of the registry (commit SHA). For fork PRs, use the fork SHA.
`base-ref`	No	Base git reference for diff (default: `origin/main`)

Action Outputs

Output	Description
`results`	JSON object with test results including `status`, `total`, `passed`, `failed`, and `scanners` array

CLI

The action can also be run directly via CLI:

poetry run python -m scan_test_action.cli \
--provider github-actions \
--provider-config '{"token": "...", "owner": "...", "repo": "...", "workflow_id": "..."}' \
--registry-path . \
--registry-repo org/scanner-registry \
--registry-ref abc123def456 \
--base-ref origin/main

Development

Prerequisites

Python 3.12+
Poetry
Docker (for module tests)
act (for module tests)

Test Orchestration Flow

The TestOrchestrator coordinates test execution on a single CI/CD provider. It receives pre-loaded test definitions and orchestrates their execution in parallel.

┌──────────────────┐ ┌──────────────────┐
│ Test Definitions │ │ Provider │
│ (pre-loaded) │────▶│ Dispatch │
└──────────────────┘ └────────┬─────────┘
│
┌──────────────────┐ │
│ Results │◀───────┘
│ Aggregation │
└──────────────────┘

Usage

orchestrator=TestOrchestrator(provider=my_provider)
results=awaitorchestrator.run_tests(
test_definitions={"org/scanner": test_def},
registry_repo="org/registry",
registry_ref="abc123",
)

Execution Steps

Dispatch Tests: Send all tests to the provider in parallel using asyncio.gather()
Wait for Completion: Each dispatched test is polled until complete or timeout
Aggregate Results: Collect results into ScannerResult objects

Result Structure

Each scanner produces a ScannerResult containing:

scanner_id: The scanner identifier (e.g., "boostsecurityio/trivy-fs")
results: Sequence of TestResult objects from the provider

Error Handling

Empty test definitions: Returns empty list immediately
Dispatch failure: Returns error result with exception message
Wait failure: Returns error result with exception message
Partial failure: Successful results are preserved alongside error results

Scanner Detection

The action compares the PR branch against the base branch to identify modified scanners. A scanner is considered modified if any file under scanners/<org>/<scanner>/ has changed.

Only scanners with a tests.yaml file are tested. Scanners without test definitions are skipped.

How It Works

Compares git refs (e.g., origin/main vs HEAD) to find changed files
Extracts unique scanner identifiers from paths like scanners/org/scanner/file.yaml
Filters to only scanners that have a tests.yaml file
Returns the list of testable scanners

Git Reference Resolution

In CI environments like GitHub Actions, branch refs often exist with an origin/ prefix. The detector automatically tries both forms when resolving references.

Test Definition Format

Scanner tests are defined in tests.yaml files within each scanner directory:

version: "1.0"tests:
- name: "Smoke test - source code"type: "source-code"source:
url: "https://github.com/OWASP/NodeGoat.git"ref: "main"
- name: "Container image scan"type: "container-image"source:
url: "https://github.com/example/docker-app.git"ref: "v1.0"scan_paths:
- "app"
- "api"

Fields

Field	Required	Description
`version`	Yes	Schema version (currently "1.0")
`tests`	Yes	List of test specifications
`tests[].name`	Yes	Human-readable test name
`tests[].type`	Yes	Either "source-code" or "container-image"
`tests[].source.url`	Yes	Git repository URL (HTTPS)
`tests[].source.ref`	Yes	Git reference (branch, tag, or commit SHA)
`tests[].scan_paths`	No	Paths to scan (default: ["."])
`tests[].timeout`	No	Test timeout (default: "5m")

Matrix Expansion

Each test is expanded into matrix entries for CI execution. A test with multiple scan_paths creates one matrix entry per path, enabling parallel execution.

Provider Architecture

The action supports multiple CI/CD providers through a common interface. Each provider handles dispatching tests and polling for results.

Plugin System

Providers are registered as entry points in pyproject.toml and loaded dynamically by key:

fromscan_test_action.providers.loadingimportload_provider_manifest# Load provider by key (e.g., from config or CLI)manifest=load_provider_manifest("github-actions")
# Create provider instance using the factoryasyncwithmanifest.provider_factory(config) asprovider:
results=awaitprovider.dispatch_scanner_tests(...)

Each manifest contains:

config_cls: The provider's Pydantic configuration class
provider_factory: The async context manager factory (typically Provider.from_config)

To add a new provider:

Create the provider module under scan_test_action/providers/<name>/
Add a manifest.py with the ProviderManifest instance
Register it in pyproject.toml under [tool.poetry.plugins."scan_test_action.providers"]
Run poetry install to update the entry points

Provider Interface

Providers implement PipelineProvider[T], a generic interface where T is the dispatch state type:

@dataclass(frozen=True, kw_only=True)classMyProvider(PipelineProvider[str]):
# Configuration fields...asyncdefdispatch_scanner_tests(...) ->str:
# Dispatch tests, return state for pollingreturnrun_idasyncdefpoll_status(dispatch_state: str) ->Sequence[TestResult] |None:
# Return results when complete, None when still runningreturnresultsifcompleteelseNone

The generic type allows providers to pass any state between dispatch and poll - from a simple run ID to complex objects with multiple identifiers.

Wait Logic

The base class provides wait_for_completion() which handles polling with timeout. Providers only implement the dispatch and poll methods.

GitHub Actions Provider

The GitHubActionsProvider dispatches tests to a GitHub Actions workflow and polls for completion.

Configuration

When calling this action from a GitHub workflow, the provider configuration is passed as a JSON object:

- uses: boostsecurityio/scanner-registry-testing/test-action@mainwith:
provider: github-actionsprovider-config: | { "token": "${{ secrets.TEST_RUNNER_TOKEN }}", "owner": "boostsecurityio", "repo": "test-runner-github", "workflow_id": "scanner-test.yml", "ref": "main" }

Field	Required	Description
`token`	Yes	GitHub token with workflow permissions
`owner`	Yes	Repository owner
`repo`	Yes	Repository name
`workflow_id`	Yes	Workflow file name or ID
`ref`	No	Branch to run workflow on (default: "main")

Dispatch ID for Deterministic Matching

The provider generates a unique dispatch ID (UUID) for each workflow dispatch. This ID is passed as a workflow input and should be displayed in the workflow's run-name. The provider uses this ID to reliably find the correct workflow run among concurrent executions.

Test runner workflow requirements:

Accept dispatch_id as a workflow input
Include the dispatch ID in run-name (e.g., run-name: "[${{ inputs.dispatch_id }}] Scanner Tests")

Session Management

The provider uses a single aiohttp.ClientSession for its lifetime, configured with the base URL and authorization headers. Use the from_config async context manager to ensure proper cleanup.

GitLab CI Provider

The GitLabCIProvider dispatches tests to a GitLab CI pipeline and polls for completion.

Configuration

- uses: boostsecurityio/scanner-registry-testing/test-action@mainwith:
provider: gitlab-ciprovider-config: | { "trigger_token": "${{ secrets.GITLAB_TRIGGER_TOKEN }}", "api_token": "${{ secrets.GITLAB_API_TOKEN }}", "project_id": "boostsecurityio/test-runner-gitlab", "ref": "main" }

Field	Required	Description
`trigger_token`	Yes	GitLab Pipeline Trigger Token for dispatching pipelines
`api_token`	Yes	GitLab Project Access Token (Guest role, read_api scope) for polling
`project_id`	Yes	Project ID (numeric) or path (e.g., "org/project")
`ref`	No	Branch to run pipeline on (default: "main")

Authentication

The provider uses two separate tokens with minimal privileges:

Pipeline Trigger Token (trigger_token): Created in GitLab project settings under CI/CD > Pipeline trigger tokens. Used to dispatch pipelines via the /trigger/pipeline endpoint without requiring any header authentication.
Project Access Token (api_token): Created in GitLab project settings under Access Tokens. Requires:
- Role: Guest (minimum privilege needed)
- Scope: read_api
Used with Bearer token authentication to poll pipeline status via the standard API endpoints.

Pipeline Variables

The provider passes test configuration as pipeline variables:

SCANNER_ID: Scanner being tested (e.g., "boostsecurityio/trivy-fs")
REGISTRY_REF: Git commit SHA of the registry
REGISTRY_REPO: Registry repository in org/repo format
MATRIX_TESTS: JSON array of test matrix entries

Project ID Handling

The project_id can be either:

A numeric ID (e.g., "12345")
A URL-encoded path (e.g., "boostsecurityio%2Ftest-runner")
An unencoded path (e.g., "boostsecurityio/test-runner") - the provider will URL-encode it automatically

Azure DevOps Provider

The AzureDevOpsProvider dispatches tests to an Azure DevOps pipeline and polls for completion.

Configuration

- uses: boostsecurityio/scanner-registry-testing/test-action@mainwith:
provider: azure-devopsprovider-config: | { "token": "${{ secrets.AZURE_PAT }}", "organization": "my-org", "project": "my-project", "pipeline_id": 42 }

Field	Required	Description
`token`	Yes	Azure DevOps Personal Access Token with Build permissions
`organization`	Yes	Azure DevOps organization name
`project`	Yes	Azure DevOps project name
`pipeline_id`	Yes	Pipeline definition ID (numeric)

Template Parameters

The provider passes test configuration as pipeline template parameters:

SCANNER_ID: Scanner being tested (e.g., "boostsecurityio/trivy-fs")
REGISTRY_REF: Git commit SHA of the registry
REGISTRY_REPO: Registry repository in org/repo format
MATRIX_TESTS: JSON array of test matrix entries

Authentication

Azure DevOps uses Basic authentication with an empty username and the Personal Access Token as the password. The token is automatically base64-encoded by the provider.

Bitbucket Pipelines Provider

The BitbucketProvider dispatches tests to a Bitbucket Pipeline and polls for completion.

Configuration

- uses: boostsecurityio/scanner-registry-testing/test-action@mainwith:
provider: bitbucketprovider-config: | { "token": "${{ secrets.BITBUCKET_TOKEN }}", "workspace": "my-workspace", "repo_slug": "test-runner-bitbucket", "branch": "main" }

Field	Required	Description
`token`	Yes	Bitbucket OAuth access token
`workspace`	Yes	Bitbucket workspace slug
`repo_slug`	Yes	Repository slug
`branch`	No	Branch to run pipeline on (default: "main")

Pipeline Variables

The provider passes test configuration as custom pipeline variables:

SCANNER_ID: Scanner being tested (e.g., "boostsecurityio/trivy-fs")
REGISTRY_REF: Git commit SHA of the registry
REGISTRY_REPO: Registry repository in org/repo format
MATRIX_TESTS: JSON array of test matrix entries

Custom Pipeline Selector

The provider triggers a custom pipeline with the pattern test-scanner. Your bitbucket-pipelines.yml should define this custom pipeline:

pipelines:
custom:
test-scanner:
- step:
name: Run Scanner Testsscript:
- echo "Running tests for $SCANNER_ID"

Authentication

Bitbucket uses OAuth Bearer token authentication. The token should have permissions to trigger pipelines on the target repository.

Name		Name	Last commit message	Last commit date
.circleci		.circleci
.github		.github
.makefiles		.makefiles
scan_test_action		scan_test_action
tests		tests
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
AGENT.md		AGENT.md
Makefile		Makefile
README.md		README.md
action.yaml		action.yaml
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Scan Test Action

Overview

Usage

GitHub Action

Action Inputs

Action Outputs

CLI

Development

Prerequisites

Test Orchestration Flow

Usage

Execution Steps

Result Structure

Error Handling

Scanner Detection

How It Works

Git Reference Resolution

Test Definition Format

Fields

Matrix Expansion

Provider Architecture

Plugin System

Provider Interface

Wait Logic

GitHub Actions Provider

Configuration

Dispatch ID for Deterministic Matching

Session Management

GitLab CI Provider

Configuration

Authentication

Pipeline Variables

Project ID Handling

Azure DevOps Provider

Configuration

Template Parameters

Authentication

Bitbucket Pipelines Provider

Configuration

Pipeline Variables

Custom Pipeline Selector

Authentication

About

Topics

Resources

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages