testing-utils

Test framework tools and helpers for performance stack project.

Overview

This repository provided utilities to assist with test automation, log handling, and result parsing. It is designed to be a set of helper libraries for test frameworks or custom test runners.

Logging

testing-utils uses Python's standard logging module. All modules log via a package logger, which you can configure in your application. By default, the logger uses a NullHandler. To see logs, configure logging in your main script:

importlogginglogging.basicConfig(level=logging.INFO)

or provide logging config in pytest configuration:

log_cli = true
log_cli_level = DEBUG
log_cli_format = %(asctime)s %(levelname)s %(message)s
log_cli_date_format = %Y-%m-%d %H:%M:%S

You can also attach handlers or change the log level for the testing_utils logger specifically:

importlogginglogger=logging.getLogger("testing_utils")
logger.setLevel(logging.DEBUG)

Features

• Test scenarios libraries: Rust and C++ libraries for implementing test scenarios.
• Build tools: Utilities for interacting with Bazel and Cargo.
• Log container: A container for storing and querying logs.
• ResultEntry and subclasses: Structured representation of test log entries.
• Scenario: Utilities for defining and running test scenarios.

Installation

virtualenv usage is recommended:

python -m venv .venv
source .venv/bin/activate

Install testing-utils:

pip install . --config-settings editable_mode=strict

Install testing-utils in editable mode with additional dev dependencies:

pip install -e .[dev] --config-settings editable_mode=strict

--config-settings editable_mode=strict is required by Pylance plugin in VS Code. Package will work without it, but autocompletion won't work properly.

Usage

Test scenarios utilities

Libraries should be included as a dependency in Cargo.toml (Rust) or BUILD (C++).

Main components:

• TestContext - responsible for listing and running scenarios.
• Scenario and ScenarioGroup - base classes for defining scenarios and groups.
• run_cli_app - runs CLI application based on provided arguments and test context.

Build tools

Get Cargo metadata

Cargo metadata is obtained using "cargo metadata" command. CWD must be set to Cargo project.

fromtypingimportAnyfromtesting_utilsimportcargo_metadatametadata: dict[str, Any] =cargo_metadata()

Find target path

Find path to executable based on provided target name.

frompathlibimportPathfromtesting_utilsimportBazelToolstarget_name="target_name"build_tools=BazelTools()
target_path: Path=build_tools.find_target_path(target_name)
...

Select executable

This feature is to ensure flexible usage in pytest context. Additional configuration is required.

Expected flags depend on implementation, refer to select_target_path docs. Add options to conftest.py:

frompathlibimportPathdefpytest_addoption(parser):
parser.addoption(
"--target-path",
type=Path,
help="Path to test scenarios executable. Search is performed by default.",
)
parser.addoption(
"--target-name",
type=str,
default="rust_test_scenarios",
help='Test scenario executable name. Overwritten by "--target-path".',
)

Usage:

importpytestfrompathlibimportPathfromtesting_utilsimportBazelToolsdeftest_example(request: pytest.FeatureRequest) ->None:
build_tools=BazelTools()
target_path: Path=build_tools.select_target_path(request.config)
...

Run build

Run build for selected target.

fromtesting_utilsimportBazelToolstarget_name="target_name"build_tools=BazelTools()
target_path: Path=build_tools.build(target_name)
...

Log container and ResultEntry example

Usage as container:

fromtesting_utilsimportLogContainer, ResultEntrylc=LogContainer()
lc.add_log(
ResultEntry({
"timestamp": "2025-06-05T07:46:11.796134Z",
"level": "DEBUG",
"fields": {"message": "Debug message"},
"target": "target::DEBUG_message",
"threadId": "ThreadId(1)",
})
)
logs=lc.get_logs()
# Output: debug message.print(logs[0].message)

Usage as JSON log trace parser:

fromtesting_utilsimportLogContainer, ResultEntry# "messages" is a list of JSON logs.logs= [ResultEntry(msg) formsginmessages]
lc=LogContainer(logs)
lc.contains_log(field="message", pattern="SomeExampleAction") # True

Usage as log filter:

fromtesting_utilsimportLogContainer, ResultEntry# "messages" is a list of JSON logs.logs= [ResultEntry(msg) formsginmessages]
lc=LogContainer(logs)
lc_only_info=lc.get_logs_by_field(field="level", pattern="INFO")

Scenario example

Scenario is a base class containing basic test behavior. Test execution results are provided using two fixtures:

• results - executable run results
• logs - logs from run

build_tools, scenario_name and test_config are marked as abstract and must be implemented.

Example implementation:

importpytestfromtesting_utilsimportScenario, ScenarioResult, LogContainer, CargoTools, BuildToolsclassTestExample(Scenario):
@pytest.fixture(scope="class")defbuild_tools(self) ->BuildTools:
returnCargoTools()
@pytest.fixture(scope="class")defscenario_name(self) ->str:
return"example_scenario_name"@pytest.fixture(scope="class")deftest_config(self) ->dict[str, Any]:
return {"runtime": {"task_queue_size": 256, "workers": 1}}
deftest_example(self, results: ScenarioResult, logs: LogContainer) ->None:
...

Execution timeout uses "--default-execution-timeout" set in conftest.py, or is set to 5 seconds by default.

stderr is shown, but not captured by default. To capture stderr use:

classTestExample(Scenario):
defcapture_stderr(self) ->bool:
returnTrue
...

Methods can be overridden to utilize test-specific fixtures:

importpytestfromtesting_utilsimportScenarioclassTestExample(Scenario)
@pytest.fixture(scope="class", params=[1, 4, 256])defqueue_size(self, request: pytest.FixtureRequest) ->int:
returnrequest.param@pytest.fixture(scope="class")deftest_config(self, queue_size: int) ->dict[str, Any]:
return {"runtime": {"task_queue_size": queue_size, "workers": 1}}
...

Development

• Python 3.12+ required.
• Code style is enforced with ruff.

To run the tests, use:

pytest -vs .