Skip to content

Testing

Jump to bottom
Daniel Ellison edited this page Sep 3, 2026 · 1 revision

Testing

Kai has roughly 5,900 tests across 167 test modules. This page explains how to run them, how the suite and CI are organized, and what patterns to follow when writing new tests.

Running tests and checks

make test# full backend test suite (pytest, verbose)
make lint # ruff check
make check # ruff check + ruff format --check
make format # ruff format (auto-fix)
make typecheck # pyright, strict
make client-check # Workshop client: typecheck + vitest + verify-generated

Or directly:

.venv/bin/pytest tests/
.venv/bin/pytest tests/test_bot.py -k "test_help"# run specific tests
.venv/bin/pytest tests/ -x # stop on first failure

Auxiliary checks, all run by CI:

make check-install-constraints # pip dry-runs against the constraints file
make module-sizes # module size budget (scripts/module-sizes.py)
make audit-deps # pip-audit over installed dependencies

Suite structure

The suite has three broad tiers rather than a one-file-per-module mapping:

  • Core: the classic surface: test_bot.py (plus test_bot_totp.py), test_config.py, test_claude.py, test_pool.py, test_webhook.py / test_webhook_api.py, test_install.py, test_review.py, test_triage.py, test_sessions.py, test_history.py, voice, services, locks, and friends.
  • Memory: the semantic memory subsystem and its eval harnesses.
  • Workshop: about 81 test_workshop_*.py modules covering the canonical core (store, projections, channels, agents, runs, delivery, memory queries, settings, scheduling, client API). Shared helpers live in tests/workshop_delivery.py and tests/workshop_profiles.py.

tests/conftest.py provides global fixtures that apply to every test automatically, giving safety guarantees individual tests don't need to remember to set up.

The Workshop browser client has its own suite: make client-check runs the TypeScript typecheck, vitest, and verify-generated, which rebuilds the client bundle and fails if the committed output under the static directory drifts from source. The built bundle is committed, so a client change without a rebuild fails CI.

CI topology

CI runs five jobs on every push and PR:

  1. changes: classifies the changed paths with scripts/ci_change_scope.py into client / full / dependency outputs. Docs-only changes take a fast path; client-only PRs run the lighter client lane.
  2. quality: install constraints, client check, lint and format check, type check.
  3. tests: a three-way matrix (core, memory, workshop) sharded by scripts/ci_test_shard.py, so the suite runs in parallel lanes.
  4. check: the required-lane gate that branch protection points at.
  5. core-workshop: installs without the telegram extra and runs the Workshop lane, proving the Telegram-free boundary holds.

The CI scripts have their own regression tests (tests/test_ci_change_scope.py, tests/test_ci_test_shard.py), and .github/CI.md documents the workflow in the repo. A separate scheduled workflow (dependency-audit.yml) runs the dependency audit.

Common patterns

Filesystem isolation

Tests that touch the filesystem use pytest's tmp_path fixture. Sessions, config, and history tests create real SQLite databases and config files in temp directories rather than mocking the filesystem. This catches real path-handling bugs that mocks would hide.

Async testing

Most of Kai's code is async. The suite sets asyncio_mode = "auto" in pyproject.toml, so plain async def test functions just work; no pytest.mark.asyncio marker is needed. Use AsyncMock from unittest.mock for async collaborators:

asyncdeftest_something(self):
mock_fn=AsyncMock(return_value="result")
# ...

Telegram handler mocking

Bot tests use factory functions to create mock Telegram Update and Context objects:

  • _make_update(text, user_id) - creates a mock Update with a message
  • _make_context(bot) - creates a mock CallbackContext

These set up the minimum attributes handlers need (message text, user ID, chat ID, bot instance) without pulling in the full python-telegram-bot object graph.

Subprocess mocking

Backend, transcribe, and TTS tests mock asyncio.create_subprocess_exec (or subprocess.run for sync code) to simulate subprocess behavior without actually running external binaries. This makes tests fast and deterministic.

HTTP mocking

Service proxy tests use aioresponses to mock outbound HTTP requests:

withaioresponses() asmocked:
mocked.post("https://api.example.com/v1/chat", payload={"result": "ok"})
# call the service proxy...

Attribute patching

monkeypatch.setattr is the preferred way to patch module-level attributes and globals. It's cleaner than unittest.mock.patch for module globals and auto-restores after each test.

Writing new tests

  1. Put tests in the corresponding file: a change to webhook.py gets tests in test_webhook.py or test_webhook_api.py; a Workshop module gets a test_workshop_<area>.py home.
  2. Group with classes: use class TestFeatureName to organize related tests.
  3. Write plain async def tests; the auto asyncio mode handles the rest.
  4. Use real databases when testing database interactions (via tmp_path); mock databases only when the test isn't about database behavior.
  5. Never use real timeouts: mock time or use tiny values, so the suite stays fast and deterministic.
  6. Add docstrings to tests that verify non-obvious behavior, and follow the commenting style used in the existing tests.
  7. Client changes rebuild the bundle: run make client-check before pushing anything that touches workshop-client/; CI fails on a stale committed bundle.

Clone this wiki locally