Typeberry Testing

E2E tests for Typeberry - a JAM node implementation by Fluffy Labs.

Check out our performance statistics over time at typeberry.fluffylabs.dev.

Status

Test Category	Status	Description
Docker Works		Tests Docker image functionality and basic operations
Docker Imports		Docker image is able to import standard block dumps
Docker Imports Full		Imports the 100k-block full-chainspec dump (nightly; dump fetched from a release asset). Multi-hour soak — runs on a dedicated beefier non-perf runner.
Docker Conformance		Tests JAM conformance using Docker with latest conformance test suite
Docker Test Vectors		Tests W3F test vectors using Docker with latest test suite
NPM Works		Tests NPM package installation and basic functionality
NPM Imports		NPM package (in a thin docker image) imports standard block dumps
NPM Minifuzz Tests		Minifuzz against the NPM package (thin docker image)
Picofuzz Fallback		Tests fallback functionality using prepared fuzz messages
Picofuzz Safrole		Tests Safrole protocol implementation with fuzzing
Picofuzz Storage		Tests storage functionality with comprehensive fuzzing
Picofuzz Storage Light		Tests storage functionality with lightweight fuzzing
Picofuzz Full Chain		Imports the 100k-block full-chainspec dump via the fuzz protocol (nightly, perf stats). Multi-hour soak — runs on a dedicated beefier non-perf runner.
Minifuzz Burn		Burn-in testing for extended fuzzing operations
Minifuzz Forks		Tests fork handling and process management
Minifuzz No Forks		Tests single-process operation without forking

Standalone tools

• picofuzz
• perf-graph

Running Tests

Prerequisites

• Node.js 20+
• Docker
• npm

Setup

# Clone the repository with recursive submodules
git clone --recursive https://github.com/FluffyLabs/typeberry-testing.git
# Or if already cloned, initialize submodules recursively
git submodule update --init --recursive
# Install dependencies
npm install
# Provision the typeberry image under test (tagged `typeberry:test`).# TARGET: docker (published image) | npm (npm package, wrapped in a docker image) | source (build w/ test-runner)# VERSION: defaults to `next` (latest main); pin with `<semver>-<sha>` or `<semver>`.
TARGET=npm bash .github/actions/provision-typeberry/provision.sh

Submodules

This repository uses the following submodules:

• picofuzz-conformance-data JAM conformance traces for picofuzz execution.
• picofuzz-stf-data JAM test vectors for picofuzz execution.
• picofuzz-full-chain-data Full-chainspec 100k-block fuzz-message dataset (~150 MB). Not initialized by default (update = none); opt in with: git -c submodule.picofuzz-full-chain-data.update=checkout submodule update --init picofuzz-full-chain-data

Running All Tests

npm test

Note:npm test runs all*.test.ts files, including the ~1.5 h full-chain dump tests below — prefer running individual test files.

Running Individual Tests

# Docker functionality tests
npm exec tsx --test tests/docker-works.test.ts
# Docker conformance tests
npm exec tsx --test tests/docker-conformance.test.ts
# Docker test vectors
npm exec tsx --test tests/docker-test-vectors.test.ts
# Picofuzz tests
npm exec tsx --test tests/picofuzz/fallback.test.ts
npm exec tsx --test tests/picofuzz/safrole.test.ts
npm exec tsx --test tests/picofuzz/storage.test.ts
npm exec tsx --test tests/picofuzz/storage_light.test.ts
# Minifuzz tests
npm exec tsx --test tests/minifuzz/burn.test.ts
npm exec tsx --test tests/minifuzz/faulty.test.ts
npm exec tsx --test tests/minifuzz/forks.test.ts
npm exec tsx --test tests/minifuzz/no_forks.test.ts
# Full-chain dump tests (long: ~1.5h each; dump fetched via block-dumps/full/fetch.sh)
npm exec tsx --test tests/docker-imports-full.test.ts
npm exec tsx --test tests/picofuzz/full_chain.test.ts

Running Picofuzz

Picofuzz is a lightweight fuzzing tool that sends prepared fuzz messages using the Fuzz protocol:

# Run picofuzz directly
npm start -w @fluffylabs/picofuzz [options] <directory><socket># Options:# -f, --flavour <spec> JAM spec: tiny | full (default: tiny)# -r, --repeat <count> Number of repetitions (default: 1)# -s, --stats <file> Append aggregated stats to a CSV file# -h, --help Show help# Examples:
npm start -w @fluffylabs/picofuzz picofuzz-stf-data/picofuzz-data/fallback /tmp/jam_target.sock
npm start -w @fluffylabs/picofuzz -r 10 picofuzz-stf-data/picofuzz-data/safrole /tmp/jam_target.sock
npm start -w @fluffylabs/picofuzz -s results.csv picofuzz-stf-data/picofuzz-data/storage /tmp/jam_target.sock
See more details about [picofuzz](./picofuzz).
# Using Dockercd picofuzz
docker build -t picofuzz .
docker run picofuzz [options] <directory><socket>

PR Benchmark Workflow

The PR Benchmark workflow allows you to test and benchmark PRs from the typeberry repository before merging. It runs the complete picofuzz test suite against a specific Docker image build and compares the results with baseline performance metrics.

Usage

Manual Trigger:

1. Go to Actions > PR Benchmark
2. Click "Run workflow"
3. Enter the PR number from fluffylabs/typeberry (e.g., 704)

The workflow will automatically find the latest successful build-docker workflow run for that PR.

Automated Trigger:

curl -X POST \
-H "Authorization: token YOUR_PAT" \
https://api.github.com/repos/FluffyLabs/typeberry-testing/dispatches \
-d '{"event_type": "benchmark-pr", "client_payload": {"pr_number": "704"}}'

Requirements:

• TYPEBERRY_PAT secret with permissions to read artifacts and post comments to fluffylabs/typeberry

Project Structure

├── tests/
│ ├── docker-works.test.ts # Docker image functionality tests
│ ├── docker-conformance.test.ts # JAM conformance tests using Docker
│ ├── docker-test-vectors.test.ts # W3F test vectors using Docker
│ ├── picofuzz/ # Performance testing
│ │ ├── common.ts # Common utilities for picofuzz tests
│ │ ├── fallback.test.ts # Fallback performance
│ │ ├── safrole.test.ts # Safrole performance
│ │ ├── storage.test.ts # Storage performance
│ │ └── storage_light.test.ts # Lightweight storage performance
│ └── minifuzz/ # Minifuzz compatibility
│ ├── burn.test.ts # Repeated execution
│ ├── faulty.test.ts # Fault tolerance tests
│ ├── forks.test.ts # Fork handling tests
│ └── no_forks.test.ts # Single-chain tests
├── picofuzz/ # Performance testing
│ ├── index.ts # Main entry point
│ ├── args.ts # Argument parsing
│ ├── files.ts # File processing utilities
│ ├── socket.ts # Socket communication
│ ├── stats.ts # Statistics collection
│ └── package.json # Package configuration

Contributing

This repository contains end-to-end tests for the Typeberry project. Each test suite runs in isolation as separate CI jobs to provide granular feedback on different aspects of the system.

License

Mozilla Public License 2.0 (MPL-2.0)