Turn games into verifiable, cost-aware benchmarks for any agent.
Playproof separates four concerns that game benchmarks often blur together:
- Execution. Launch or emulate the game and apply one declared input at a time.
- Observation. Show the evaluated agent only the allowed frame or text representation.
- Evidence. Privately collect engine state, saves, events, rendered state, or platform receipts.
- Verification. Recompute earned milestones and sign the exact run, build, contract, inputs, cost, and latency.
The agent can be an API model, a local policy, Claude Code, Codex CLI, OpenCode, a reinforcement-learning policy, a multi-agent system, or any custom harness. The core interface has no model, provider, orchestration, or sandbox dependency.
pnpm add @tangle-network/playproofNode.js 20.19 or newer is required. Python is needed only by adapters whose worker is implemented in Python, such as PyBoy and the generic desktop worker.
import{generateKeyPairSync}from'node:crypto'import{executeBenchmark,typeAgentDriver}from'@tangle-network/playproof'import{makeNative2048,NATIVE_2048_INPUTS}from'@tangle-network/playproof/adapters/native-2048'// Any BenchmarkTarget works. This one is bundled and needs no ROM.consttarget=makeNative2048(0)// Replace the body with a model call. Return one word from the game's// vocabulary, and what the decision cost.constdriver: AgentDriver={asyncact(frame,history,context){return{input: NATIVE_2048_INPUTS[0]!,costUsd: 0}},}const{ privateKey }=generateKeyPairSync('ed25519')constresult=awaitexecuteBenchmark(target,driver,{budgetUsd: 1,maxTurns: 100,actor: {kind: 'agent',id: 'my-agent-v1'},signer: { privateKey,keyId: 'benchmark-recorder-v1'},})console.log(result.record.verified)// [ 'first-legal-move', 'first-merge' ]console.log(result.signed)BenchmarkTarget pins the game, exact build digest, platform capabilities, milestone contract, and reference inputs. executeBenchmark uses the shared episode engine, records every decision and measured cost, recomputes progression, and emits an Ed25519-signed publication envelope.
An agent is anything that implements one method.
interfaceAgentDriver{act(frame: string,history: readonly{input: string;frame: string}[],context: {turn: numbermaxTurns: numberseed: numberspentUsd: numberremainingBudgetUsd: numberguidance?: stringobservation?: {text: string;images?: readonlyObservationImage[]}signal?: AbortSignal},): Promise<{input: string;costUsd: number}>}No model, provider, orchestration or sandbox dependency. An API model, a local policy, Claude Code, Codex CLI, OpenCode, a trained policy and a multi-agent system all reach the game the same way. The verifier cannot tell them apart.
Ready-made drivers ship for an OpenAI-compatible endpoint, any CLI, a persistent process, and an asynchronous sandbox. See docs/agent-drivers.md.
One benchmark says how an agent did on one game. An arena asks the question a single game cannot: does the ranking survive a change of game?
profile.opus harness=./harnesses/claude-code model=claude-opus-5 transport=persistent
profile.greedy harness=none policy=./policies/greedy transport=persistent
game.puzzle adapter=native-2048 target=2048
objective.score goal=maximize:score horizon=2000 budgetUsd=5
protocol.det frameskip=1 sticky=0 seeds=1
sensor.ascii pixels=off channels=-
reps 3
npx tsx matrix.mts study.matrix --out runs/study/cells.jsonA cell is one profile, one game, one objective, one protocol, one sensor. None of those has a default, because each one changes what the number means. The runner plays every cell, attests every replay, and reports mean pairwise Kendall tau-b between the per-game rankings.
Two facts the arena exists to keep visible:
- How an agent is asked for a decision changes its score more than the model does. The same program scores 4 under one process per decision and 1948 under one process per episode.
- A profile can be asked to build a player instead of playing one. It gets a practice game and a build budget, leaves an executable behind, and that program is scored cold. Build cost is reported apart from the score.
See docs/arena.md.
An episode has three stop conditions, and the record names the one that fired.
record.stoppedBy | What happened |
|---|---|
maxTurns | The turn limit was reached. |
budget | The dollar budget was reached. |
gameOver | The game reported that it is finished. |
The game-over stop is opt-in, because episode length is a denominator. Two rounds of a study compare only if both played to the same turn limit.
const{ record }=awaitplayEpisode(game,contract,driver,budgetUsd,maxTurns,seed,signal,{stopAtGameOver: true,})Every record carries gameOver whether or not the stop is armed. A record that
says stoppedBy: 'maxTurns' next to gameOver: true kept paying for decisions
after the game was over. On ALE Breakout that was 150 of 300 decisions, and
every evidence channel was byte-identical across them.
See docs/episodes.md.
A game can be benchmarked when a bridge provides:
- a finite input vocabulary;
- an observation channel;
- at least one progression signal the agent cannot directly author; and
- an honest verification declaration.
Reference trajectories can come from a human, a scripted policy, another agent, platform events, or deterministic frontier exploration. Blind discovery can identify candidate changing memory channels before a game-specific semantic adapter is written.
“Any game” does not mean zero integration. It means the benchmark core remains unchanged while the platform bridge declares inputs, observations, evidence, trust, and calibration. Anti-cheat-protected or networked games may only permit platform receipts or approved title-side instrumentation.
| Agent drivers | The four transports, and writing your own. |
| Arena | Profiles, games, protocols, sensors, authoring, transfer. |
| Adapters | Every bridge, what it proves, and what it does not. |
| Observation | Text and pixels, and where the boundary sits. |
| Verification | Modes, evidence tiers, milestone contracts, signing. |
| Calibration | Whether a contract separates skill from noise. |
| Episodes | Stop conditions and the game-over rule. |
| Long-horizon runs | Segments, steering, resume, analysts. |
| Releasing | How a version is cut and published. |
Playproof does not implement DLL injection, runtime patching, anti-cheat bypasses, credential extraction, arbitrary memory writes, network manipulation, or shell-string evaluation.
Unknown and control-character desktop inputs are no-ops. Worker frames, helper output, HTTP bodies, save files, and event files are bounded while being read. Credentials remain in caller-owned driver or adapter configuration and are not included in signed benchmark artifacts.
Observation images are bounded the same way: a declared media type checked against the file's own magic bytes, canonical base64, a per-image byte and dimension cap, and a per-turn total. A breach fails the turn instead of quietly showing the agent something smaller.
Please report vulnerabilities through the process in SECURITY.md.
corepack enable
corepack prepare pnpm@11.17.0 --activate
pnpm install
pnpm ciThe release gate runs boundary checks, strict typechecking, adversarial tests, a production build, an exact packed-tarball audit, and clean-consumer imports for every public subpath.
Real PyBoy controls additionally require a legally obtained ROM matching the pinned reference identity:
PLAYPROOF_ROM=/legal/path/Tetris.gb \
PLAYPROOF_PYTHON=python \
pnpm test:pyboyThe stable-retro and ALE gates need no ROM, because both emulators ship one:
pip install stable-retro
PLAYPROOF_REQUIRE_RETRO=1 pnpm test:retro
pip install ale-py
PLAYPROOF_REQUIRE_ALE=1 pnpm test:aleApache-2.0.
If you use Playproof in your work, please cite it:
@software{stone_yaish_playproof,
author = {Stone, Drew and Yaish, Aviv},
title = {Playproof},
url = {https://github.com/tangle-network/playproof}
}Machine-readable citation metadata is available in CITATION.cff.