Paddock

Automated eval & improvement loop for AI agents. Generates test scenarios, runs the agent, scores with multi-model consensus (Claude + Gemini + GPT), and iteratively patches code until quality targets are met.

Judges run in one of two auth modes, picked per-deployment:

• Direct API (default for public users) — Claude / Gemini / GPT via their respective vendor APIs.
• Vertex AI (for compliance-aligned / GCP-aligned deployments) — Claude + Gemini via Google Cloud Vertex AI with Application Default Credentials. The GPT judge stays direct (OpenAI has no Vertex offering).

How It Works

Scenarios (.yml) → Agent Runtime (mock channel) → 3 LLM Judges → Consensus
│
pass ≥ 80%?
/ \
YES NO
│ │
git push Analyze + Patch
│
Sandbox OK?
/ \
YES NO
│ │
Commit Revert
│
← repeat

1. Load scenarios from .paddock/scenarios/ in the target project (YAML files organized by category)
2. Run each scenario against the agent via a mock channel — captures responses, tool calls, errors, timing
3. Up to 3 LLM judges (Claude, Gemini, GPT) — registered based on which auth paths are configured — independently score each run on correctness, tool usage, SOUL compliance, response quality, error handling. Vertex-mode deployments typically run 2 (Claude + Gemini via Vertex); direct-mode users typically run all 3
4. Consensus: median scores + majority vote → pass/fail/partial
5. If failing: analyzer finds patterns, patcher generates code fixes, sandbox validates (type-check + build)
6. Repeat until pass rate ≥ threshold or budget exhausted
7. Git: all work on eval/* branches, push on success

Install

git clone https://github.com/cleanslice/paddock.git
cd paddock
bun install
cp .env.example .env
# Add your API keys to .env

Requirements

• Bun runtime
• At least one judge configured — either:
  • Direct API: an LLM API key (Claude preferred, Gemini / GPT optional for multi-judge consensus), or
  • Vertex AI: VERTEX_PROJECT_ID + VERTEX_REGION plus the optional peer deps @anthropic-ai/vertex-sdk + @google/genai

Environment Variables

Paddock's Claude and Gemini judges run in one of two modes, auto-detected from environment. Pick whichever fits your deployment:

Direct API mode (default for public users)

# Required (at least one)
CLAUDE_CODE_OAUTH_TOKEN=token1,token2,token3 # Comma-separated, auto-rotation on rate limit
ANTHROPIC_API_KEY=sk-ant-... # Fallback# Optional (for 3-judge consensus)
GEMINI_API_KEY=AIza...
OPENAI_API_KEY=sk-...

Vertex AI mode (for compliance-aligned / GCP-aligned deployments)

When Vertex env is set, the Claude and Gemini judges skip API-key auth and authenticate via Google Cloud Application Default Credentials (gcloud auth application-default login locally, Workload Identity Federation in production). API keys are still required for the OpenAI judge — OpenAI has no Vertex offering.

# Required — both must be set to activate Vertex mode
VERTEX_PROJECT_ID=your-gcp-project
VERTEX_REGION=us-east5
# Optional — declare the full Vertex judge panel as a comma-separated list# of model IDs (max 3 entries). Lets you run multiple Claude judges (e.g.# Sonnet + Opus) for 3-judge consensus without needing OpenAI. When unset,# paddock defaults to 1 Claude (EVAL_CLAUDE_JUDGE_MODEL) + 1 Gemini# (EVAL_GEMINI_JUDGE_MODEL).
VERTEX_JUDGES=claude-sonnet-4-6,claude-opus-4-7,gemini-2.5-pro
# Optional — adds an OpenAI judge alongside the Vertex ones (always direct).# Omit for FedRAMP-strict deployments that can't call api.openai.com.
OPENAI_API_KEY=sk-...

The env-var names are deliberately platform-named (VERTEX_*) rather than vendor-named: in this mode Vertex hosts both judge providers (Claude via @anthropic-ai/vertex-sdk, Gemini via @google/genai), so a single platform-named env namespace gates both judges symmetrically.

Provider for each entry in VERTEX_JUDGES is inferred from the model-name prefix: claude-* → Anthropic-on-Vertex, gemini-* → Google-on-Vertex. Putting a gpt-* model in the list errors fast at startup — OpenAI is not on Vertex; use OPENAI_API_KEY to add an OpenAI judge.

Vertex mode requires installing two optional peer dependencies (paddock declares them as optional so direct-API users don't pay the install cost):

npm install @anthropic-ai/vertex-sdk @google/genai

Other overrides

EVAL_REPO_ROOT=/path/to/agent-repo
EVAL_AGENT_DIR=/path/to/agent-repo/.agent
EVAL_LLM_MODEL=claude-sonnet-4-6
EVAL_CLAUDE_JUDGE_MODEL=claude-sonnet-4-6
EVAL_GEMINI_JUDGE_MODEL=gemini-2.5-pro
EVAL_OPENAI_JUDGE_MODEL=gpt-4o
# Judge tuning — applies to every judge (direct + Vertex)
EVAL_JUDGE_THINKING_BUDGET=8000 # Reasoning tokens per call. 0 disables# thinking on Claude/Gemini (OpenAI reasoning# models clamp to "medium").
EVAL_JUDGE_MAX_TOKENS=16000 # Max output tokens per judge call. Generous# on purpose so thinking + final scored# output fit on hard scenarios.

Usage

CLI

# Full eval loop (10 scenarios, auto-improve, git branch + push)
bun run eval --repo /path/to/agent-repo
# Quick smoke test (3 scenarios, no improvement)
bun run eval:quick --repo /path/to/agent-repo
# Evaluate only, no code changes
bun run eval:no-improve --repo /path/to/agent-repo
# Test specific category
bun run eval:category tool_use --repo /path/to/agent-repo
# Preview loaded scenarios
bun run scenarios --repo /path/to/agent-repo

All CLI Flags

--repo Path to agent runtime repo (required or set EVAL_REPO_ROOT)
--agent-dir Path to .agent directory
--categories Comma-separated: tool_use,memory,conversation,edge_case,multi_turn,error_recovery
--difficulties Comma-separated: easy,medium,hard,adversarial
--count Number of scenarios (default: 10)
--threshold Pass rate 0-1 (default: 0.8)
--max-iter Max improvement iterations (default: 5)
--no-improve Evaluate only, skip auto-improvement
--no-push Don't push to git

MCP Server

Paddock runs as an MCP server so your agent can call it directly:

bun run mcp

Available MCP tools:

Tool	Description
eval_run	Run full eval cycle
eval_status	Check progress
eval_report	Get detailed results
eval_scenarios	Preview scenarios
eval_abort	Stop running eval

Scenarios

Scenarios live in .paddock/scenarios/ inside the target project (the agent being tested).

Setup

# In your agent project:
cp -r /path/to/paddock/.paddock.example .paddock

Or paddock will auto-copy .paddock.example on first run if .paddock/ doesn't exist.

Directory Structure

your-agent-project/
└── .paddock/
├── config.json # Eval settings (optional)
└── scenarios/
├── tool_use/
│ ├── web-search.yml
│ └── file-ops.yml
├── conversation/
├── memory/
├── multi_turn/
├── edge_case/
└── error_recovery/

Scenario Format

id: tool-web-search-basiccategory: tool_usedifficulty: easyname: Basic web searchdescription: User asks to search the webexpectedBehavior: Agent calls web_search immediately, summarizes resultsmessages:
- text: "What's the weather in London?"from: eval-user
- text: "And in Tokyo?"from: eval-userdelayMs: 2000successCriteria:
- dimension: correctnessdescription: Used search tool and returned resultsweight: 0.4
- dimension: tool_usagedescription: Chose web_search toolweight: 0.3
- dimension: response_qualitydescription: Concise summaryweight: 0.3

Fields

Field	Required	Description
id	yes	Unique kebab-case identifier
category	yes	tool_use, memory, conversation, multi_turn, edge_case, error_recovery
difficulty	yes	easy, medium, hard, adversarial
name	yes	Short human-readable name
description	yes	What this scenario tests
expectedBehavior	yes	What the agent should do
messages	yes	Array of {text, from, delayMs?} — the "user" messages
successCriteria	yes	Array of {dimension, description, weight} — weights sum to 1.0

Dimensions

Dimension	What It Measures
correctness	Did the agent produce the right result?
tool_usage	Did it pick the right tools with correct params?
soul_compliance	Does the response match SOUL.md personality?
response_quality	Is the response clear, well-structured?
error_handling	How did it handle errors or edge cases?

Development

Project Structure

paddock/
├── src/
│ ├── index.ts # MCP server entry (stdio)
│ ├── cli.ts # CLI entry
│ ├── types.ts # All shared types
│ ├── runner/
│ │ ├── mock-channel.ts # In-memory channel for testing
│ │ └── agent-runner.ts # Boots agent, sends scenarios, captures traces
│ ├── scenario/
│ │ ├── loader.ts # YAML file loader from .paddock/
│ │ └── generator.ts # LLM-based scenario generation
│ ├── evaluator/
│ │ ├── judge.ts # Single-model judge
│ │ ├── consensus.ts # Multi-model consensus (median + majority vote)
│ │ ├── criteria.ts # Scoring prompt template
│ │ └── providers/ # Claude, Gemini, OpenAI judge implementations
│ ├── improver/
│ │ ├── analyzer.ts # Failure pattern detection
│ │ ├── patcher.ts # LLM-generated code patches
│ │ └── sandbox.ts # Type-check + build validation
│ ├── git/
│ │ └── branch-manager.ts # Git branch operations
│ ├── loop/
│ │ ├── orchestrator.ts # Main eval loop state machine
│ │ └── budget.ts # Iteration/time/cost limits
│ └── mcp/
│ └── server.ts # MCP tool definitions
├── .paddock.example/ # Default scenarios (copied to target projects)
├── package.json
└── tsconfig.json

Key Concepts

Mock Channel — Implements IChannelGateway from the agent runtime. Captures all outbound messages. Provides simulateIncoming() and waitForAllResponses() for programmatic testing.

Tracing Proxy — Wraps each agent tool to record params, results, timing, errors. Dangerous tools (exec, shutdown, etc.) are blocked and return stubs.

Consensus Engine — Runs N judges in parallel. Per-dimension score = median (robust to outliers). Verdict = majority vote. Agreement < 50% → "partial" (flagged for human review).

Patcher Safety — Can only modify files matching allowlist (.agent/SOUL.md, src/slices/**/*.ts, etc.). Max 200 lines per patch. Type-check gate after every patch. Auto-revert on failure.

Commands

bun run eval# Full eval loop
bun run eval:quick # 3 scenarios, no improve
bun run eval:full # 10 scenarios with improve
bun run scenarios # Preview scenarios
bun run mcp # Start MCP server
bun run typecheck # Type-check

Adding a Judge Provider

Three steps:

1. Create src/evaluator/providers/your-provider.ts implementing JudgeProvider:

importtype{JudgeProvider,JudgePrompt,TokenUsage}from"../../types"exportclassYourJudgeProviderimplementsJudgeProvider{name="your-model"model: stringusage: TokenUsage={inputTokens: 0,outputTokens: 0,totalTokens: 0}constructor(apiKey: string,model="your-model-id"){this.model=model}asynccomplete(prompt: string|JudgePrompt): Promise<string>{// Call your LLM API, return raw text. Populate this.usage along the way.}}

2. Add a variant to the JudgeProviderConfig discriminated union in src/types.ts:

exportinterfaceYourJudgeConfig{type: "your-provider"model: stringapiKey: string}exporttypeJudgeProviderConfig=|ClaudeJudgeConfig|ClaudeVertexJudgeConfig|GeminiJudgeConfig|GeminiVertexJudgeConfig|OpenAIJudgeConfig|YourJudgeConfig// ← add here

3. Add a case to the factory's switch in src/evaluator/providers/factory.ts:

case"your-provider":
returnnewYourJudgeProvider(config.apiKey,config.model)

TypeScript's exhaustive never check in the factory's default branch will flag any new variant that lacks a case at build time.

To register the new judge from CLI / MCP based on env vars, mirror the buildJudgeConfigs pattern in src/cli.ts and src/mcp/server.ts.

Runtime Integration

Paddock requires a small change in the target agent's channel system — adding a mock channel type:

// In channel.types.ts:exporttypeChannelConfig=|{type: "telegram";token: string}|{type: "slack";botToken: string;appToken: string}|{type: "mock";instance: IChannelGateway}// ← for Paddock// In channel.gateway.ts:case"mock":
returnconfig.instance

License

MIT