SwarmBench

Adversarial multi-agent benchmarking system for prompts and models.

SwarmBench pits multiple AI agents against each other in structured adversarial debates to benchmark prompt and model performance across multiple dimensions. Each agent represents a unique (model, prompt) pair with immutable beliefs and emotion biases, generating claims, attacking others' claims, refining under pressure, and being scored by a deterministic evaluator.

Core insight: You are not orchestrating reasoning. You are orchestrating competing preference gradients. Truth emerges as the least breakable structure.

Bring your own agents and skills. SwarmBench is designed so that anyone can drop any Claude Code agent definition (.md) or skill into the debate. If you have a custom agent or skill — for any domain, workflow, or opinion — it can argue alongside (or against) the built-in swarm.

Core Properties

• Adversarial reasoning -- agents argue, attack, and defend claims under pressure
• Multi-agent swarm dynamics -- any number of (model, prompt) pairs compete simultaneously
• Open agent roster -- any agent or skill definition can join the debate; no fixed roster beyond the 7 core swarm roles
• Evaluation rigor -- deterministic metrics, conflict heuristics, and drift detection
• Anti-deception -- belief immutability, forced conflict, arbiter blindness
• Self-tuning -- 9-step self-improving loop with A/B shadow testing
• Cost transparency -- per-run token usage and USD cost breakdown (input, output, cache read/write)

Documentation

Architecture

Document	Description
System Architecture	Full architecture: swarm core, embeddings, evaluator, self-improving loop
Metrics & Heuristics	All scoring formulas, drift detection rules, threshold logic, stability controls
Self-Improving Loop	9-step update cycle, A/B shadow runs, audit trail, emotion injection
Retry Controller	4-level enforcement engine, target thresholds, data logging
Anti-Deception Layer	Belief immutability, forced conflict, arbiter blindness, sincerity checks

Design

Document	Description
Agent Definitions	7 core swarm agents, beliefs, emotions, global constraints, token controls
Skill Definitions	5 constrained skills with strict I/O schemas and fail conditions
Command Definitions	Phase orchestration, anonymization, execution flow
Benchmarking System	Multi-model + multi-prompt benchmarking design
Real-Time Chat Window	Live debate viewer with threading, filtering, metrics overlay
Dashboard Wireframe	Full dashboard layout: tiles, heatmaps, conflict graph, controls

Plans

Document	Description
Implementation Plan	7-phase build plan with milestones and deliverables
Task DAG	80 tasks across 7 phases with full dependency graph

Reference

Document	Description
Naming	Naming options and recommendations

Bringing Your Own Agents & Skills

SwarmBench treats any Claude Code agent or skill definition as a first-class debate participant. You are not limited to the built-in roster.

Quick Start

# Copy all agents and skills from your ~/.claude directory into the project
npm run sync
# Or point at any other .claude directory
CLAUDE_DIR=~/work/.claude npm run sync

This copies:

• ~/.claude/agents/*.md → agents/
• ~/.claude/skills/** → skills/ (files and directories)

Both directories are gitignored by default — your library stays private unless you explicitly commit it.

What Can Participate

Source	Format	Examples
Claude Code agent definitions	agents/*.md	ai-engineer.md, legal-compliance-checker.md
Claude Code skill definitions	skills/*.md or skills/<name>/	architecture-patterns/, jsdoc/
Your own custom agents	Any .md you write	domain-expert.md, contrarian.md
Your own custom skills	Any .md you write	evaluate-risk.md, compress-argument.md

Writing a Custom Agent

Drop a .md file into agents/ (or ~/.claude/agents/ before running npm run sync). The file should describe:

• Role — what domain or perspective this agent argues from
• Beliefs — what it assumes to be true and will not abandon
• Constraints — how it is forced to argue (e.g. "attack every untested claim")
• Emotion bias — optional tone modifier (skeptical, aggressive, measured)

After running npm run sync, any files in agents/ serve as examples.

Writing a Custom Skill

Drop a .md file into skills/. A skill is a constrained task that agents call instead of generating freeform output:

• Describe the skill's purpose, input schema, and output schema
• Define failure conditions (what the skill must reject)
• Keep it narrow — one skill, one job

See skills/architecture-patterns/ for a multi-file example.

Implementation Status

Phase 0 — Foundation ✅

• Global constraint prompt, 7 core agent system prompts, 5 skill definitions
• Emotion modifier injection, output validator (hedging / capitulation / verbosity detection)
• Anonymization pipeline (identity stripping + claim shuffling)

Phase 1 — Core Execution ✅

• Zod data models, agent config loader, LLM client interface + Anthropic implementation
• SwarmOrchestrator: 4-phase loop (Generate → Attack → Refine → Arbitrate)
• CLI with scenario runner, verbose logger, JSON transcript output
• Token usage tracking and per-run cost analysis (input · output · cache read/write)
• Support for user-supplied domain agents loaded from agents/ via npm run sync
• Tightened attack semantics: no pure capitulation, every attack must name the failure mechanism

Phase 2 — Embeddings & Similarity 🔜

Agreement leakage detection, shallow attack detection, semantic drift

Phase 3 — Evaluator & Retry Controller 🔜

Per-agent scoring, conflict metrics, 4-level retry escalation

Phase 4–6 — Self-Improving Loop, Benchmarking, Dashboard 📋

Planned

System Layers

Agents (belief + emotion bias)
|
v
Skills (hard behavioral constraints)
|
v
Commands (phase orchestration)
|
v
Evaluation Layer (scoring + conflict metrics)
|
v
Retry Controller (4-level enforcement)
|
v
Self-Improving Loop (9-step threshold tuning)

System Flow

User Input / Scenario
|
v
Swarm Initialization (agents = model + prompt pairs)
|
v
Iterative Argument Loop (N iterations)
1. Generate --> claims (via generate_claims skill)
2. Attack --> critique others (via attack_claims skill, anonymized)
3. Refine --> revise under pressure (via refine_claim skill)
4. Arbitrate --> prune / approve (via arbitrate skill, blind)
|
v
Evaluation + Retry (4-level escalation per phase)
|
v
Embeddings & Similarity Pipeline
- Agreement leakage detection
- Shallow attack detection
- Semantic drift detection
|
v
Evaluator Agent --> per-agent scores & flags
|
v
Self-Improving Loop --> adjust thresholds (9-step cycle)
|
v
Dashboards & Reporting --> ranked scorecards