A small, reusable evaluation harness that measures collusion and covert-coordination behavior in multi-agent LLM systems, swept across the axes that are the genuine whitespace in current multi-agent eval: population homogeneity (same-vendor / cross-vendor / cross-capability / cross-training-paradigm), communication channel richness, network topology, and — throughout — the dissociation between an agent population's propensity to coordinate against the system objective and its capability to actually execute that coordination.
Status: greenfield. Nothing is built yet. This repo currently holds the phased requirement spec and its execution checklist; Phase 0 (see below) is the immediate target.
- Reuses, doesn't reinvent. Forks Colosseum's DCOP-formalized cooperative environments and its regret-relative-to-cooperative-optimum metric rather than authoring a new scenario/game. Runs inside Inspect AI's Task/Solver/Scorer primitives rather than a bespoke harness.
- Cost-first. Phase 0 is a hard $0 — free-tier and self-hosted open-weight models only. No phase escalates model tier until the phase below it has proven the plumbing works.
- Propensity and capability, kept separate. An action-based scorer (regret vs. the cooperative optimum) and an intent-based LLM-judge scorer are computed independently and never fused into one number — the gap between them (Colosseum's "collusion on paper" phenomenon) is the interesting quantity, not a combined score.
- Not a publishable, statistically-powered study; not a novel environment; not a production monitoring system. It's an illustrative, extensible harness plus a directional go/no-go signal.
| Phase | Cost ceiling | Goal |
|---|---|---|
| 0 — Plumbing & baseline | $0 | Prove the harness runs end-to-end on free-tier/self-hosted models; establish a benign-vs-secret-channel baseline; freeze the logging schema. |
| 1 — Illustrative grid | <$50 | Cross-vendor mini-tier model arms (Gemini Flash-Lite, GPT-5 nano, Claude Haiku 4.5, DeepSeek Flash, + the Phase-0 open-weight model), fractional-factorial sweep across every axis, directional result table + a cost-grounded next-cell recommendation. |
| 2 — Capability-tier stretch | small reserved budget | Escalate the one or two most interesting Phase 1 cells to flagship models; doubles as a capability-tier canary test; seeds a research-directions memo. |
Full requirements, design principles, risks, and exit criteria: REQUIREMENTS.md. Execution checklist for Phase 0 and Phase 1: TASKS.md.
| Dependency | Role |
|---|---|
| Colosseum (arXiv:2602.15198) | DCOP environments, cooperative-optimum regret metric, topology sweeps, secret-channel probe — primary fork target |
| Inspect AI (UK AISI) | Task/Solver/Scorer harness layer, native multi-agent support, single model abstraction across vendors + local endpoints |
| EconEvals (arXiv:2503.18825) | Litmus-test scoring template, anti-saturation synthetic-difficulty principle |
| Secret Collusion / SCHEME (arXiv:2402.07510, arXiv:2605.29178) | Channel-richness and topology as explicit swept variables |
| On the Fragility of AI Agent Collusion (arXiv:2603.20281) | Cross-algorithm vs. cross-capability heterogeneity as separate axis conditions |
This repo also reuses engineering (not research) prior art from two sibling in-house repos for the model/provider layer — see REQUIREMENTS.md Appendix C:
refract(src/refract/llm_client.py,config.py) — free-tier RPD/RPM/TPD-aware provider fallback chains and per-model rate-limit pacing.penumbra(llm_client.py,systems/base.py) — judge/proxy provider-family independence (self-enhancement bias control), usage logging, and aProtocol-typed adapter pattern.
Inspect's native model abstraction (get_model("provider/model")) is the primary interface for
swapping hosted and locally-served models — the sibling repos' rate-limiting/fallback pattern is
worth porting for Phase 0's free-tier runs; their hand-rolled model-dispatch code is not, since
Inspect already solves the problem they were compensating for.
Not yet runnable — Phase 0 scaffolding (environment fork, Inspect wiring, model-layer wrapper) is the current work; see TASKS.md §Phase 0.
REQUIREMENTS.md Full spec: functional/non-functional requirements, phase plan, risks
TASKS.md Phase 0 / Phase 1 execution checklist
.env.example Free-tier provider keys + local endpoint config (Phase 0)
requirements.txt inspect-ai + free-tier provider SDKs (lazy-imported)