Agent Reliability Arena

Production readiness benchmarking for AI agents.

Built by Aviskaar Applied AI Research Lab.

---

The Problem

Existing benchmarks (GAIA, τ-bench, AgentBench, SWE-bench) measure agent capability — can it do the task? They don't measure production reliability — will it do it consistently, gracefully, and safely under real-world conditions?

Agent Reliability Arena answers: "Is this agent ready for production?"

It stress-tests your agent across five dimensions that matter in deployment: consistency, robustness under input variation, tool failure recovery, multi-turn memory coherence, and enterprise-grade resilience.

---

Five Eval Tracks

Track	What It Tests	Key Metric
Consistency	Output stability across identical inputs (N runs)	Mean pairwise semantic similarity
Robustness	Performance under paraphrased and noisy inputs	Success rate across input variants
Tool Failure	Recovery from injected tool call failures	Tool recovery rate
Memory Drift	Factual coherence across multi-turn conversations	LLM-judge coherence score
Enterprise Realism	Schema drift, permission denial, audit-log completeness	Enterprise Readiness Index (ERI)

---

Quickstart

Install

pip install agent-reliability-arena

Or install from source for development:

git clone https://github.com/aviskaar/ara.git
cd ara
pip install -e ".[dev]"

Run an evaluation

arena run configs/example_agent.yaml

Reports are saved to reports/ as Markdown and JSON.

Launch the dashboard

streamlit run dashboard/app.py

---

Requirements

• Python 3.10+
• An ANTHROPIC_API_KEY for paraphrase generation and LLM judging (Robustness, Memory Drift, Enterprise Realism tracks)
• An OPENAI_API_KEY if using langchain_react or openai_assistants agent types

Set your keys in the environment before running:

export ANTHROPIC_API_KEY="sk-ant-..."export OPENAI_API_KEY="sk-..."# if using OpenAI-backed agents

---

Configuration

Define your agent in a YAML file:

agent:
name: "My Customer Service Agent"type: langchain_react # langchain_react | openai_assistants | custommodel: "gpt-4o"system_prompt: "You are a helpful customer service agent."tools:
- name: search_orderstype: apidescription: "Search customer orders by order ID or email"
- name: refund_tooltype: apidescription: "Initiate a refund for a given order ID"eval:
tracks:
- consistency
- robustness
- tool_failure
- memory_drift
- enterprise_realismruns_per_track: 10task_suite: customer_service_v1 # built-in or path to custom .yamlparaphrase_model: claude-sonnet-4-20250514paraphrase_provider: anthropictimeout_seconds: 60output:
report_format: markdownleaderboard_submit: falseexport_failure_clips: trueoutput_dir: reports

Configuration Reference

agent

Field	Type	Required	Description
name	string	yes	Human-readable agent name
type	enum	yes	langchain_react, openai_assistants, or custom
model	string	no	Model identifier (default: gpt-4o)
system_prompt	string	no	System prompt injected into the agent
tools	list	no	Tool definitions (name, type, description)
entry_point	string	no	Required for custom type: "module.path::function_name"

eval

Field	Type	Default	Description
tracks	list	[consistency]	Tracks to run
runs_per_track	int	10	Agent invocations per track
task_suite	string	general_v1	Built-in suite name or path to custom YAML
paraphrase_model	string	claude-sonnet-4-20250514	Model used for paraphrase generation and LLM judging
timeout_seconds	int	60	Per-run timeout

output

Field	Type	Default	Description
report_format	enum	markdown	markdown or json
export_failure_clips	bool	true	Include failure run traces in report
output_dir	string	reports	Directory for generated reports

---

Supported Agent Frameworks

LangChain ReAct

Bring any AgentExecutor. Arena builds stub tools from your YAML definitions and wraps the executor automatically:

agent:
type: langchain_reactmodel: "gpt-4o"

OpenAI Assistants API

Thread-per-run evaluation using the Assistants v2 API. Requires OPENAI_API_KEY:

agent:
type: openai_assistantsmodel: "gpt-4o"

Custom Python Agent

Any run(prompt: str) -> str function, sync or async. Point to it with entry_point:

agent:
type: customentry_point: "my_package.my_agent::run"

The function must accept a single prompt: str argument and return a string.

---

Built-in Task Suites

Suite	Tasks	Description
general_v1	5	General reasoning, tool use, and multi-step planning
customer_service_v1	5	Order management, refunds, and product support

Custom Task Suites

Create a YAML file with the following structure and pass its path as task_suite:

name: my_custom_suitedescription: "My domain-specific evaluation tasks"tasks:
- id: task_001prompt: "Your main task prompt here"expected_keywords: ["keyword1", "keyword2"]turns:
- prompt: "First turn in a multi-turn conversation"
- prompt: "Second turn — tests memory retention"
- prompt: "Third turn — tests graceful conclusion"

Each task supports:

• id — unique identifier
• prompt — primary task prompt (used by single-turn tracks)
• expected_keywords — list of terms expected in the output
• turns — list of prompts for multi-turn tracks (Memory Drift, Enterprise Realism)

---

Python API

fromarenaimportArenaRunnerimportasynciorunner=ArenaRunner("configs/my_agent.yaml")
report=asyncio.run(runner.run())
print(f"Overall score: {report.overall_score:.3f}")
print(f"Enterprise Readiness: {report.enterprise_readiness_index:.3f}")
print(f"Summary: {report.summary}")
forresultinreport.track_results:
print(f" {result.track.value:<22}{result.score:.3f}{result.label}")

Streaming track results

Use the on_track_complete callback to process results as each track finishes:

defon_complete(result):
print(f"Track complete: {result.track.value} → {result.score:.3f}")
runner=ArenaRunner("configs/my_agent.yaml", on_track_complete=on_complete)
report=asyncio.run(runner.run())

Report models

ArenaReport
├── agent_name: str
├── timestamp: str# ISO 8601 UTC
├── arena_version: str
├── overall_score: float# computed property, mean across tracks
├── enterprise_readiness_index: float|None
├── summary: str
├── config: ArenaConfig
└── track_results: list[TrackResult]
├── track: TrackName
├── score: float# 0.0 – 1.0
├── label: str# human-readable verdict
├── runs: list[AgentRun]
├── failure_clips: list[dict]
└── metadata: dict

Saving reports manually

fromarena.reporterimportsave_reportpaths=save_report(report, output_dir="reports")
# Returns {"markdown": Path(...), "json": Path(...)}

---

Score Interpretation

Score	Grade	Verdict
0.95 – 1.00	A	Production Ready
0.85 – 0.94	B	Conditionally Ready
0.70 – 0.84	C	Needs Improvement
0.55 – 0.69	D	Significant Gaps
0.00 – 0.54	F	Not Production Ready

The Enterprise Readiness Index (ERI) is a weighted composite of consistency (35%), tool recovery (35%), and audit coverage (30%). It is only produced when the enterprise_realism track is included.

---

Dashboard

Arena ships with a Streamlit dashboard for interactive evaluation and report visualization.

streamlit run dashboard/app.py

Features:

• Configure agent and tracks without editing YAML
• Upload an existing arena_config.yaml
• Live progress during evaluation
• Radar chart of track scores
• Failure clip browser
• One-click Markdown and JSON report export

---

Project Structure

ara/
├── arena/
│ ├── __init__.py # Public API: ArenaRunner, ArenaReport, TrackName
│ ├── cli.py # `arena run` command
│ ├── runner.py # Top-level orchestrator
│ ├── models.py # Pydantic data models
│ ├── config.py # YAML config loader
│ ├── agent_adapter.py # Adapters: LangChain, OpenAI Assistants, Custom
│ ├── reporter.py # Markdown + JSON report generator
│ ├── tracks/
│ │ └── __init__.py # Five track implementations + TRACK_MAP registry
│ ├── scorers/
│ │ └── __init__.py # Semantic similarity, LLM judge, tool recovery, ERI
│ └── injectors/
│ └── __init__.py # Tool failure, paraphrase, schema drift injectors
├── configs/
│ ├── example_agent.yaml
│ └── task_suites/
│ ├── general_v1.yaml
│ └── customer_service_v1.yaml
├── dashboard/
│ └── app.py # Streamlit dashboard
├── pyproject.toml
└── Makefile

---

Development

# Install with dev dependencies
make install
# Run linter
make lint
# Run tests
make test# Clean build artifacts and reports
make clean

---

Contributing

Contributions are welcome. Please open an issue first to discuss major changes.

1. Fork the repository
2. Create a feature branch (git checkout -b feat/my-track)
3. Make your changes and add tests
4. Run make lint and make test
5. Open a pull request

To add a new eval track, implement BaseTrack in arena/tracks/__init__.py and register it in TRACK_MAP.

---

License

Apache 2.0 — Aviskaar Applied AI Research Lab