A comprehensive, production-ready framework for evaluating AI agents across multiple dimensions of capability. Built for teams who need to benchmark, test, and improve their AI agents with measurable metrics.
Research shows leading AI agents complete only 30-35% of multi-step tasks1. Without proper evaluation, teams ship agents that fail silently in production. This framework provides structured evaluation across:
- Task completion - Does the agent achieve its goals?
- Tool usage efficiency - Does it use the right tools, not too many?
- Error recovery - Can it recover from failures?
- Latency - Is it fast enough for user-facing apps?
- Consistency - Does it produce reliable results across runs?
pip install agent-evaluation-frameworkOr add to your project:
poetry add agent-evaluation-frameworkfromagent_evalimportEvaluator, Benchmark, Metrics# Define your agentasyncdefmy_agent(input: str) ->str:
# Your agent implementationreturnawaitagent.complete(input)
# Create evaluatorevaluator=Evaluator(
agent=my_agent,
benchmarks=[
Benchmark.task_completion(),
Benchmark.tool_efficiency(),
Benchmark.error_recovery(),
Benchmark.latency(),
],
metrics=Metrics.default()
)
# Run evaluationresults=awaitevaluator.evaluate(
test_suite="path/to/test_cases.json",
iterations=3
)
# View resultsprint(results.summary())
print(results.metrics())Benchmarks define what to test:
| Benchmark | What It Measures |
|---|---|
task_completion | Did the agent achieve the goal? |
tool_efficiency | Optimal tool usage |
error_recovery | Recovery from failures |
latency | Response time |
consistency | Stability across runs |
context_usage | Token efficiency |
The framework calculates:
- Success Rate - % of tasks completed
- Average Latency - P50, P95, P99
- Token Efficiency - Tokens per successful task
- Error Rate - Failure frequency
- Recovery Rate - Failures that were recovered
Define test cases in JSON:
[
{
"id": "task_001",
"input": "Find all files modified yesterday",
"expected": "List of file paths",
"context": {
"files": ["a.txt", "b.txt"],
"last_modified": "2025-01-01"
}
}
]Works with any agent that implements the AgentProtocol:
fromagent_evalimportAgentProtocolclassMyAgent(AgentProtocol):
asyncdefrun(self, input: str, context: dict=None) ->str:
# Your implementationpassasyncdeftools(self) ->list[dict]:
# Return available toolsreturn [{"name": "search", "description": "Search files"}]Built-in adapters for popular frameworks:
- OpenAI Agents SDK
- LangChain Agents
- LangGraph
- Agno
- CrewAI
- Custom (bring your own)
fromagent_eval.adaptersimportlangchain_adapter, openai_adapter# Use with LangChainevaluator=Evaluator(
agent=langchain_adapter(my_langchain_agent),
benchmarks=[...]
)Test CLI capabilities:
fromagent_eval.benchmarksimportterminal_benchresults=awaitevaluator.evaluate(
benchmark=terminal_bench(
tasks=[
{"command": "ls -la", "expected_output": "file listing"},
{"command": "grep -r 'pattern' .", "expected_output": "matches"},
]
)
)Test code generation:
fromagent_eval.benchmarksimportcode_completionresults=awaitevaluator.evaluate(
benchmark=code_completion(
test_suite="tests/code_tasks.json",
language="python",
timeout=30
)
)Test agent collaboration:
fromagent_eval.benchmarksimportmulti_agentresults=awaitevaluator.evaluate(
benchmark=multi_agent(
agents=[researcher, writer, reviewer],
workflow="research_write_review"
)
)# OpenAI
OPENAI_API_KEY=sk-...
# Anthropic
ANTHROPIC_API_KEY=sk-antik...
# Custom
AGENT_EVAL_LOG_LEVEL=INFO
AGENT_EVAL_DB_PATH=./eval.db# eval.yamlevaluation:
iterations: 3timeout: 60parallel: 4benchmarks:
task_completion:
threshold: 0.8latency:
p95_threshold_ms: 5000storage:
backend: sqlitepath: ./eval.dbreporting:
format: jsonoutput: ./results/The evaluator returns detailed results:
results=awaitevaluator.evaluate(...)
# Summaryprint(results.summary())
# Agent: my_agent# Benchmarks: 4# Success Rate: 85.2%# Avg Latency: 2.3s# Score: 82/100# Detailed metricsprint(results.metrics())
# {# "task_completion": {"success_rate": 0.89, "score": 89},# "tool_efficiency": {"score": 78},# "error_recovery": {"score": 85},# "latency": {"score": 75}# }# Exportresults.to_json("./results.json")
results.to_csv("./results.csv")
results.to_prometheus(port=9090)# .github/workflows/eval.yml
- name: Agent Evaluationrun: | agent-eval run \ --agent ./agent.py \ --benchmarks task_completion,latency \ --threshold 75fromagent_eval.integrationsimportlangsmithevaluator=Evaluator(
agent=my_agent,
callbacks=[langsmith_callback(project="my-agent")]
)fromagent_eval.integrationsimportbraintrustevaluator=Evaluator(
agent=my_agent,
callbacks=[braintrust_callback(project="my-agent")]
)fromagent_evalimportBenchmark, BenchmarkBuilderclassMyBenchmark(BenchmarkBuilder):
name="my_custom"asyncdefrun(self, agent, test_case):
result=awaitagent.run(test_case.input)
returnself.score(result, test_case.expected)
defscore(self, result, expected) ->float:
returnfloat(result==expected)fromagent_evalimportMetricsBuilderclassMyMetrics(MetricsBuilder):
name="my_metrics"asyncdefcalculate(self, traces):
return {
"my_metric": sum(t.my_valuefortintraces) /len(traces)
}classEvaluator:
def__init__(
self,
agent: AgentProtocol,
benchmarks: list[Benchmark],
metrics: Metrics=Metrics.default()
):
...
asyncdefevaluate(
self,
test_suite: str|list[dict],
iterations: int=1,
parallel: int=1
) ->EvaluationResults:
...classBenchmark:
@staticmethoddeftask_completion(threshold: float=0.8) ->Benchmark:
...
@staticmethoddeftool_efficiency(
max_tools: int=10,
optimal_range: tuple= (1, 5)
) ->Benchmark:
...
@staticmethoddeferror_recovery(
max_retries: int=3
) ->Benchmark:
...
@staticmethoddeflatency(
p50_threshold_ms: int=1000,
p95_threshold_ms: int=5000
) ->Benchmark:
...- agent-a2a-bridge — A2A protocol for multi-agent communication
- agent-memory-store — Persistent memory for AI agents
- prompt-version-control — Version control for AI prompts
MIT
We welcome contributions! See CONTRIBUTING.md for guidelines.