Benchpress

CLI tool for benchmarking LLM providers and models. Measures response time, quality (tool accuracy, coherence, relevance), and cost — then generates a presentation-grade PDF report with LLM-generated insights.

Designed for comparing the same model across different providers (e.g., Claude Sonnet via Anthropic direct vs. via Gloo AI) to answer: is the provider adding latency, reducing quality, or saving enough money to justify the tradeoff?

Features

• Multi-provider benchmarking — test multiple providers in a single run, each with its own auth (API key or OAuth client_credentials)
• Head-to-head comparisons — tag models with base_model to group the same underlying model across providers
• LLM-generated insights — configurable report LLM generates executive summaries and per-comparison analysis
• Per-provider framing — insights_context lets you control the tone of generated insights (e.g., acknowledge a provider's guardrails or value-add)
• PDF reports — timestamped, presentation-ready reports with side-by-side metrics and delta indicators
• Flexible auth — API keys via env vars, or OAuth2 client_credentials flow with Basic or POST body methods
• Custom headers — per-provider and per-model extra_headers for toggling features like extended context

Installation

Requires Python 3.11+.

# Clone the repo
git clone <repo-url>&&cd ai-benchpress
# Install in editable mode
pip install -e .# Or with pipx for isolated install
pipx install -e .

Quick Start

1. Copy the example config and environment file:

cp benchpress.yaml.example benchpress.yaml
cp .env.example .env

2. Add your API keys to .env:

ANTHROPIC_API_KEY=sk-ant-...
# For OAuth providers:# GLOO_AI_CLIENT_ID=...# GLOO_AI_CLIENT_SECRET=...

3. Edit benchpress.yaml to configure your providers and models.

4. Validate your config:

benchpress validate

5. Run benchmarks:

benchpress run -v

Configuration

All configuration lives in benchpress.yaml. See benchpress.yaml.example for a fully documented example.

Providers

Each provider needs a name, base_url, and authentication:

providers:
# API key auth
- name: "anthropic"base_url: "https://api.anthropic.com/v1"api_key_env: "ANTHROPIC_API_KEY"models: [...]# OAuth client_credentials auth
- name: "gloo-ai"base_url: "https://platform.ai.gloo.com/ai/v2/"oauth:
token_url: "https://platform.ai.gloo.com/oauth2/token"client_id_env: "GLOO_AI_CLIENT_ID"client_secret_env: "GLOO_AI_CLIENT_SECRET"scopes: ["api/access"]auth_method: "basic"# or "post_body"models: [...]# No auth (e.g., local Ollama)
- name: "ollama"base_url: "http://localhost:11434/v1"models: [...]

Cross-Provider Comparisons

Tag models with base_model to group the same underlying model across providers:

# Under anthropic provider
- id: "claude-sonnet-4-5"display_name: "Sonnet 4.5"base_model: "claude-sonnet-4-5"# Under gloo-ai provider
- id: "gloo-anthropic-claude-sonnet-4.5"display_name: "Gloo AI - Sonnet 4.5"base_model: "claude-sonnet-4-5"# same tag = head-to-head comparison

Provider Insights Context

Control how the report LLM frames each provider:

- name: "gloo-ai"insights_context: | Gloo AI provides faith-based guardrails and prompt enhancements. Acknowledge the value these safety features add beyond raw performance.

Extra Headers

Add custom headers at the provider or model level:

- name: "anthropic"extra_headers:
X-Custom-Header: "value"# applied to all modelsmodels:
- id: "claude-opus-4-6"extra_headers:
anthropic-beta: "interleaved-thinking-2025-05-14"# per-model

Model headers merge on top of provider headers (model wins on conflict).

Report & Insights LLM

Configure which LLM generates report insights:

report:
results_dir: "./results"report_path: "./reports"# Option A: reference an existing providerllm_provider: "anthropic"llm_model: "claude-sonnet-4-5"# Option B: standalone config (mutually exclusive with Option A)# llm:# base_url: "https://api.anthropic.com/v1"# api_key_env: "ANTHROPIC_API_KEY"# model: "claude-sonnet-4-5"

If no LLM is configured, reports are generated with metrics only.

CLI Reference

# Run benchmarks
benchpress run [OPTIONS]
-c, --config Config file path (default: benchpress.yaml)
-e, --env .env file path
-m, --models Comma-separated model filter
-p, --providers Comma-separated provider filter
-n, --num-requests Override number of requests per model
--concurrency Override concurrency level
--interval Override interval between requests (ms)
-o, --output Override report output directory
--no-pdf Skip PDF generation, only save JSON
-v, --verbose Print per-model results during run
# Re-generate PDF from previous JSON results
benchpress report -i results/benchpress-YYYYMMDD-HHMMSS.json
-c, --config Config file (needed for LLM insights)
-o, --output Override report output directory
# List configured models
benchpress list-models
# Validate config and check env vars
benchpress validate

Output

Each run produces:

• JSON results in results/benchpress-YYYYMMDD-HHMMSS.json — raw data for all requests
• PDF report in reports/benchpress-report-YYYYMMDD-HHMMSS.pdf — presentation-ready report

Report Structure

1. Title & Executive Summary — config, winner badges, LLM-generated narrative
2. Head-to-Head Comparisons — side-by-side metrics with deltas for matched models
3. Additional Models — unmatched models in a compact table
4. Data Appendix — full metrics table, per-category breakdown, error summary

Project Structure

ai-benchpress/
benchpress/
cli.py # CLI commands (typer)
config.py # YAML config loading + auth (API key, OAuth)
client.py # HTTP client for OpenAI-compatible APIs
runner.py # Async benchmark orchestration
prompts.py # Test prompt generation
scorer.py # Result scoring (tool accuracy, coherence, relevance)
models.py # Pydantic data models
report.py # PDF report generation + LLM insight calls
benchpress.yaml.example
.env.example
pyproject.toml

Contributing

1. Fork the repo and create a feature branch
2. Install in editable mode: pip install -e .
3. Make your changes
4. Test with a small run: benchpress run -n 3 -v
5. Verify config loading: benchpress validate
6. Submit a pull request

Adding a New Provider

1. Add the provider block to benchpress.yaml with auth config
2. Add models with base_model tags if comparing against existing providers
3. Add insights_context to frame the provider in report narratives
4. Run benchpress validate to check credentials
5. Test with benchpress run -p your-provider -n 3 -v

Guidelines

• All providers use the OpenAI-compatible /chat/completions endpoint
• Keep benchpress.yaml.example up to date with any new config fields
• Credential values belong in .env, never in YAML files — use *_env fields to reference env var names