riven bench

Public benchmark harness for five-way AI provider comparison. Scores riven FORGE against Atlas, Comet, Perplexity, and ChatGPT on the same tasks, the same prompts, and the same rubric — every run.

Results are published at rivenai.io/bench.

---

what it measures

Category	Metric	Method
code	FORGE task pass rate	LLM judge vs. ground truth
search	web search accuracy	factual recall over curated query set
voice	voice round-trip latency	median TTFT over 5 cold starts
FORGE	swarm parallelism ceiling	real concurrent agent execution
latency	time-to-first-token	median TTFT + TPS, 5 reps × 3 prompt classes

Each category uses a fixed dataset with a consistent judge rubric. No cherry-picking: every category runs every provider on every run, or the run is marked partial and excluded from aggregate scores.

---

how to run it

prerequisites

• bash >= 4.x
• curl
• jq
• python3 (stdlib only)
• bc
• API keys for each provider you want to bench (see below)

quick start

git clone https://github.com/rivenai/bench
cd bench

# Set provider keys
export RIVEN_BENCH_KEY="rk-..."
export OPENAI_API_KEY="sk-..."           # for ChatGPT column
export ANTHROPIC_API_KEY="sk-ant-..."   # for Atlas/Comet columns
export PERPLEXITY_API_KEY="pplx-..."    # for Perplexity column

# Run all benchmarks
bash run_all.sh

Results land in results/ as CSV files and a markdown summary.

run a subset

# Swarm bench only — riven_forge column
bash swarm-bench.sh --providers riven_forge

# Specific tasks
bash swarm-bench.sh --tasks SB-001,SB-004

# Dry run (riven_forge only, no competitor calls)
bash swarm-bench.sh --dry-run

run individual scripts

script	what it tests	output
01_latency.sh	TTFT, TPS, total latency for 5 models × 3 prompt classes × 5 reps	results/latency_*.csv
02_embeddings.sh	embedding latency + vector dim across batch sizes 1/8/32	results/emb_*.csv
03_mmlu.sh	zero-shot MMLU accuracy (50 questions, 5 categories)	results/mmlu_*.csv
04_gsm8k.sh	grade-school math accuracy (20 problems)	results/gsm8k_*.csv
05_competitive.sh	same tasks vs OpenAI, Anthropic, Perplexity	results/competitive_*.csv
swarm-bench.sh	five-way swarm orchestration, scored by LLM judge	history/<date>/swarm-bench-report.md
run_all.sh	runs 01–07 + swarm in sequence	results/logs/<timestamp>/

---

output format

latency CSV

model, prompt_class, rep, ttft_ms, tps, total_ms, in_tok, out_tok, ok

swarm bench report

A markdown file at history/<YYYY-MM-DD>/swarm-bench-report.md with:

• per-provider score table
• regression alerts (exit 1 if any provider drops > 25% vs prior run)
• raw task-level results as JSON in history/<date>/tasks/

---

datasets

file	description
datasets/mmlu_50.jsonl	50 MMLU-style questions, 10 per category
datasets/gsm8k_20.jsonl	20 grade-school math word problems

Each JSONL row schema is documented in the respective script header.

---

adding providers

The provider list is in swarm-bench.sh under PROVIDERS. To add a new provider:

1. Add an entry to the PROVIDERS array with name, api_base, model, and auth_env fields.
2. Set the corresponding API key env var before running.
3. Open a PR — results for new providers will appear in the next scheduled run.

---

scheduled runs

The harness runs nightly via cron on sbx-api.rivenai.io. Results are pushed to rivenai.io/bench automatically. To subscribe to email notifications when results update, use the form on the bench page.

---

license

MIT — see LICENSE.

Contributions welcome. If you find an error in the methodology or results, open an issue.