ownbench

Mine your own repo's git history into a private, re-runnable coding eval — then race local and frontier models on bugs that actually happened in your codebase.

Public leaderboards tell you how a model does on other people's code. ownbench tells you how it does on yours.

Why

• Coding-model rankings shift monthly; re-evaluating "can a local model handle my repo yet?" by vibes is unreliable.
• Public benchmarks are contaminated and out-of-distribution for your stack, your conventions, your bug shapes.
• Your git history already contains hundreds of labelled examples: a fix commit is a (broken file, instruction, fixed file) triple for free.

How it works

1. mine — walk history for single-file fix commits with small diffs. Parent version = broken input, commit message = instruction, committed version = reference.
2. run — send each broken file + instruction to any OpenAI-compatible endpoint (llama.cpp, vLLM, Ollama, LiteLLM, OpenRouter, or a frontier API).
3. score — deterministic, no LLM judge: exact match, or edit-F1 between the model's diff and the true fix. Whole-file rewrites are penalised the way a reviewer would penalise them.
4. report — per-model table, per-language breakdown, and the tasks where every model still fails.

Install

pip install ownbench # zero runtime dependencies, Python ≥3.10

Quick start

# 1. Build a private eval from your repo (writes <repo>/.ownbench/tasks.jsonl)
ownbench mine ~/code/myrepo --max-tasks 30
# 2. Race two models on itexport OWNBENCH_BASE_URL=http://localhost:8000/v1 # any OpenAI-compatible URLexport OWNBENCH_API_KEY=sk-...
ownbench run ~/code/myrepo/.ownbench/tasks.jsonl --model qwen3-coder-32b
ownbench run ~/code/myrepo/.ownbench/tasks.jsonl --model gpt-5-mini \
--base-url https://api.openai.com/v1 --api-key $OPENAI_API_KEY# 3. Score both runs and render the report
ownbench score ~/code/myrepo/.ownbench/tasks.jsonl \
~/code/myrepo/.ownbench/results-*.jsonl
ownbench report ~/code/myrepo/.ownbench/scores.jsonl

Example output:

| model | score | n | exact | partial | wrong | unchanged | invalid | error | avg latency |
|-----------------|-----------|----|-------|---------|-------|-----------|---------|-------|-------------|
| gpt-5-mini | **0.612** | 30 | 14 | 9 | 5 | 1 | 1 | 0 | 8.4s |
| qwen3-coder-32b | **0.471** | 30 | 9 | 10 | 8 | 2 | 1 | 0 | 21.7s |

Scoring

status	meaning	score
exact	output ≡ the real fix (modulo trailing whitespace)	1.0
partial / wrong	edit-F1 between model's changed lines and the true fix's changed lines	0–1
unchanged	returned the broken file untouched	0
invalid	no code block in the reply	0
error	endpoint failure after retries	0

Edit-F1 = mean of F1(added lines) and F1(removed lines), whitespace-normalised. Precision punishes drive-by rewrites; recall punishes missing the fix.

Honest limitations

• Contamination: if your repo is public and old, frontier models may have trained on the fix commits. Mine with --since to restrict to recent history, or use a private repo.
• Commit messages vary in quality; mine drops subjects under 12 chars and known noise (typo/lint/format/bump), but garbage-in still applies.
• Exact-match + edit-F1 is a proxy, not a test suite. A semantically correct alternative fix scores partial, not perfect. Treat scores as relative between models on the same task set.
• Single-file tasks only. Cross-file refactors are out of scope by design — they don't fit a single-completion eval.

CLI reference

ownbench mine <repo> [--max-tasks 30] [--max-commits 2000] [--max-diff-lines 40] [--since DATE] [--out PATH]
ownbench run <tasks.jsonl> --model NAME [--base-url URL] [--api-key KEY] [--limit N] [--timeout 300] [--out PATH]
ownbench score <tasks.jsonl> <results.jsonl...> [--out PATH]
ownbench report <scores.jsonl> [--out report.md]

Env vars: OWNBENCH_BASE_URL, OWNBENCH_API_KEY.

License

MIT