Feat/agent bench production grade - #30
Merged
Merged
Conversation
A repo-pointed pnpm store must never be committed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
The agent-bench recording proxy (spec §6.4) points every agent at one meter. Claude Code honours ANTHROPIC_BASE_URL; freecode ignored the equivalent because the SDK is constructed with the catalogue URL. baseURLFor() gives every provider id the same hook (MINIMAX_BASE_URL for the bench) — a bench-only override, not a documented user setting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
Pass-through HTTP proxy (spec §6.4): no cache, no retries, no body rewrite. Tokens parsed off Anthropic Messages JSON/SSE and OpenAI Chat Completions, normalized to an INCLUSIVE inputTokens (Anthropic's wire excludes cache fields, OpenAI's includes them) so the rate card's cache discount is correct for both shapes. USD from a committed rate card with a vintage — an unknown model prices as undefined, never zero. The log doubles as the isolation audit: a non-model path is a leak, and an empty log is unmetered, not clean. No secrets on disk (no headers, no bodies). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
One container per trial on an --internal docker network: no route to the internet, so the only exit is the recording proxy on the network gateway — the proxy log becomes a real egress audit. Stated residue: other host services on the gateway IP stay reachable; the internet does not. The image bakes pinned agent versions (freecode from a released binary — at trial time there is no network to install with), $HOME is a throwaway so no memory carries between trials, and env values ride bare -e NAME flags so argv.json never contains a secret. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
Metering on by default: each trial gets its own proxy, the runner injects
the base-URL env, and the folded tokens/USD/turns/auditOk land in the trial
record and the published matchup JSON. --isolate runs the trial in a
container on the internal network (proxy bound on its gateway), reads the
agent's version from the image, resolves {benchDir} to the ro mount, and
docker-rm's a timed-out container — the docker client dying does not stop
one. Also restores TrialRecord.artifactDir, which had been dropped while
run.ts still assigned it: tsx executes without typechecking, so
test:agent-bench now runs tsc -p bench/agent-bench first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52fThe verdict is the harness's, in Docker, on patch.diff — grade.ts only shuttles patches in and verdicts out. One harness invocation per (agent, trial): the harness keys on instance_id, so trials must not share a predictions file. An empty patch is resolved:false without spending a container; a harness error leaves null, which the page counts against rather than dropping. Verdicts merge into report.json (graded: true) and --publish re-publishes the matchup, flipping the page's headline from "Produced a patch" to "Resolved". Free to re-run: grading never touches a model. Needs docker + pip install swebench (SWEBENCH_PYTHON overrides). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
One reproducible tar.gz per run (sorted, owner/mtime pinned — same evidence, same bytes) plus a sha256 sidecar: report, prompts, argv, patches, agent stdout/stderr, proxy logs, usage/audit folds, grading output. The proxy log carries no headers or bodies, so the bundle is publishable as-is; the checksum is the claim anyone can re-derive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
Token and cost bars off the one meter, with the traps §7 names handled in the rendering: an unmetered agent shows "unmetered", never $0; worst single-trial cost sits beside the mean; and the §7.2 cost-on-solved-intersection line appears only once the matchup is graded, suppressed below 3 shared resolved instances. A caveat lists per-agent metering coverage whenever any trial went unmetered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
The bench/agent-bench README collapses to a pointer at the repo-root AGENT-BENCH.md (same job as EVAL.md/TRACE.md): setup, metering and the wire conventions it normalizes, --isolate, bench:grade, bench:bundle, when to run what, and §8's honest status — every layer exists, but isolation and the grader still need their first live smoke run (docker group + pip install swebench), and a publishable number is a run that used all of them. Benchmark.md becomes the instrument picker; CLAUDE.md/AGENTS.md/EVAL.md/ README.md rows updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
First live smoke: the installed freecode release binary predates the baseURLFor hook, talked around the proxy, and the page counted its zeroed usage as a metered $0.0000 trial. Metered now means the proxy saw model calls (turns > 0) — in the summary, the matrix cells, and the §7.2 intersection alike. Also folds in the smoke-run rows: the meter's inclusive normalization verified against MiniMax's shim (t2 cache-read == t1 total input), and the leak audit caught Claude Code's HEAD /api/hello probe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELXKXKpNy38kvVfjwJC52f
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.