evalspec

Gold-label evaluation framework for LLM agents. Measure what your model actually does, track it over time, and catch regressions before they ship.

Quick start

pip install evalspec
# Create a dataset
cat > datasets/analytics.yaml <<EOFquestions: - id: Q-001 question: "How many swaps happened last quarter?" gold_answer: "tool_call" expected_tool: "get_swap_counts" expected_behaviour: "answer_with_citation" language: ENEOF# Run against OpenAIexport OPENAI_API_KEY="sk-..."
evalspec-run --all --provider openai --model gpt-4o --tag baseline-v1
# Record baseline and check for regressions
evalspec-regression --record baselines/gpt4o.json --provider openai --model gpt-4o
evalspec-regression --check baselines/gpt4o.json --provider openai --model gpt-4o

Gold labels

Every question gets a gold_answer that defines correct behavior:

Label	Meaning	Measured by
ABSTAIN	Model must refuse	was_refused=True
CLARIFY	Model must ask for clarification	asked_clarification=True
tool_call	Model must call the expected tool	expected_tool in called tools

Agents

Provider	Flag	Environment
OpenAI	--provider openai --model gpt-4o	OPENAI_API_KEY
opencode	--provider opencode --model deepseek	opencode CLI
Mock	--mock --mock-mode perfect	None
HTTP	--agent-url http://localhost:8080	None

CLI tools

Command	Purpose
evalspec-run	Run evaluation harness
evalspec-split	80/20 stratified holdout split
evalspec-compare	Side-by-side model comparison
evalspec-regression	CI regression gate
evalspec-leakage	Parametric leakage filter

Model comparison

evalspec-run --all --provider openai --model gpt-4o --tag gpt4o --report runs/gpt4o.json
evalspec-run --all --provider openai --model gpt-4o-mini --tag gpt4o-mini --report runs/gpt4o-mini.json
evalspec-compare runs/gpt4o.json runs/gpt4o-mini.json --html compare.html

CI gate

# .github/workflows/eval.ymlon: [pull_request]jobs:
eval:
runs-on: ubuntu-lateststeps:
- uses: actions/checkout@v4
- run: pip install evalspec openai pyyaml
- run: evalspec-regression --check baselines/gpt4o.json -p openai --model gpt-4oenv:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Philosophy

• Gold labels, not heuristics: Every question defines what "correct" means (refuse, clarify, or call a specific tool).
• Version-frozen: Reports include dataset hashes so you know exactly which corpus a score refers to.
• Held-out split: 80/20 stratified split prevents overfitting to the eval set.
• CI-gated: 3% regression tolerance means prompt or model changes don't silently degrade quality.

License

MIT