Measure agent behavior, compare changes on the same cases, and improve prompts or skills without showing the final test cases to the optimizer.
The evaluation path runs in your TypeScript process. Model calls happen only through the clients and agents you configure.
New to the package? Read concepts first — it takes five minutes and defines every word used here.
Looking for a measured result (a lift, a null, a parity verdict)? The canonical registry is evidence/ — machine-readable records, a generated index, and a freshness gate.
pnpm add @tangle-network/agent-evalThis example is offline and complete. Copy it, run it, then replace the agent and the judge with your product code.
import{defineAgentEval}from'@tangle-network/agent-eval/contract'interfaceSupportCase{id: stringkind: 'support'}constevalKit=defineAgentEval<SupportCase,string>({scenarios: [{id: 'refund',kind: 'support'},{id: 'shipping',kind: 'support'},{id: 'cancel',kind: 'support'},],agent: async(prompt,scenario)=>String(prompt).includes('ticket') ? `Ticket ${scenario.id}: on it.` : 'On it.',judge: {name: 'ticket-id',dimensions: [{key: 'present',description: 'The answer includes the ticket id'}],score: ({ artifact, scenario })=>{constpresent=artifact.includes(scenario.id) ? 1 : 0return{dimensions: { present },composite: present,notes: ''}},},baselineSurface: 'Answer politely.',expectUsage: 'off',})console.log((awaitevalKit.evaluate()).aggregates.byJudge)console.log((awaitevalKit.evaluate({surface: 'Answer politely and cite the ticket id.'})).aggregates.byJudge,)Each call runs every case, records what the agent produced, applies the same judge, and returns score distributions.
Three words carry this example. A case is one task the agent must do. A surface is the value being changed: a prompt, a skill, or a serialized configuration. A judge is a function that scores one produced result.
expectUsage: 'off' is set because this agent makes no paid calls.
The default, 'assert', fails a run whose cells report no cost receipt.
Keep the default whenever real model calls happen.
Runnable copy: examples/evaluate-a-change.
Optimization methods may return a bounded SearchHistoryReceipt over Eval's canonical hash-chained SearchLedger. Existing callers keep working and see missing-history coverage. Autonomous and publication-grade runs set searchHistoryPolicy: 'require-complete' to refuse an incomplete planned denominator before the untouched final cases are opened.
The receipt is a small proof envelope, not another event log. Exact candidates, attempts, failures, decisions, and missing ids remain in the ledger. See complete optimization search history.
Every row is a function you call. Each links to a runnable example.
| When to call it | What you give it | What you get back |
|---|---|---|
defineAgentEval() — you changed a surface and must know whether it helped | cases, an agent, a judge, a starting surface | evaluate() for scores, improve() for a search plus a release decision |
selfImprove() — you want candidate generation, scoring, and a release decision in one call | cases, an agent, a judge, a starting surface | a report, a winner surface, and a gateDecision |
analyzeRuns() — the runs already happened and no agent needs to run again | RunRecord[] | an InsightReport: distributions, paired lift, judge agreement, cost, failure clusters |
fromFeedbackTable() (example) / fromOtelSpans() (example) — your data is in a table or an OTel collector, not in RunRecord shape | source rows or spans | RunRecord[] ready for analyzeRuns() |
planCampaignRun() / runCampaign() — you need direct control of the case grid, or you must see it before paying for it | cases, a dispatch function, judges, a run directory | a per-cell schedule, then a campaign result with cached cells |
loadEvalFixtureScenarios() — agents should add cases as folders on disk | evals/<name>/PROMPT.md plus checks | Scenario[] for runCampaign() |
compareOptimizationMethods() — two search methods must be compared at equal budget | methods, a starting surface, train, selection, and final cases | per-method final lift, intervals, pairwise contrasts, and cost |
gepaOptimizationMethod() / skillOptOptimizationMethod() — official GEPA or Microsoft SkillOpt should own the search | an objective, a recipe or trainer, an optimizer budget | an optimization method for the comparison above |
externalTextOptimizationMethod() — another package owns text search and you keep the scoring | the package identity, limits, and a run callback | the same, with the final cases never exposed |
SurfaceProposer — candidate generation belongs to your product | a propose() function | candidates the campaign executes, scores, and gates |
runProfileMatrix() — the same cases must run across models or profiles | axes of models and profiles, cases | one row per cell, with an explicit unknown model rather than an invented one |
ExperimentTracker — a candidate must beat its parent across N repetitions | reps with scores, run ids, and evidence references | a KEEP / ITERATE / NOISE / REGRESSION verdict with git provenance |
sealExperiment() / openSealedExperiment() — the result must convince someone who does not trust you | arms, an admission funnel, an estimand, an interval, a decision table | a hashed rule tree, and executors that can run no other rule |
runEquivalenceCheck() / VERIFICATION_STRATEGIES — the work has no held-out test suite | a claim, two blind arms, an injected checker | a certification that names who vouched and how it can fail |
AnalystRegistry.runExact() — a batch of runs failed and you need cited findings | recorded evidence, a declared analyst list | findings with evidence references, an execution plan, and a receipt |
analyzeTraces() — you have one question about a recorded run ("what first caused this failure?") | stored traces, the question, a DSPy RLM engine with a cost cap | an answer, findings with evidence references, and the investigation trajectory |
runAnalystBenchmark() — an analyst's accuracy must be measured, not assumed | labeled issues and exact span locations | scored findings, trace reads, model calls, tokens, cost, and runtime |
deltaRepair() — a finding must be graded by executing the repair it proposes | a trajectory, an analyst finding, a sandbox | the repair's measured effect against a no-fix control |
replayVerify() — you must know whether a recorded failure still reproduces | a recorded shell trajectory and its pinned image | a re-execution verdict and the divergences found |
analyzeSupervisorRun() — a recursive or supervised run directory must be read | a run directory | counts that stay missing when a measurement is missing, never zero |
buildRlDataset() — scored runs should become training data | run records and preferences | reward, preference, and supervised rows |
Benchmarks, user drivers, executors, built-in judges, completion checkers, and judge adapters all take the same ChatClient.
You own model execution: Agent Eval issues no provider request and never receives a provider credential.
import{createChatClient}from'@tangle-network/agent-eval'constchat=createChatClient({transport: 'custom',defaultModel: 'openai/gpt-4.1',maximumAttempts: 3,chat: async(request,opts)=>myProviderClient(request,opts),})On Agent Runtime, profileChatClient({ profile, executor, context }) from @tangle-network/agent-runtime/kernel is that transport: every call runs one exact AgentProfile and reports its measured usage, retries, and served model identity.
Use sandbox-sdk for Sandbox and mock in tests.
A custom adapter must return a ChatResponse and declare maximumAttempts before a capped cost account can dispatch it.
ChatResponse carries the whole execution record across that boundary: the served model id, measured input/output/reasoning/cached tokens, billed USD or an explicit unknown, the finish reason, and the per-token log probabilities the expectation judge scores on.
The official GEPA and SkillOpt optimizers run through a Python bridge. Install commands, version pins, and the reason for each pin: GEPA, SkillOpt, and DSPy.
| Import | Use |
|---|---|
@tangle-network/agent-eval/contract | Define an evaluation, run it, improve it, and analyze existing runs. |
@tangle-network/agent-eval/campaign | Campaigns, optimization methods, comparisons, storage, and release rules. |
@tangle-network/agent-eval/experiment | Experiments as sealed objects: registered rules, funnels, estimands, refusals. |
@tangle-network/agent-eval/analyst | Built-in and custom trace analysts, labeled comparison, costs, and reports. |
@tangle-network/agent-eval/trace-repair | Grade one analyst finding by executing the repair it proposes. |
@tangle-network/agent-eval/trajectory-replay | Re-execute a recorded shell trajectory and check whether its failure reproduces. |
@tangle-network/agent-eval/traces | Store, replay, and inspect structured traces. |
@tangle-network/agent-eval/reporting | Statistical comparisons and report rendering. |
@tangle-network/agent-eval/supervisor-run | Read recursive run directories without collapsing missing measurements to zero. |
@tangle-network/agent-eval/profile-cell | Create and validate portable agent-profile identities. |
@tangle-network/agent-eval/ledger-core | Append-only hash-chained journal with idempotent append and chain verification. |
@tangle-network/agent-eval/benchmarks | Benchmark adapters and retrieval metrics. |
@tangle-network/agent-eval/rl | Export rewards, preferences, and training rows. |
@tangle-network/agent-eval/wire | HTTP and RPC schemas for other languages. |
@tangle-network/agent-eval/adapters/http | Run campaign cells on remote workers over HTTP. |
Use the root import for common primitives. Use a subpath when you want an explicit capability boundary.
| Question | Read |
|---|---|
| What do these words mean? | docs/concepts.md |
| Why does this package exist, and where is it going? | docs/charter.md |
Which run* function do I want? | docs/eval-surface-map.md |
| How do I choose a candidate-generation method? | docs/campaign-proposers.md |
What is in an InsightReport? | docs/insight-report.md |
| How do I register an experiment as a sealed object? | docs/experiment.md |
| How is something certified without an answer key? | docs/verification-strategies.md |
| Where does every verifier land its result? | docs/verdicts.md |
| How do I turn a coding-agent session log into runs? | docs/code-agent-intake.md |
| How do I score a string from another language? | docs/wire-protocol.md |
The example index lists every runnable example.
pnpm install
pnpm typecheck
pnpm typecheck:examples
pnpm test
pnpm buildPython compatibility tests use the locked dependencies:
cd clients/python
uv sync --frozen --extra dev --group gepa-release
AGENT_EVAL_EXPECT_GEPA_RELEASE=1 \
uv run --frozen --extra dev --group gepa-release \
pytest tests/test_gepa_release_compatibility.py tests/test_gepa_bridge.py
uv sync --frozen --extra dev --group skillopt-source --group gepa-source
uv run --frozen pytest
uv sync --frozen --extra dev --extra dspy
uv run --frozen pytest tests/test_dspy_metric.pyMIT.