A mini-LangSmith: record, visualize, evaluate, and compare AI agent executions. Records agent runs (planner → tool → retrieval → LLM), renders them as an interactive execution graph, scores answers for faithfulness / hallucination, and tracks cost + latency.
- Backend FastAPI · SQLAlchemy 2 (async) · Pydantic v2 · PostgreSQL
- Frontend Next.js 14 (App Router) · TypeScript · Tailwind · React Flow · Recharts
- SDK
agentops_sdk— thin Python tracer - Infra Docker Compose
cp .env.example .env
docker compose up --build # db :5432, backend :8000, frontend :3000# in another shell, once backend is healthy:
pip install httpx # for the demo/seed scripts
python seed.py # seeds prompts + tests, runs demo agent, evaluatesOpen http://localhost:3000 (API docs at http://localhost:8000/docs).
- Dashboard — total runs, success rate, cost today, avg latency, cost-by-model, latency-by-stage.
- Trace Explorer — searchable/filterable run list.
- Run Detail — React Flow execution graph, per-node JSON inspector, cost breakdown, evaluation scorecard.
- Evaluations — failure analysis + per-run faithfulness / relevance / hallucination.
- Prompt Lab — version A/B: success-rate, cost, latency, hallucination deltas.
- Settings — API key + SDK snippet.
fromagentops_sdkimportAgentOpsao=AgentOps(base_url="http://localhost:8000", api_key="dev-key-change-me")
withao.run("support-agent", user_id="u1", user_query="refund policy?") asrun:
withrun.step("retrieval", "vector_search", query="refund policy") ass:
s.detail["documents"] = [{"text": "Refund period is 30 days.", "source": "policy.md",
"score": 0.9, "rank": 1, "used": True}]
withrun.step("llm", "answer", model="claude-haiku-4-5", provider="anthropic") ass:
s.detail.update(response="The refund period is 30 days.",
input_tokens=800, output_tokens=32, cost=0.0015)
run.finish("The refund period is 30 days.")Steps buffer client-side and flush in one POST /trace when the run exits. Exceptions
inside a run/step block are captured as status=failed — tracing never crashes your app.
Offline heuristics by default (token-overlap cosine) — zero cost, no API key.
To use an LLM judge, set in .env:
AGENTOPS_EVAL_LLM_PROVIDER=gemini
AGENTOPS_EVAL_LLM_API_KEY=<google-ai-studio-key>
AGENTOPS_EVAL_LLM_MODEL=gemini-2.5-flash
Both paths return {faithfulness, retrieval_relevance, hallucination_score, unsupported_claims}.
| Method | Path | Purpose |
|---|---|---|
| POST | /runs/create | open a run (auth: X-API-Key) |
| POST | /trace | submit steps |
| POST | /runs/end | close a run, roll up totals |
| GET | /runs | list runs (filters: status, agent_name, q) |
| GET | /runs/{id} | full trace + evaluation |
| POST | /evaluate/{id} | score a run |
| GET | /analytics/cost · /analytics/latency | analytics |
| POST | /prompts · GET /prompts/compare | prompt versions + A/B |
| POST | /tests/cases · /tests/run | regression suite |
| WS | /ws/traces | live trace stream |
users · agents · agent_runs · trace_steps · tool_calls · retrieval_events · llm_calls · evaluations · prompt_versions · test_cases · test_results · cost_metrics.
Schema is created on backend startup (create_all). Swap in Alembic when it churns.
backend/app/ models, schemas, routers/{ingest,query,evaluate,analytics,prompts,tests}, eval/engine, ws
sdk/ agentops_sdk tracer
demo_agent.py realistic traces (success / hallucinated / tool-failure)
seed.py prompts + test cases + demo + evaluate
docker-compose.yml
- Writes require the API key (
X-API-Key): ingest (/runs/*,/trace),/evaluate(prevents unauth triggering paid LLM-judge calls / economic DoS), and/prompts+/testswrites. - Read endpoints are open (
GET /runs,/runs/{id},/analytics/*,/prompts) so the keyless browser dashboard can load them. This exposes trace contents to anyone who can reach the API and is not multi-tenant-safe — a shared browser key is not a real secret. For production, front reads with per-user session auth and scope every query byuser_id. - Heuristic eval is bag-of-words cosine, not embeddings — swap
eval/engine._cosine. - Analytics aggregate on-read; precompute if run volume grows.