Quickstart

Quickstart

Get EvalCI running in under 5 minutes.

→ Full docs: https://synapsekit.github.io/synapsekit-docs/docs/evalci/quickstart

---

Step 1 — Write an eval case

Create a file named eval_*.py in your test directory:

# tests/evals/eval_rag.pyfromsynapsekitimporteval_casefrommyapp.pipelineimportrag_pipeline@eval_case(min_score=0.80, max_cost_usd=0.01, max_latency_ms=3000)asyncdefeval_rag_relevancy():
result=awaitrag_pipeline.ask("What is SynapseKit?")
score=awaitresult.score_relevancy(
reference="SynapseKit is a Python framework for building LLM applications."
)
return {"score": score, "cost_usd": result.cost_usd, "latency_ms": result.latency_ms}

---

Step 2 — Add the workflow

# .github/workflows/eval.ymlname: EvalCIon:
pull_request:
jobs:
eval:
runs-on: ubuntu-lateststeps:
- uses: actions/checkout@v4
- uses: SynapseKit/evalci@v1with:
path: tests/evalsthreshold: "0.80"env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

---

Step 3 — Add your API key

1. Go to your repo → Settings → Secrets and variables → Actions
2. Click New repository secret
3. Name: OPENAI_API_KEY, Value: your key
4. Click Add secret

---

Step 4 — Open a PR

Push and open a pull request. EvalCI posts a results comment and passes/fails the check automatically.

---

Local testing

pip install synapsekit[openai]
synapsekit test tests/evals/ --threshold 0.80