EvalCI by SynapseKit

LLM quality gates for every PR. Run your @eval_case suites automatically and block merge if quality drops below threshold.

• Zero infrastructure — runs entirely in GitHub Actions
• 2-minute setup
• Works with any LLM provider (OpenAI, Anthropic, Gemini, and 35 more)
• Posts a formatted results table as a PR comment
• Sets Action outputs for downstream steps

---

Quickstart

Add .github/workflows/eval.yml to your repo:

name: EvalCIon:
pull_request:
jobs:
eval:
runs-on: ubuntu-lateststeps:
- uses: actions/checkout@v4
- uses: SynapseKit/evalci@v1with:
path: tests/evalsthreshold: "0.80"env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

That's it. EvalCI will:

1. Install synapsekit into the runner
2. Discover and run all @eval_case-decorated functions under tests/evals/
3. Post a results table as a PR comment
4. Fail the check if any case scores below threshold

---

Example eval file

# tests/evals/test_rag.pyfromsynapsekit.testingimporteval_case@eval_case(min_score=0.80, max_cost_usd=0.01, max_latency_ms=3000)deftest_rag_relevancy(eval_context):
result=my_rag_pipeline("What is SynapseKit?")
returneval_context.score_relevancy(result, reference="SynapseKit is a Python library...")
@eval_case(min_score=0.75)deftest_rag_faithfulness(eval_context):
result=my_rag_pipeline("How do I install SynapseKit?")
returneval_context.score_faithfulness(result, context=retrieved_docs)

---

PR Comment

EvalCI posts a comment like this on every PR:

EvalCI Results

	Test	Score	Cost	Latency
✅	test_rag_relevancy	0.850	$0.0050	1200ms
❌	test_rag_faithfulness	0.650	$0.0120	2500ms

1/2 passed · Threshold: 0.80 · SynapseKit EvalCI

---

Inputs

Input	Description	Default
path	Path to eval files or directory	.
threshold	Global minimum score (0.0–1.0)	0.7
extras	pip extras for synapsekit (e.g. openai,anthropic)	openai
synapsekit-version	synapsekit version to install, or latest	latest
github-token	Token for posting PR comments	${{ github.token }}
fail-on-regression	Fail if score regresses vs. baseline	false
token	EvalCI backend API token (future)	—

Outputs

Output	Description
passed	Number of eval cases that passed
failed	Number of eval cases that failed
total	Total number of eval cases run
mean-score	Mean score across all eval cases

---

Using outputs in downstream steps

- uses: SynapseKit/evalci@v1id: evalwith:
path: tests/evals
- run: | echo "Passed: ${{ steps.eval.outputs.passed }}/${{ steps.eval.outputs.total }}" echo "Mean score: ${{ steps.eval.outputs.mean-score }}"

---

Multiple providers

- uses: SynapseKit/evalci@v1with:
extras: "openai,anthropic"threshold: "0.75"env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

---

Badge

[![EvalCI](https://github.com/{owner}/{repo}/actions/workflows/eval.yml/badge.svg)](https://github.com/{owner}/{repo}/actions/workflows/eval.yml)

---

Documentation

Full documentation is available at synapsekit.github.io/synapsekit-docs/docs/evalci/overview

Overview	What EvalCI is and how it works
Quickstart	Set up in 5 minutes
Writing eval cases	How to write @eval_case functions
Action reference	All inputs, outputs, and configuration
Examples	RAG, agents, multi-provider workflows

About

EvalCI is built on SynapseKit — an async-native Python framework for LLM applications with 35 provider integrations, RAG pipelines, agents, graph workflows, verifiable audit trails, and a built-in evaluation framework.

• Documentation
• SynapseKit
• Issues
• Discussions