rag-eval

Evaluation toolkit for RAG (Retrieval-Augmented Generation) pipelines. Measures what matters at each stage: chunk quality, retrieval quality, and answer quality — all without calling an LLM judge.

What it covers

Stage	Metrics
Chunking	Fixed-size (with overlap), paragraph boundary, sentence boundary
Retrieval	Context Precision, Context Recall, NDCG@K, MRR
Answer quality	Faithfulness (trigram overlap), Answer Relevance (Jaccard)
End-to-end	RagEvaluator.eval() runs all stages in one call

Install

pip install -e .

Quickstart

fromragevalimportRagEvaluatordefmy_retriever(query, docs, k):
# plug in your actual retriever
...
evaluator=RagEvaluator(retriever=my_retriever)
result=evaluator.eval(
query="What causes transformer attention to scale quadratically?",
docs=corpus,
relevant_ids=[2, 7],
answer="Attention computes pairwise dot products across all tokens, growing as O(n²) in sequence length.",
k=5,
)
print(result)

query: "What causes transformer attention to scale quadratically?"
k: 5
── Retrieval ─────────────────────────────────────
context_precision: 0.8000
context_recall: 1.0000
ndcg: 0.9307
mrr: 1.0000
── Answer ────────────────────────────────────────
faithfulness: 0.8571
answer_relevance: 0.6667

Chunking strategies

fromrageval.chunkingimportchunk_fixed, chunk_by_paragraph, chunk_by_sentencetext=open("doc.txt").read()
chunks=chunk_fixed(text, size=512, overlap=64)
chunks=chunk_by_paragraph(text, min_size=64, max_size=1024)
chunks=chunk_by_sentence(text, max_size=512)

Retrieval metrics

fromrageval.metricsimportcontext_precision, context_recall, ndcg_at_k, mrrretrieved= ["doc_3", "doc_1", "doc_7", "doc_9", "doc_2"]
relevant= {"doc_1", "doc_7"}
print(context_precision(retrieved, relevant, k=5)) # 0.4print(context_recall(retrieved, relevant)) # 1.0print(ndcg_at_k(retrieved, relevant, k=5)) # 0.6934print(mrr(retrieved, relevant)) # 0.5

Answer quality metrics

fromrageval.metricsimportfaithfulness, answer_relevancecontext="Transformers compute attention across all token pairs, giving O(n²) complexity."answer="Attention scales quadratically because every token attends to every other token."query="Why is attention quadratic?"print(faithfulness(answer, context)) # 0.75 — fraction of answer sentences grounded in contextprint(answer_relevance(answer, query)) # 0.60 — Jaccard on non-stopword tokens

Design

No LLM judge. Faithfulness uses trigram overlap against the retrieved context. Answer relevance uses Jaccard similarity on non-stopword tokens. Both are fast, deterministic, and need no API key — suitable for CI.

Pluggable retriever.RagEvaluator takes any (query, docs, k) → list[tuple[doc, score]] callable. Swap in BM25, a vector store, or a mock for testing.

Testing

pytest -q

...........................................
43 passed in 0.08s