A local RAG evaluation pipeline on real SEC filings (Apple FY 2024/2025 10-Ks, Microsoft FY 2025 10-K, Apple Q4 FY2025 earnings) — built to understand where financial document retrieval breaks down, not just to make a demo that works. The work here surfaced a real metric bug that is now merged into DeepEval.
Financial analysts and AI teams building RAG over SEC filings face a dangerous failure mode: the system answers confidently even when it shouldn't. Standard RAG evals don't catch the difference between "I don't know" (good) and "Revenue was $387.2B" when the actual number is $383.3B (dangerous). This project measures that gap on real data.
An end-to-end local RAG pipeline — SEC EDGAR ingestion through answer generation — with a 26-question evaluation suite grounded against real figures from Apple (FY 2024 + FY 2025 10-Ks and the Q4 FY2025 earnings release) and Microsoft (FY 2025 10-K). Questions span two issuers and three document types (10-K narrative, balance-sheet statements, earnings-call/guidance), each scored against a document-type-specific pass threshold (see Heterogeneous thresholds).
SEC EDGAR (10-K PDF)
→ pdfplumber (section-aware PDF parsing)
→ Chunker (configurable overlap, section boundaries preserved)
→ Ollama nomic-embed-text (local embeddings)
→ Supabase pgvector / Docker (local vector store)
→ Top-k retrieval
→ Ollama llama3 (local LLM, $0 API cost)
→ Metadata-aware section routing (optional re-rank by query intent)
→ Evaluation layer (DeepEval metrics + custom section-aware precision/recall,
multi-hop per-hop scoring, table-extraction quality, eval→action feedback)
| Layer | Tool | Why |
|---|---|---|
| PDF parsing | pdfplumber | Handles financial table extraction reasonably well |
| Embeddings | nomic-embed-text | Local, free, strong on financial terminology |
| Vector store | Supabase + pgvector | SQL + vector in one; production-representative |
| LLM | llama3 via Ollama | Fully local — no API costs, reproducible |
| Orchestration | Plain Python | Debuggable; no hidden abstractions |
The first 3-question probe against Apple's FY 2024 10-K (ground truth pulled directly from the filing) is what exposed the core failure mode:
| Question | Ground Truth | System Response | Verdict |
|---|---|---|---|
| Total net revenue vs FY 2023? | $391.0B (+2% YoY) | Refused — "not found in context" | ✅ Honest refusal |
| Gross margin %? | 46.2% | Refused — "not found in context" | ✅ Honest refusal |
| R&D spend + % of revenue? | $31.4B / 8.0% | Answered confidently with wrong figures | ❌ Confident hallucination |
Key finding: 2/3 honest refusals. 1/3 confident hallucination with precise but incorrect numbers.
The dangerous failure mode is not "I don't know" — it's "The answer is X" where X is wrong and sounds credible.
The dataset (src/eval/eval_dataset.json) now holds 26 questions grounded against real filings from two issuers:
| Dimension | Split |
|---|---|
| Issuer | 22 × Apple, 4 × Microsoft |
| Document type | 13 × 10k_filing, 9 × balance_sheet, 4 × earnings_call |
| Source period | Apple FY2024 + FY2025 10-Ks, Apple Q4 FY2025 earnings release, Microsoft FY2025 10-K |
Every figure is sourced from filings and official press releases (e.g., Apple FY2025 revenue $416.2B, Apple Q4 FY2025 revenue $102.5B, Microsoft FY2025 revenue $281.7B / net income $101.8B). No figures are fabricated.
A single pass threshold across all financial document types is wrong: structured statements (a balance sheet is either right or it isn't) should tolerate zero hallucination, while hedged / forward-looking narrative can be scored more leniently. The runner picks the threshold per question from its document_type:
| Document type | Pass threshold |
|---|---|
balance_sheet | 0.95 |
10k_filing | 0.85 |
annual_report | 0.80 |
earnings_call | 0.70 |
This is applied client-side today (each metric is constructed with the resolved threshold) because DeepEval's native threshold_overrides argument is still landing upstream — see Issue #2775 and the demo in PR #2790. The custom section-aware precision metric (src/eval/metrics.py) runs alongside DeepEval's built-ins and groups overlapping chunks by doc_id:section_id so redundant retrieval windows count as a single relevant unit.
Running DeepEval's ContextualPrecisionMetric on this pipeline exposed a metric-level bug: overlapping chunks (10-20% overlap, standard for preserving table/section boundaries) were being penalized as independent retrieval failures — making eval scores worse as chunk quality improved.
That finding became GitHub Issue #2594, and the fix I authored — grouping retrieval contexts by source and correcting the weighted cumulative precision formula — was merged into DeepEval as PR #2743.
The eval found a bug in the eval framework, and the fix shipped. That's the point.
This project is the real-world workload that surfaced several issues in DeepEval (Confident AI's open-source LLM eval framework). The custom metrics in src/eval/metrics.py are the local prototypes of ideas being upstreamed:
| # | Type | Status | Contribution |
|---|---|---|---|
| #2743 | PR | ✅ Merged | ContextualPrecisionMetric: source-grouping of retrieval contexts + weighted cumulative precision formula fix |
| #2594 | Issue | Closed | Contextual Precision over-penalizes overlapping chunks in financial-document RAG (the original bug report) |
| #2775 | Issue | Open | Feature: eval metrics for heterogeneous financial document chunks / per-document-type thresholds |
| #2788 | Issue | Open | Bug: ContextualRecallMetric over-penalises overlapping chunks (parallel to #2594) |
| #2790 | PR | Open | Docs example: heterogeneous financial document RAG evaluation with threshold_overrides |
| #2789 | PR | Open | Regression fixtures for ContextualRecallMetric overlapping chunks (closes #2788) |
| #2787 | PR | Open | Regression fixtures for ContextualPrecisionMetric overlapping chunks (rebased on #2743) |
| #2819 | PR | Open | AgentLoopDetectionMetric — detect infinite loops / cyclical tool-call patterns in agent traces |
The overlapping-chunk penalty documented in Lessons Learned is exactly the behavior fixed in #2743 and being hardened by the regression PRs above.
The eval ideas I've proposed across the RAG/eval ecosystem aren't just issues — each is implemented and verified here, on real Apple filings. This repo is the reference implementation behind the proposals:
| Idea (upstream proposal) | Where proposed | Implementation in this repo |
|---|---|---|
| Overlapping-chunk precision (source grouping) | deepeval #2594 → merged #2743 | section_aware_precision + chunk_dedup.py |
| Overlapping-chunk recall (union coverage) | deepeval #2788/#2789 | section_aware_recall — recall is monotonic under redundancy |
| Per-document-type thresholds | deepeval #2775/#2790 | THRESHOLD_OVERRIDES + document_type dataset tagging |
| Metadata-aware section routing | llama_index #22032/#21862 | retrieval/section_router.py (opt-in metadata_routing) |
| Multi-hop per-hop quality | mastra #18258, phoenix #13407, openinference #3256, weave #6946/#7280 | eval/hop_scorer.py + multihop_dataset.json — surfaces the weakest hop |
| Structured table-extraction quality | firecrawl #3587/#3817 | eval/table_eval.py + table_ground_truth.json (real 10-K cells) |
| Eval → action feedback loop | langsmith #2929 | eval/feedback_loop.py — maps weak metrics to config knobs (recommend-only) |
| Automated numeric-hallucination detection | addresses the project's core "confident wrong number" thesis | eval/hallucination.py — extracts asserted figures, flags any unsupported by context |
git clone https://github.com/Ruthwik-Data/finrag-eval
cd finrag-eval
docker-compose up -d
python scripts/init_db.py
# Download the two most recent 10-Ks (FY 2024 + FY 2025) and ingest each
python src/ingestion/edgar_download.py --ticker AAPL --count 2
python src/ingestion/ingest.py --input data/raw/<downloaded-filing>.htm --ticker AAPL
# Query and evaluate
python src/retrieval/query.py "What was Apple's total net revenue for fiscal year 2025?"
python src/eval/run_eval.py # full 26-question suite
python src/eval/run_eval.py --compare # raw vs. deduped comparisonEach eval question prints its doc_type and the pass threshold applied to it. Manual verdicts from the initial probe are in notes.
The custom metrics are decoupled from DeepEval (the framework is an optional import), so the full metric suite is unit-tested with no external stack — no DB, no Ollama, no LLM calls:
make dev # install dev/test deps
make test# run the suite (also runs in CI on every push/PR)Covered by tests: section-aware precision/recall (incl. redundancy-monotonicity), metadata routing, multi-hop hop-scoring, table-extraction quality, the feedback loop, numeric-hallucination detection, threshold resolution, and dataset integrity. CI runs on Python 3.11/3.12 via GitHub Actions (.github/workflows/ci.yml).
The project's thesis — overlap helps retrieval but hurts naive eval — is isolated with three runs:
make ablation # runs raw / --deduplicate / --metadata-routing and stores each- Confident hallucination is worse than refusal. A system that says "I don't know" is safer than one that gives a precise wrong number. Calibrating refusal behavior is a product decision, not just a technical one.
- Overlap helps retrieval, hurts naive evals. Increasing chunk overlap improved answer grounding but lowered DeepEval precision scores — because the metric penalized redundant chunks as misses. Evaluation metrics can lie about retrieval quality.
- Local-first forced honesty. Running fully locally ($0 API cost) meant I couldn't rely on GPT-4 to paper over weak retrieval. The results are less polished but more honest.
- 26-question eval is illustrative, not statistically significant
- Ground truth extracted from Apple/Microsoft filings and official press releases — possible human error on edge cases
- Retrieval scores require the relevant filings (Apple FY2024/FY2025 10-Ks, Microsoft FY2025 10-K, Apple Q4 FY2025 release) to be ingested; dataset labels are provided regardless of what is currently in the vector store
- llama3 locally is weaker than GPT-4 class models
- The custom section-aware / hallucination metrics use keyword/figure heuristics (an optional embedding relevance mode exists); they are designed to demonstrate the failure modes, not to replace an LLM judge
- Section detection in pdfplumber is heuristic and may miss boundaries in complex filings