Skip to content

feat(scan): on-device receipt parser, OCR fixes and a review screen - #43

Merged
Ivan-LB merged 3 commits into
devfrom
feature/receipt-scan-parser
Aug 18, 2026
Merged

Ivan-LB merged 3 commits into
devfrom
feature/receipt-scan-parser

Conversation

@Ivan-LB

@Ivan-LB Ivan-LB commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Summary

  • Spike first: docs/spikes/receipt-scan-spike.md — what the scanner does today (with the extraction code run verbatim on fixtures), how the industry does it (Expensify/Zoho/Wave/Dext/Fetch = cloud OCR + humans; indie apps market on-device), Apple's free on-device stack verified against the iOS SDK (RecognizeDocumentsRequest + DataDetection.moneyAmount, VisionKit document camera / .currency live scan, Foundation Models), free/OSS/cloud options evaluated against "money data never leaves the phone", and the decisions (D1–D8).
  • Pure ReceiptParser (Utilities/Receipt/): rows clustered from OCR line boxes, receipt amount grammar (thousands separators, comma decimals, whole pesos, OCR digit confusions, %/dates/ids ignored), es/en label hierarchy with exclusions (SUBTOTAL, IVA, PROPINA, EFECTIVO, CAMBIO, TARJETA, PUNTOS…), scored candidates + ranked alternatives, merchant (brand map + prettifying), printed date (dd/mm vs mm/dd by language signal, plausibility window), category hint onto the five chips.
  • OCR fixes (Utilities/ReceiptOCR.swift): image orientation is now passed to Vision (was dropped → portrait receipts read sideways), recognition languages es+en resolved at runtime (was en-US default), single RecognizeDocumentsRequest with Apple's money/date detectors as a second opinion, RecognizeTextRequest fallback, decode/downscale off the main actor.
  • Camera (CameraManager rewrite): RotationCoordinator for correct EXIF on stills, guarded async capturePhoto() (the old shutter could crash on an unconfigured session), torch, gallery import (PhotosPicker), the never-drawn per-frame rectangle detection removed, retain cycle gone. VNDocumentCameraViewController wrapped behind FeatureFlags.useSystemDocumentCamera (default off) so both capture surfaces can be A/B'd on a real device.
  • Review screen replaces the disabled-text-field card: thumbnail, editable amount (keypad), "also on the receipt" chips, merchant, date (from the receipt / today), category chips pre-selected, honest copy for low confidence / nothing found, failed state that never dead-ends, VoiceOver announcements + modal overlays, Warm Meadow tokens, all strings via Strings.swift with es-419. Both entry points (+ sheet and Money tab) migrated. Saved rows now carry merchant + receipt date + category (was "Recibo escaneado", now, nil).
  • Deleted CameraViewModel, TextRecognizer, OCRUtilities (dead), TotalConfirmationView. CLAUDE.md invariant 7 corrected + new invariant 10; gotchas.yaml entry.

Why

The old scanner systematically recorded the cash tendered instead of the total on Mexican receipts (TOTAL / EFECTIVO / CAMBIO → 100.00 instead of 41.18), dropped leading digits (1234.56 → 234.56), rejected coffees (< 10), read portrait photos without orientation, and none of it was testable. Spike details in docs/spikes/receipt-scan-spike.md §1.2.

Test plan

  • xcodebuild build -scheme Savely (iPhone 17 Pro sim, Xcode 27 beta locally)
  • xcodebuild test -scheme Savely — 102/102 unit tests, incl. new ReceiptParserTests, ReceiptFixtureTests (16 hand-authored JSON fixtures), ReceiptOCRIntegrationTests (real Vision on a script-rendered receipt: total 57.50, merchant, date, subtotal alternative; and the same bitmap rotated + tagged sideways still reads)
  • swiftlint --strict clean on all touched files
  • Simulator UI test ReceiptScanUITests (needs xcrun simctl addmedia <udid> SavelyTests/Fixtures/Images/synthetic-oxxo.png): + → Scan a receipt → Choose from Photos → review shows 57.50 / OXXO / 14-08 / Food → Save → row appears in Money
  • Manual on device (cannot be done in the Simulator — no camera): shutter capture in portrait and landscape, torch, "Open Settings" round trip after denying access, and flip FeatureFlags.useSystemDocumentCamera = true to compare the system document camera
  • Manual: collect real es-MX/US receipts with the DEBUG "Export OCR dump" button, anonymize, drop into SavelyTests/Fixtures/Receipts/ (this is where accuracy numbers will come from — the current fixtures prove rules, not real-world accuracy)

Risks

  • Fixtures are hand-authored layouts (each says so in source); real-receipt accuracy is unmeasured until phase 2. Everything the parser produces is a draft the user confirms — nothing is saved silently.
  • ExpenseTrackerViewModel.addExpense(description:amount:date:category:) no longer flips showError (the scanner shows the message inline); the form path still does. Two hardcoded error literals there moved to Strings.Errors.
  • pbxproj edited by script (files added/removed, SavelyTests/Fixtures folder reference in the test target's Resources phase). Please open once in Xcode.
  • Locally built with Xcode 27 beta / iOS 27 SDK; CI pins Xcode 26.0 — the APIs used are iOS 26 (RecognizeDocumentsRequest, DataDetection), so CI should compile; watch the first run.

Checklist

  • Branch named with approved prefix (feature/)
  • Commits follow Conventional Commits
  • No hardcoded user-facing strings (all routed through Strings.swift)
  • No secrets staged
  • CLAUDE.md / knowledge files updated
  • Tests added or updated for behavior changes

Rewrite the receipt scanner around a pure, fixture-tested ReceiptParser
and fix the Vision pipeline around it. All on-device; no network.

Parser (Utilities/Receipt/): rows clustered from line boxes, a receipt
amount grammar (thousands separators, comma decimals, whole pesos, OCR
digit confusions, %/dates/ids ignored), es/en label hierarchy with
exclusions (SUBTOTAL, IVA, PROPINA, EFECTIVO, CAMBIO, TARJETA...),
scored candidates with alternatives, merchant, printed date, category
hint. Fixes the old heuristics that saved EFECTIVO instead of TOTAL,
dropped leading digits ("1234.56" -> 234.56) and rejected totals < 10.

OCR (Utilities/ReceiptOCR.swift): RecognizeDocumentsRequest with the
image orientation passed and es+en recognition languages; Apple's
money/date detectors as a second opinion; RecognizeTextRequest fallback.

Camera: RotationCoordinator so stills carry the right EXIF orientation,
guarded async capture (no crash on an unconfigured session), torch,
gallery import, no per-frame rectangle detection; VNDocumentCamera
behind FeatureFlags.useSystemDocumentCamera for a device A/B.

UI: ReceiptScanFlowView (capture -> reading -> review) replaces the
disabled-text-field card: editable amount, alternative chips, merchant,
date, category, thumbnail, honest empty/failed states, VoiceOver
announcements and modal overlays, Warm Meadow tokens, all strings in
Strings.swift with es-419.

Tests: 16 JSON fixtures + grammar/date/label unit tests, a real-Vision
integration test on a script-rendered receipt (upright and rotated),
and a Simulator UI test through the Photos picker.
@Ivan-LB
Ivan-LB merged commit 3ceedbe into dev Aug 18, 2026
1 check passed
@Ivan-LB
Ivan-LB deleted the feature/receipt-scan-parser branch August 18, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant