feat(scan): on-device receipt parser, OCR fixes and a review screen - #43
Merged
Merged
Conversation
Rewrite the receipt scanner around a pure, fixture-tested ReceiptParser
and fix the Vision pipeline around it. All on-device; no network.
Parser (Utilities/Receipt/): rows clustered from line boxes, a receipt
amount grammar (thousands separators, comma decimals, whole pesos, OCR
digit confusions, %/dates/ids ignored), es/en label hierarchy with
exclusions (SUBTOTAL, IVA, PROPINA, EFECTIVO, CAMBIO, TARJETA...),
scored candidates with alternatives, merchant, printed date, category
hint. Fixes the old heuristics that saved EFECTIVO instead of TOTAL,
dropped leading digits ("1234.56" -> 234.56) and rejected totals < 10.
OCR (Utilities/ReceiptOCR.swift): RecognizeDocumentsRequest with the
image orientation passed and es+en recognition languages; Apple's
money/date detectors as a second opinion; RecognizeTextRequest fallback.
Camera: RotationCoordinator so stills carry the right EXIF orientation,
guarded async capture (no crash on an unconfigured session), torch,
gallery import, no per-frame rectangle detection; VNDocumentCamera
behind FeatureFlags.useSystemDocumentCamera for a device A/B.
UI: ReceiptScanFlowView (capture -> reading -> review) replaces the
disabled-text-field card: editable amount, alternative chips, merchant,
date, category, thumbnail, honest empty/failed states, VoiceOver
announcements and modal overlays, Warm Meadow tokens, all strings in
Strings.swift with es-419.
Tests: 16 JSON fixtures + grammar/date/label unit tests, a real-Vision
integration test on a script-rendered receipt (upright and rotated),
and a Simulator UI test through the Photos picker.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/spikes/receipt-scan-spike.md— what the scanner does today (with the extraction code run verbatim on fixtures), how the industry does it (Expensify/Zoho/Wave/Dext/Fetch = cloud OCR + humans; indie apps market on-device), Apple's free on-device stack verified against the iOS SDK (RecognizeDocumentsRequest+DataDetection.moneyAmount, VisionKit document camera /.currencylive scan, Foundation Models), free/OSS/cloud options evaluated against "money data never leaves the phone", and the decisions (D1–D8).ReceiptParser(Utilities/Receipt/): rows clustered from OCR line boxes, receipt amount grammar (thousands separators, comma decimals, whole pesos, OCR digit confusions,%/dates/ids ignored), es/en label hierarchy with exclusions (SUBTOTAL, IVA, PROPINA, EFECTIVO, CAMBIO, TARJETA, PUNTOS…), scored candidates + ranked alternatives, merchant (brand map + prettifying), printed date (dd/mm vs mm/dd by language signal, plausibility window), category hint onto the five chips.Utilities/ReceiptOCR.swift): image orientation is now passed to Vision (was dropped → portrait receipts read sideways), recognition languages es+en resolved at runtime (was en-US default), singleRecognizeDocumentsRequestwith Apple's money/date detectors as a second opinion,RecognizeTextRequestfallback, decode/downscale off the main actor.CameraManagerrewrite):RotationCoordinatorfor correct EXIF on stills, guarded asynccapturePhoto()(the old shutter could crash on an unconfigured session), torch, gallery import (PhotosPicker), the never-drawn per-frame rectangle detection removed, retain cycle gone.VNDocumentCameraViewControllerwrapped behindFeatureFlags.useSystemDocumentCamera(default off) so both capture surfaces can be A/B'd on a real device.Strings.swiftwith es-419. Both entry points (+sheet and Money tab) migrated. Saved rows now carry merchant + receipt date + category (was"Recibo escaneado", now, nil).CameraViewModel,TextRecognizer,OCRUtilities(dead),TotalConfirmationView.CLAUDE.mdinvariant 7 corrected + new invariant 10;gotchas.yamlentry.Why
The old scanner systematically recorded the cash tendered instead of the total on Mexican receipts (
TOTAL / EFECTIVO / CAMBIO→ 100.00 instead of 41.18), dropped leading digits (1234.56→ 234.56), rejected coffees (< 10), read portrait photos without orientation, and none of it was testable. Spike details indocs/spikes/receipt-scan-spike.md§1.2.Test plan
xcodebuild build -scheme Savely(iPhone 17 Pro sim, Xcode 27 beta locally)xcodebuild test -scheme Savely— 102/102 unit tests, incl. newReceiptParserTests,ReceiptFixtureTests(16 hand-authored JSON fixtures),ReceiptOCRIntegrationTests(real Vision on a script-rendered receipt: total 57.50, merchant, date, subtotal alternative; and the same bitmap rotated + tagged sideways still reads)swiftlint --strictclean on all touched filesReceiptScanUITests(needsxcrun simctl addmedia <udid> SavelyTests/Fixtures/Images/synthetic-oxxo.png):+→ Scan a receipt → Choose from Photos → review shows 57.50 / OXXO / 14-08 / Food → Save → row appears in MoneyFeatureFlags.useSystemDocumentCamera = trueto compare the system document cameraSavelyTests/Fixtures/Receipts/(this is where accuracy numbers will come from — the current fixtures prove rules, not real-world accuracy)Risks
source); real-receipt accuracy is unmeasured until phase 2. Everything the parser produces is a draft the user confirms — nothing is saved silently.ExpenseTrackerViewModel.addExpense(description:amount:date:category:)no longer flipsshowError(the scanner shows the message inline); the form path still does. Two hardcoded error literals there moved toStrings.Errors.pbxprojedited by script (files added/removed,SavelyTests/Fixturesfolder reference in the test target's Resources phase). Please open once in Xcode.RecognizeDocumentsRequest,DataDetection), so CI should compile; watch the first run.Checklist
feature/)Strings.swift)CLAUDE.md/ knowledge files updated