feat(rag): LanceDB retrieval, cited answers, and PDF fidelity fixes

[Rl0007]

Adds a retrieval layer and an Ask/RAG Lab UI over parsed documents, and fixes several pipeline defects found while running two real ICAI study PDFs through it.

Why

Wikify could turn PDFs into a browsable wiki, but not answer questions about them. The motivating case is "give me all the job descriptions across all the PDFs" — an exhaustive intent that similarity search answers badly, because top-k has no idea whether the right answer is 3 items or 300.

Measured on the demo corpus: naive top-8 vector search returns 8 sections across 4 of 5 documents (6 correct). A metadata filter returns all 15 across 5 of 5. Naive silently misses 9 sections and a whole document.

Retrieval — wikify/rag/

• LanceDB store with local model2vec embeddings (256d). No server, no API key, no GPU.
• Four modes: vector, fts, hybrid (RRF-fused), and filter — which returns every match with no top-k truncation.
• Contextual retrieval: chunks are embedded with a document > hierarchy path prefix (kept out of displayed text). Free here, because the section tree already existed.
• Parent-document retrieval: search small chunks, hand the model whole sections.
• Permissions are a pre-filter inside the query, never a post-hoc trim. An omitted ACL decision throws instead of searching everything.

Answering

• Intent router (exhaustive / semantic / hybrid) with follow-up rewriting. The decision and its plain-language reason are shown in the UI.
• Every citation carries a resolved page, line span and verbatim quote, verified before display. Verification is two gates: fuzzy prose match and exact equality on figures, signs and statutory references. A flipped (+) to (-) previously passed a 0.85 similarity check — for exam prep, a citation that renders as confirmed while misstating a rate is worse than none.
• Refusal requires two legs to agree: a below-floor rerank is overruled by a strong embedding match. This fixed a class of false refusals where the answer sat at rank 2 while the system said it could not find it.
• Per-question cost and token usage, measured correctly (see below).
• Conversations persist to Wikify Ask Session / Wikify Ask Message.

Pipeline fixes (found on real documents)

Defect	Effect
Sectioning aborted on the first heading longer than the 140-char title column, after deleting the old tree	Silently dropped 93.6% of a 236-page document. Coverage 6.4% to 99.8%
Page classification gated on chars < 250 AND drawings > 40	No born-digital diagram page can satisfy both. 1/236 to 231/236 routed correctly
Rate tables encoded as mermaid flowcharts	Row-to-rate correspondence destroyed. Now HTML tables; diagrams parsed, repaired and validated before storage
Verdicts frozen at the baseline parse	Remediated pages still read "review". 30 pass to 205 on one document
Remediation could adopt a near-empty candidate over a good one	A page's canonical content became # TIE TIT Molo at composite 0.058
search() re-sorted by fusion score after reranking, then sliced	The reranked winner was cut off. A covered question was refused while its answer sat at fusion rank 10
Cost counted only what the response body printed	Synthesis reports price via a litellm callback, so ~92% of the spend was recorded as $0.00. Real cost is $0.047/ask, not $0.003

Performance

Measured A/B on the same question, three runs each:

	before	after
Ask latency (median)	19.81s	16.00s
index_status index scans	one per project (40 on dev)	1
FTS rebuilds per propagation pass	up to 10	1
Redis round-trips per sectionise	~592	2
Mermaid parses per page	up to 9	1

Rerank batches now run concurrently — frappe.local is unbound on a pool thread, so the API key is resolved on the calling thread and passed in. Call count and spend are identical to the cent.

Frontend

/ask and /rag-lab: sources render before the answer, citation chips, page/line provenance, and a naive-vs-routed comparison showing what similarity search missed. Unranked results no longer draw a meaningless full score bar.

Mobile across the app: shell navigation, drill-downs replacing desktop splits, and graph views that degrade to a grouped list rather than an unreadable canvas.

Evaluation

12 golden questions scoring recall, precision and completeness, plus a self-contained HTML scorecard. Routed retrieval beats naive 66% to 87-90% recall on the demo corpus. Routing is an LLM call, so that is a range, not a point.

Known open items

Being explicit rather than presenting this as finished. These were surfaced by a structured review of this branch and are not fixed here:

1. session is two different identifiers sharing one name. The frontend mints a realtime correlation token; the backend treats it as a Wikify Ask Session docname. So every ask opens a new session, conversation history never replays, and has_permission on a non-existent name raises DoesNotExistError for non-Administrators — the first non-admin user to ask a question gets a 404. Administrator short-circuits the check, which is why testing missed it. Needs the field split into stream and session.
2. Propagation invalidation is on the wrong seam.Source Page.on_update fires for almost nothing in production, because every real write goes through frappe.db.set_value. Invalidation belongs in engine/store.py, which already calls itself the write funnel.
3. replace_sections is not atomic. The title clip removed one trigger, not the hazard: delete-then-insert with no savepoint, and the failure handler commits the partial tree and marks the import Review.
4. The mermaid gate re-implements mermaid's grammar in regex while the renderer uses the real parser client-side. Valid constructs it does not know are treated as broken and the diagram is discarded.
5. set_canonical_markdown does not invalidate the verdict, so an agent edit can leave "pass 0.99" on text nobody scored.
6. One eval test (test_g1) pins a naive correct_count that moved when the corpus was re-sectioned.
7. The CA-exam grading (7/12) predates the coverage, sectioning and reranker fixes and should be re-run.

Testing

test_rag_core (28), test_rag_api (38), test_evidence (32), test_diagrams (21), test_regions (8), test_sectionize (23), test_ask_cost (6), test_ask_history (16), test_page_propagation (13), test_remediate_adoption (12), test_canonical_verdict (4), plus the pre-existing suites for blast radius. Verified end to end against wikify.localhost with two real 180- and 236-page ICAI documents.

[greptile-apps]

Too many files changed for review (130 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.