Spec 0.7 — pdf-inspector rescue candidate + semantic retrieval

[NagariaHussain]

Specs only — slices 32–35. No implementation.

Why?

Two gaps, both measured against the real corpus before writing anything.

Remediation pays an LLM to fix what a free parser already fixes. Every page goes through the remediation pass, so a 398-page manual costs ~800 model calls. But on the pages the harness flags, a deterministic Rust parser (pdf-inspector) recovers the text almost perfectly at ~5ms/page and zero cost — content_recall 0.565 → 0.998 on review pages.

Nothing can find content by meaning.search_sections substring-matches the whole query against title/hierarchy_path and never reads a body. Measured over 180 generated questions: recall@10 = 0.000. Not weak — zero.

What?

Slice
32	Structure guard + pdf-inspector adapter
33	Inspector as a remediation candidate + LLM short-circuit
34	LanceDB retrieval spine + reindex + eval harness
35	search_wiki agent tool

Two independent tracks: 32→33 and 34→35. Land 32 first.

How?

pdf-inspector is a candidate, never the baseline.pymupdf4llm stays the baseline. Inspector output is only ever adopted per-page, through the existing best-of-N adoption in remediate.py.

This is the load-bearing decision. Inspector scores 0.997 content_recall vs 0.666 (n=120) — but on the obs-gyn manual it drops headings 132 → 10 and list items 320 → 0, while inflating table rows 35 → 220. sectionize splits on #, so adopting that would silently flatten the section tree while every score in the UI went up. parser_artifacts can't catch it — its patterns are tuned to pymupdf4llm's failure signatures.

So structure_preserved() becomes a hard eligibility gate. Calibrated on the measurement, it rejects 36/60 obs-gyn adoptions and 7/60 on Nephrology — it blocks the document where inspector destroys structure and admits the one where it doesn't.

Because inspector is free and computed before the paid candidates, a page it already carries past pass_threshold skips the VLM and cleanup calls entirely. That's the cost win.

Retrieval ships pure vector, not hybrid. Vector 0.794 recall@10 vs 0.611 full-text; default RRF hybrid measured worse than vector alone (0.728) because weak FTS ranking drags the fusion down. Vector's margin is +0.32 on semantic and +0.20 on vague queries, ~0 on keyword — keyword is already covered by frappe-wiki's existing FTS5 index.

Index target is Wiki Document. Embeddings via litellm.embedding (already a dependency). Five scattered Wiki Document.content writers consolidate behind one store.set_wiki_content() funnel, mirroring 0.6's set_section_markdown move.

Notes

• Based on main, not develop — develop isn't on the remote and the project guide says it has unrelated history.
• Out-of-scope findings recorded in the plan so they aren't mistaken for 0.7 fallout: images.py:71-103 Case 1 is already dead code (its marker fires 0 times across all 578 pages of both manuals), and pymupdf4llm silently invokes Tesseract OCR on hundreds of pages.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ck1Fqm4WELZj2GDFQSx5CW