Skip to content

Consume Doc-Lineage #3 shared PDF extraction instead of pdf-primary regex path #947

Description

@stranske

Why

The work-environment appendix names one shared “extract structured facts from a PDF, with page pointers” library instead of three implementations, reducing three maintenance surfaces to one (INFORMATION-REQUEST-RESPONSE.md Appendix §What looks worth systematizing, item 2). Inv-Man-Intake currently uses PdfPrimaryExtractionProvider for pyodide-light (src/inv_man_intake/extraction/providers/pdf_primary.py:20-26) and optionally stranske_pdf_extract Docling (src/inv_man_intake/extraction/service.py:104-115, 118-128); it lacks a Doc-Lineage adapter once Doc-Lineage #3 lands.

Scope

Replace the in-repo PDF primary-provider path with a thin adapter to the Doc-Lineage extraction library, including page pointers and mandatory OCR fallback, while retaining the ExtractionService port at src/inv_man_intake/extraction/service.py:27-35.

Non-Goals

Do not block on Doc-Lineage #3 merging; use a feature flag or optional extra until a stable import path publishes, then remove pdf-primary. Do not remove pptx_primary.py or implement the Doc-Lineage comparison/lineage engine. An unused adapter while build_pyodide_light_service() defaults to PdfPrimaryExtractionProvider is not sufficient.

Tasks

  • Add src/inv_man_intake/extraction/providers/doc_lineage_adapter.py wrapping the Doc-Lineage ci(deps): bump actions/download-artifact from 7 to 8 #3 entry point and mapping results to ExtractedDocumentResult in src/inv_man_intake/extraction/providers/base.py:8-13.
  • Add the extraction-doc-lineage optional dependency group in pyproject.toml for the package exposing doc_lineage.extract.
  • Update build_pyodide_light_service() in src/inv_man_intake/extraction/service.py:104-115 to select DocLineageExtractionProvider when installed and temporarily fall back to PdfPrimaryExtractionProvider otherwise.
  • Add tests/extraction/providers/test_doc_lineage_adapter.py with a synthetic PDF fixture and expected page pointers.
  • Update tests/extraction/test_extraction_service.py to assert Doc-Lineage selection when the extra is installed.
  • Test the deliberate break by returning page=None from doc_lineage_adapter.py, capture the failing test output, and revert the change.

Acceptance Criteria

  • tests/extraction/providers/test_doc_lineage_adapter.py::test_adapter_emits_page_pointers passes and verifies SourceLocation.page values from Doc-Lineage.
  • tests/extraction/test_extraction_service.py::test_pyodide_light_prefers_doc_lineage_when_installed passes with the extraction-doc-lineage extra installed.
  • Changing doc_lineage_adapter.py to return page=None causes tests/extraction/providers/test_doc_lineage_adapter.py::test_adapter_emits_page_pointers to fail; the change is reverted afterward.

Implementation Notes

Reference Doc-Lineage #3 for page pointers and OCR fallback. The stranske_pdf_extract Docling path in service.py:118-128 should delegate to Doc-Lineage after #3 to avoid two OCR implementations. Current baseline: python -m pytest tests/extraction/test_pdf_primary_provider.py -q passes on main.

Original Issue
## Why

The work-environment appendix names **one shared "extract structured facts from a PDF, with page pointers" library** instead of three independently built implementations, and states consolidating would cut three maintenance surfaces to one ([INFORMATION-REQUEST-RESPONSE.md Appendix §What looks worth systematizing, item 2](https://github.com/stranske/Ready/blob/main/research-program/artifacts/work-bundle/INFORMATION-REQUEST-RESPONSE.md)). Inv-Man-Intake still routes PDFs through its own `PdfPrimaryExtractionProvider` for the default pyodide-light path — regex over content streams, explicitly *not production OCR* (`src/inv_man_intake/extraction/providers/pdf_primary.py:20-26`) — and only optionally through `stranske_pdf_extract` Docling (`src/inv_man_intake/extraction/service.py:104-115`, `118-128`). **Missing behavior:** no adapter to the Doc-Lineage extraction library once [Doc-Lineage #3](https://github.com/stranske/Doc-Lineage/issues/3) lands. **Latent fragility** — a fourth PDF extraction surface if left unchanged.

## Scope

Replace the in-repo PDF primary provider path with a thin adapter to the Doc-Lineage extraction library (page pointers + mandatory OCR fallback per Doc-Lineage #3), keeping the existing `ExtractionService` port at `src/inv_man_intake/extraction/service.py:27-35`.

## Non-Goals

- Do NOT block on Doc-Lineage #3 merging — land the adapter behind a feature flag or optional extra until #3 publishes a stable import path; remove `pdf-primary` once #3 is wired.
- Do NOT remove PPTX extraction (`pptx_primary.py`) in this issue.
- Do NOT implement the comparison/lineage engine (Doc-Lineage).
- Scaffold-only completion does NOT count: leaving `build_pyodide_light_service()` on `PdfPrimaryExtractionProvider` as the default while adding an unused adapter module is a failure.

## Tasks

- [ ] Add `src/inv_man_intake/extraction/providers/doc_lineage_adapter.py` wrapping the Doc-Lineage extraction entry point from #3, mapping results to `ExtractedDocumentResult` at `src/inv_man_intake/extraction/providers/base.py:8-13`.
- [ ] Add optional dependency group `extraction-doc-lineage` in `pyproject.toml` pointing at the Doc-Lineage package once #3 exposes `doc_lineage.extract`.
- [ ] Update `build_pyodide_light_service()` at `src/inv_man_intake/extraction/service.py:104-115` to select `DocLineageExtractionProvider` when the extra is installed, falling back to `PdfPrimaryExtractionProvider` only when the extra is absent (document the fallback as temporary).
- [ ] Add `tests/extraction/providers/test_doc_lineage_adapter.py` using a synthetic PDF fixture with expected page pointers.
- [ ] Update `tests/extraction/test_extraction_service.py` to assert the doc-lineage backend is selected when the extra is present.
- [ ] Perform deliberate-break verification (see Acceptance Criteria), capture FAIL output, then revert.

## Acceptance Criteria

- [ ] Named test: `tests/extraction/providers/test_doc_lineage_adapter.py::test_adapter_emits_page_pointers` passes — extracted fields include `SourceLocation.page` values from the Doc-Lineage library, not regex-only guesses.
- [ ] Named test: `tests/extraction/test_extraction_service.py::test_pyodide_light_prefers_doc_lineage_when_installed` passes when the `extraction-doc-lineage` extra is installed.
- [ ] **Deliberate-break gate:** temporarily edit `doc_lineage_adapter.py` to return `page=None` for all fields. `tests/extraction/providers/test_doc_lineage_adapter.py::test_adapter_emits_page_pointers` **must FAIL**. Revert after capturing the failure.

## Implementation Notes

- Reference [Doc-Lineage #3](https://github.com/stranske/Doc-Lineage/issues/3) for the shared extraction contract (page pointers + OCR fallback).
- Existing Docling path via `stranske_pdf_extract` (`service.py:118-128`) should delegate to the same Doc-Lineage library after #3 to avoid two OCR implementations.
- Confirmed-green local reproduction: `python -m pytest tests/extraction/test_pdf_primary_provider.py -q` → passes today on `main`.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agents:formattedIssue has been formatted to AGENT_ISSUE_TEMPLATEenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions