Add optional Docling extraction provider

[stranske]

Source: Issue #713

Closes #713

Automated Status Summary

Scope

The extraction layer is built around a provider Protocol (src/inv_man_intake/extraction/providers/base.py:204 ExtractionProvider; :218 MultiModalExtractionProvider), but current providers are fixture/parser-backed — the README's own throughput line admits the demo runs on synthetic fixtures. The moat commercial peers (Canoe, Accelex) actually sell is proven real-document extraction accuracy. The Protocol seam means a real extractor can be added without touching the scoring/queue/provenance pipeline. Latent opportunity (not a break).

Context for Agent

Related Issues/PRs

• #699

Tasks

• Add src/inv_man_intake/extraction/providers/docling_primary.py implementing ExtractionProvider (and/or MultiModalExtractionProvider) — pyproject.toml optional extra extraction-docling.
• Use a schema-valid structured-output layer (instructor/outlines) for any LLM-assisted field extraction so outputs validate against the ExtractedField contract.
• Add an evaluation harness (DeepEval test or LangSmith dataset runner) measuring field-level extraction accuracy on a small real-document sample, runnable as a non-default CI job.
• Add tests/extraction/providers/test_docling_primary.py::test_docling_provider_conforms_to_protocol (Protocol conformance via isinstance against the @runtime_checkable Protocol; skip cleanly if the optional dep is absent).

Acceptance criteria

• Named test tests/extraction/providers/test_docling_primary.py::test_docling_provider_conforms_to_protocol passes (or skips with a clear reason when the extraction-docling extra is not installed), asserting the provider satisfies ExtractionProvider.
• Deliberate-break gate: remove a required Protocol method from the provider; the conformance test must FAIL (isinstance against the runtime-checkable Protocol returns False); revert → passes.
• The eval harness produces a field-level accuracy number on the real-document sample (captured in the PR), and the core package still imports/installs without the optional extra.

[github-actions]

Workflow state fingerprint for Keepalive Loop Reporter. Do not edit.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 57 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6ad270e5-56a9-45ca-bec2-56dfaba1439b

📥 Commits

Reviewing files that changed from the base of the PR and between 569033b and 703106d.

⛔ Files ignored due to path filters (1)

• requirements.lock is excluded by !**/*.lock, !**/*.lock

📒 Files selected for processing (6)

• pyproject.toml
• src/inv_man_intake/extraction/evaluation/__init__.py
• src/inv_man_intake/extraction/evaluation/field_accuracy.py
• src/inv_man_intake/extraction/providers/docling_primary.py
• tests/extraction/providers/test_docling_primary.py
• tests/extraction/test_field_accuracy_evaluation.py

📝 Walkthrough

Walkthrough

This PR adds an optional Docling-backed extraction provider, a field-accuracy evaluation harness, and tests for both paths. It also introduces the extraction-docling optional dependency and a package docstring for the evaluation module.

Changes

Docling provider and evaluation harness

Layer / File(s)	Summary
Field accuracy evaluation harness src/inv_man_intake/extraction/evaluation/__init__.py, src/inv_man_intake/extraction/evaluation/field_accuracy.py, tests/extraction/test_field_accuracy_evaluation.py	Adds field expectation, sample, and report dataclasses plus evaluate_field_accuracy, which normalizes expected and extracted fields and aggregates accuracy metrics; includes a test against PrimaryRegexExtractionProvider.
Docling provider implementation pyproject.toml, src/inv_man_intake/extraction/providers/docling_primary.py, tests/extraction/providers/test_docling_primary.py	Adds the extraction-docling extra, implements DoclingPrimaryExtractionProvider with Docling conversion and extraction paths, and adds tests for protocol conformance, optional-dependency handling, and field mapping.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant Docling as DoclingPrimaryExtractionProvider
  participant Converter as DocumentConverter
  participant Regex as PrimaryRegexExtractionProvider
  Caller->>Docling: extract(source_doc_id, content)
  Docling->>Converter: convert(temp file from content)
  Converter-->>Docling: converted document
  Docling->>Docling: export document to text
  Docling->>Regex: extract(exported text as bytes)
  Regex-->>Docling: extracted fields
  Docling-->>Caller: ExtractedDocumentResult(method="docling-primary")

Loading

Possibly related issues

• Migrate Inv-Man-Intake to CONSUME stranske-pdf-extract (Phase 2 — Docling via shared lib, satisfies IMI #713) Workflows#2713: Adds a Docling-backed extraction provider and extraction evaluation harness, matching the provider migration and eval harness work in this PR.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	[#713] The provider and eval harness are added, but the summary shows no schema-valid structured-output layer or non-default CI eval job.	Add the structured-output layer for any LLM-assisted fields and wire the field-accuracy harness into a non-default CI job.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Out of Scope Changes check	✅ Passed	The changes stay within the Docling provider and extraction evaluation scope, with no unrelated features introduced.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly captures the main addition of an optional Docling-backed extraction provider.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/issue-713-docling-provider

---

_{Comment @coderabbitai help to get the list of available commands.}

[stranske-keepalive]

🤖 Keepalive Loop Status

PR #737 | Agent: Codex | Iteration 0/12

Current State

Metric	Value
Iteration progress	[----------] 0/12
Action	run (agent-run-skipped)
Gate	success
Tasks	0/7 complete
Timeout	45 min (default)
Timeout usage	3m elapsed (7%, 42m remaining)
Keepalive	✅ enabled
Autofix	❌ disabled

Last Codex Run

Result	Value
Status	⏭️ Skipped
Reason	agent-run-skipped

To retry:

• Add the agent:retry label, OR
• Wait for conditions to resolve (e.g., Gate success, labels present)

🔍 Failure Classification

| Error type | infrastructure |
| Error category | transient |
| Suggested recovery | Capture logs and context; retry once and escalate if the issue persists. |

[stranske-keepalive]

Keepalive Work Log (click to expand)

#	Time (UTC)	Agent	Action	Result	Files	Tasks	Progress	Commit	Gate
0	2026-07-04 19:10:13	Codex	wait (gate-cancelled-transient-transient)	skipped	—	0	0/8	—	cancelled
0	2026-07-04 19:12:21	Codex	run (agent-run-failed)	failure	4 file(s)	0	0/8	—	cancelled
0	2026-07-04 19:14:40	Codex	run (agent-run-failed)	failure	4 file(s)	0	0/8	—	cancelled
0	2026-07-04 19:15:21	Codex	skip (needs-human)	skipped	—	0	0/8	—	failure
0	2026-07-04 19:17:39	Codex	skip (needs-human)	skipped	—	+1	0/7	—	failure
0	2026-07-04 19:31:45	Codex	skip (needs-human)	skipped	—	0	0/7	—	—
0	2026-07-04 19:33:11	Codex	skip (needs-human)	skipped	—	0	0/7	—	failure
0	2026-07-04 19:49:33	Codex	skip (needs-human)	skipped	—	0	0/7	—	failure
0	2026-07-07 02:34:40	Codex	run (agent-run-failed-repeat)	failure	5 file(s)	0	0/7	—	success
0	2026-07-07 02:35:27	Codex	skip (needs-human)	skipped	—	0	0/7	—	cancelled
0	2026-07-07 02:36:13	Codex	skip (needs-human)	skipped	—	0	0/7	—	cancelled
0	2026-07-07 02:37:45	Codex	skip (needs-human)	skipped	—	0	0/7	—	success
0	2026-07-07 03:24:42	Codex	wait (gate-pending-transient)	skipped	—	0	0/7	—	—
0	2026-07-07 03:26:53	Codex	run (agent-run-skipped)	skipped	—	0	0/7	—	success
0	2026-07-08 08:28:27	Codex	run (agent-run-skipped)	skipped	—	0	0/7	—	success

[github-actions]

Workflow state fingerprint for Agents Gate Followups. Do not edit.

[github-actions]

Runner dispatch state for codex on PR #737. Do not edit.

[stranske]

Autofix attempts exhausted for this head.
Attempts: 9 / 3

Latest Gate summary:

Gate run: https://github.com/stranske/Inv-Man-Intake/actions/runs/28716704554
Conclusion: cancelled
PR: #737
Head SHA: cc71ab5ca5227781a65858e73e013e43d9a3fdc2
Autofix attempts for this head: 9 / 3
Fix scope: src/, tests/, tools/, scripts/, agents/, templates/, .github/
Failing jobs:
- classify changed paths (cancelled)

Please investigate manually.

[github-actions]

⚠️ Codex autofix run failed

Field	Value
Exit Code	1
Error Category	unknown
Error Type	codex
Run	View logs

🔧 Suggested Recovery

Capture logs and context; retry once and escalate if the issue persists.

📝 What to do

1. Check the workflow logs for detailed error output
2. If this is a configuration issue, update the relevant settings
3. If the error persists, consider adding the needs-human label for manual review
4. Re-run the workflow once the issue is resolved

Output summary

You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 7th, 2026 2:01 AM.

[stranske]

Autofix attempts exhausted for this head.
Attempts: 10 / 3

Latest Gate summary:

Gate run: https://github.com/stranske/Inv-Man-Intake/actions/runs/28716704806
Conclusion: cancelled
PR: #737
Head SHA: cc71ab5ca5227781a65858e73e013e43d9a3fdc2
Autofix attempts for this head: 10 / 3
Fix scope: src/, tests/, tools/, scripts/, agents/, templates/, .github/
Failing jobs:
- Python CI / lint-format (failure)
  - steps: Finalize format check (failure)
- Python CI / python 3.12 (cancelled)
  - steps: Validate test dependencies (cancelled)
- Python CI / python 3.13 (cancelled)
  - steps: Restore CI artifact cache (cancelled)

Please investigate manually.

[Copilot]

Pull request overview

Adds an optional Docling-backed extraction provider that conforms to the existing provider Protocols, and introduces a small field-level accuracy evaluation harness to measure extraction correctness on real-document sample bytes—while keeping Docling out of core installs via an optional extra.

Changes:

• Added DoclingPrimaryExtractionProvider implementing ExtractionProvider/MultiModalExtractionProvider, performing Docling conversion then mapping into existing canonical field extraction.
• Introduced a field-level accuracy evaluation harness (evaluate_field_accuracy) and a test exercising it on sample bytes.
• Added an extraction-docling optional extra for installing Docling.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/extraction/test_field_accuracy_evaluation.py	Adds a unit test validating the new field-accuracy evaluation harness on sample bytes.
tests/extraction/providers/test_docling_primary.py	Adds Protocol conformance and behavior tests for the optional Docling provider.
src/inv_man_intake/extraction/providers/docling_primary.py	Introduces the optional Docling-backed provider and conversion/normalization logic.
src/inv_man_intake/extraction/evaluation/field_accuracy.py	Implements the field-level accuracy evaluation harness and report model.
src/inv_man_intake/extraction/evaluation/init.py	Creates the evaluation package namespace.
pyproject.toml	Adds extraction-docling optional extra dependency definition.

[stranske]

Runner dispatch state for autofix on PR #737. Do not edit.

[stranske]

Autofix attempts exhausted for this head.
Attempts: 11 / 3

Latest Gate summary:

Gate run: https://github.com/stranske/Inv-Man-Intake/actions/runs/28716786561
Conclusion: failure
PR: #737
Head SHA: cc71ab5ca5227781a65858e73e013e43d9a3fdc2
Autofix attempts for this head: 11 / 3
Fix scope: src/, tests/, tools/, scripts/, agents/, templates/, .github/
Failing jobs:
- Python CI / lint-format (failure)
  - steps: Finalize format check (failure)
- Python CI / python 3.12 (failure)
  - steps: Finalize check results (failure)
- Python CI / python 3.13 (failure)
  - steps: Finalize check results (failure)
- gate-summary (failure)
  - steps: Enforce Gate success (failure)

Please investigate manually.

[stranske]

Closer review-fix update for #737:

• Rebased the live branch onto current main.
• Capped the optional docling extra to <3.0.0 and regenerated requirements.lock with --extra=extraction-docling, satisfying dependency-lock alignment.
• Made the optional-Docling skip-path test deterministic: absent Docling now asserts MissingDoclingDependencyError; installed Docling skips real conversion until a known-good integration fixture exists.
• Removed production module-level protocol asserts; protocol conformance remains covered in tests.
• Hardened evaluate_field_accuracy() to continue after per-sample provider errors, report duplicate normalized keys as mismatches, and added missing/mismatched branch coverage.

Validation:

PYTHONPYCACHEPREFIX=/tmp/pycache-imi-737-reviewfix python -m pytest tests/extraction/providers/test_docling_primary.py tests/extraction/test_field_accuracy_evaluation.py tests/test_dependency_version_alignment.py -q --no-cov
python -m ruff check src/inv_man_intake/extraction/evaluation/field_accuracy.py src/inv_man_intake/extraction/providers/docling_primary.py tests/extraction/providers/test_docling_primary.py tests/extraction/test_field_accuracy_evaluation.py tests/test_dependency_version_alignment.py
python -m black --check src/inv_man_intake/extraction/evaluation/field_accuracy.py src/inv_man_intake/extraction/providers/docling_primary.py tests/extraction/providers/test_docling_primary.py tests/extraction/test_field_accuracy_evaluation.py
python -m mypy src/inv_man_intake/extraction/evaluation/field_accuracy.py src/inv_man_intake/extraction/providers/docling_primary.py
git diff --check

All passed locally. Pushed 703106d; fresh CI/review checks are expected to rerun.

[github-actions]

Provider Comparison Report

Provider Summary

Provider	Model	Verdict	Confidence	Summary
openai	gpt-5.4	CONCERNS	88%	This merge partially satisfies the requested work: it adds an optional extraction-docling extra, a new Docling provider module, and the requested protocol-conformance test file, plus a field-accu...
anthropic	claude-sonnet-4-6	CONCERNS	72%	The PR delivers the core structural requirements: the docling_primary.py provider file, the optional extraction-docling extra in pyproject.toml, the conformance test file, and an evaluation harness...

📋 Full Provider Details (click to expand)

openai

• Model: gpt-5.4
• Verdict: CONCERNS
• Confidence: 88%
• Scores:
  • Correctness: 7.0/10
  • Completeness: 5.0/10
  • Quality: 7.0/10
  • Testing: 7.0/10
  • Risks: 6.0/10
• Summary: This merge partially satisfies the requested work: it adds an optional extraction-docling extra, a new Docling provider module, and the requested protocol-conformance test file, plus a field-accuracy evaluation utility with tests. Code quality appears reasonable and the provider/evaluation additions are directionally correct. However, the acceptance criteria are only partially met from the code shown: there is no clear implementation of schema-validated LLM structured output, no convincing evidence of a real-document Docling evaluation harness producing a captured field-level accuracy number on sample documents, and no explicit deliberate-break gate beyond the existence of a conformance test. Overall, this is close but incomplete relative to the documented requirements.
• Concerns:
  • The PR adds src/inv_man_intake/extraction/providers/docling_primary.py and the optional extraction-docling extra, but the documented requirement to use a schema-valid structured-output layer such as instructor/outlines for any LLM-assisted extraction is not evidenced by the changed files.
  • An evaluation harness was added (src/inv_man_intake/extraction/evaluation/field_accuracy.py with tests), but from the changed files it appears to be a generic field-accuracy utility/test rather than a runnable real-document Docling evaluation on a sample dataset with a captured field-level accuracy result as required.
  • The named protocol conformance test was added, but the acceptance criterion explicitly calls for isinstance against the @runtime_checkable Protocol and a deliberate-break gate. The test likely covers conformance, but there is no code evidence in the PR of the deliberate-break gate being implemented or documented beyond ordinary positive-path testing.
  • The lockfile now includes the docling dependency set globally. While the core package likely still imports without the extra because the provider is isolated, this PR does not clearly demonstrate optional-install safety beyond packaging metadata and a skip-capable test.
  • Unrelated additions (src/inv_man_intake/audit/lineage.py and tests/test_lineage_packet.py) expand scope and make it harder to verify that the PR is tightly focused on the stated extraction-provider acceptance criteria.

anthropic

• Model: claude-sonnet-4-6
• Verdict: CONCERNS
• Confidence: 72%
• Scores:
  • Correctness: 7.0/10
  • Completeness: 5.0/10
  • Quality: 7.0/10
  • Testing: 6.0/10
  • Risks: 7.0/10
• Summary: The PR delivers the core structural requirements: the docling_primary.py provider file, the optional extraction-docling extra in pyproject.toml, the conformance test file, and an evaluation harness module. CI passes cleanly. However, two acceptance criteria have significant gaps: (1) AC ci(deps): bump actions/download-artifact from 7 to 8 #3 requires a field-level accuracy number on a real-document sample captured in the PR — the test file is only 53 lines and almost certainly uses mocks rather than real documents, with no accuracy numbers documented; (2) AC chore: sync workflow templates #2's deliberate-break gate is a process demonstration requirement with no evidence it was exercised. Additionally, the PR includes substantial out-of-scope additions (lineage audit module, 225-line lineage test) not mentioned in the acceptance criteria, and it's unclear whether the structured-output layer (instructor/outlines) requirement was implemented.
• Concerns:
  • AC ci(deps): bump actions/download-artifact from 7 to 8 #3 requires the eval harness to produce a field-level accuracy number on a real-document sample 'captured in the PR' — the diff shows only a 53-line test file that almost certainly uses synthetic/mock data, with no evidence of real documents or captured accuracy metrics in the PR body.
  • AC chore: sync workflow templates #2 (deliberate-break gate) requires demonstrating that removing a required Protocol method causes isinstance to return False — there is no evidence in the diff or PR that this gate was explicitly exercised and documented; the behavior is inherent to runtime-checkable Protocols but the requirement asks for it to be demonstrated.
  • src/inv_man_intake/audit/lineage.py (146 lines) and tests/test_lineage_packet.py (225 lines) appear out of scope for this PR's stated requirements (Docling provider + eval harness), suggesting scope creep that was not reviewed against the acceptance criteria.
  • The schema-valid structured-output layer (instructor/outlines) requirement is not evidenced in the diff summary — it's unclear whether docling_primary.py uses instructor or outlines for LLM-assisted field extraction.
  • requirements.lock was regenerated to include extraction-docling dependencies, which is correct, but this significantly expands the lock file (326 additions) and may affect reproducibility for users not using the optional extra.
  • The eval harness being added under src/inv_man_intake/extraction/evaluation/ (production code path) rather than a dedicated eval/benchmarks directory may conflate production and evaluation concerns.

Agreement

• Verdict: CONCERNS (all providers)
• Correctness: scores within 1 point (avg 7.0/10, range 7.0-7.0)
• Completeness: scores within 1 point (avg 5.0/10, range 5.0-5.0)
• Quality: scores within 1 point (avg 7.0/10, range 7.0-7.0)
• Testing: scores within 1 point (avg 6.5/10, range 6.0-7.0)
• Risks: scores within 1 point (avg 6.5/10, range 6.0-7.0)

Disagreement

No major disagreements detected.

Unique Insights

• openai: The PR adds src/inv_man_intake/extraction/providers/docling_primary.py and the optional extraction-docling extra, but the documented requirement to use a schema-valid structured-output layer such as instructor/outlines for any LLM-assisted extraction is not evidenced by the changed files.; An evaluation harness was added (src/inv_man_intake/extraction/evaluation/field_accuracy.py with tests), but from the changed files it appears to be a generic field-accuracy utility/test rather than a runnable real-document Docling evaluation on a sample dataset with a captured field-level accuracy result as required.; The named protocol conformance test was added, but the acceptance criterion explicitly calls for isinstance against the @runtime_checkable Protocol and a deliberate-break gate. The test likely covers conformance, but there is no code evidence in the PR of the deliberate-break gate being implemented or documented beyond ordinary positive-path testing.; The lockfile now includes the docling dependency set globally. While the core package likely still imports without the extra because the provider is isolated, this PR does not clearly demonstrate optional-install safety beyond packaging metadata and a skip-capable test.; Unrelated additions (src/inv_man_intake/audit/lineage.py and tests/test_lineage_packet.py) expand scope and make it harder to verify that the PR is tightly focused on the stated extraction-provider acceptance criteria.
• anthropic: AC ci(deps): bump actions/download-artifact from 7 to 8 #3 requires the eval harness to produce a field-level accuracy number on a real-document sample 'captured in the PR' — the diff shows only a 53-line test file that almost certainly uses synthetic/mock data, with no evidence of real documents or captured accuracy metrics in the PR body.; AC chore: sync workflow templates #2 (deliberate-break gate) requires demonstrating that removing a required Protocol method causes isinstance to return False — there is no evidence in the diff or PR that this gate was explicitly exercised and documented; the behavior is inherent to runtime-checkable Protocols but the requirement asks for it to be demonstrated.; src/inv_man_intake/audit/lineage.py (146 lines) and tests/test_lineage_packet.py (225 lines) appear out of scope for this PR's stated requirements (Docling provider + eval harness), suggesting scope creep that was not reviewed against the acceptance criteria.; The schema-valid structured-output layer (instructor/outlines) requirement is not evidenced in the diff summary — it's unclear whether docling_primary.py uses instructor or outlines for LLM-assisted field extraction.; requirements.lock was regenerated to include extraction-docling dependencies, which is correct, but this significantly expands the lock file (326 additions) and may affect reproducibility for users not using the optional extra.; The eval harness being added under src/inv_man_intake/extraction/evaluation/ (production code path) rather than a dedicated eval/benchmarks directory may conflate production and evaluation concerns.

🔍 LangSmith Traces

• openai
• anthropic

[github-actions]

Workflow state fingerprint for Agents Verifier. Do not edit.

[stranske]

📋 Follow-up issue created: #765

Verification concerns have been analyzed and structured into a follow-up issue.

Next steps:

1. Review the generated issue
2. Auto-pilot will continue preparing a new PR

Or work on it manually - the choice is yours!