feat(parsers): pluggable document parsers — MinerU / Mistral / VLM (closes #77) by KylinMountain · Pull Request #81 · VectifyAI/OpenKB

KylinMountain · 2026-06-01T06:41:15Z

Summary

Adds a pluggable parser abstraction so the file → Markdown step can be routed
through higher-accuracy online/self-hosted parsers, addressing #77 (markitdown gives
breadth but not precision on complex docs like papers). The local parser remains the
zero-config default with no new required dependencies.

New openkb/parsers/ package: Parser ABC + ParseResult, a get_parser registry, and adapters:
- local (default) — existing pymupdf/markitdown behavior, refactored behind the abstraction (behavior-preserving).
- mineru — MinerU over HTTP: hosted cloud (submit→poll→zip) or self-hosted server.
- mistral — Mistral OCR via the official mistralai SDK (sync).
- vlm — any vision LLM via the existing litellm dep (Gemini, GPT-4o, Claude, …).
Selection via .openkb/config.yamlparser: + a parsers: per-provider block; per-run CLI override openkb add --parser <name> (validated by click.Choice).
Optional deps via pip extras: openkb[mistral], openkb[mineru], openkb[parsers]; vlm needs none. Adapters lazy-import and raise actionable errors when an extra/key is missing.
Image handling unified via localize_images (writes basenames, rewrites links; reuses extract_base64_images).

Scope / non-goals

PageIndex is untouched. Long PDFs (≥ pageindex_threshold, default 20) still go to PageIndex — the parser setting governs shorter docs and non-PDF formats.
Image understanding (Feature: Optional image understanding / vision for inline and referenced images #74) not included, but the reusable parsers/vlm_client.py is factored so Feature: Optional image understanding / vision for inline and referenced images #74 can plug in later.

Test plan

Unit tests for every adapter + registry + localize_images + the converter refactor (HTTP/SDK fully mocked — no network). MinerU cloud poll+download flow covered.
Full suite green (616 passed; the only repo-wide failures are pre-existing trafilatura url_ingest tests, unrelated).
Real-PDF smoke test of the local path (examples/docs/attention-is-all-you-need.pdf → Markdown + image extraction).
Real online-parser e2e (MinerU/Mistral/Gemini) — requires API keys; not run in CI.

Review fixes included

Hardened from review: MinerU poll-loop guards (non-positive interval, empty result), anchored image-link rewrite, path-traversal-safe image filenames, VLM global-model fallback warning, narrowed LLM_API_KEY propagation (no longer defeats the Mistral key guard), and single-source-of-truth parser dispatch.

…bitrary filenames (#77)

)

…tighten tests (#77)

)

…edundant localize for local parser (#77)

…h/VALID_PARSERS (#77)

+"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
+from __future__ import annotations
+
+from pathlib import Path


…localize_images (#77)

…ction; drop redundant image rewrite (#77)

…de (#77)

…st hygiene (#77)

+
+
+def test_result_from_zip_does_not_rewrite_links(tmp_path):
+ import io, zipfile


+
+
+def test_cloud_empty_extract_result_then_done(monkeypatch, tmp_path):
+ import io, sys, types, zipfile


+
+
+def test_full_md_basename_preferred_over_endswith(tmp_path):
+ import io, zipfile


+
+
+def test_image_basename_collision_warns(tmp_path, caplog):
+ import io, zipfile, logging as _logging


+ zf.writestr("images/fig.png", b"A")
+ zf.writestr("sub/fig.png", b"B")
+ with caplog.at_level(_logging.WARNING):
+ result = _result_from_zip(buf.getvalue())


KylinMountain · 2026-06-01T06:41:15Z

Summary

Adds a pluggable parser abstraction so the file → Markdown step can be routed
through higher-accuracy online/self-hosted parsers, addressing #77 (markitdown gives
breadth but not precision on complex docs like papers). The local parser remains the
zero-config default with no new required dependencies.

New openkb/parsers/ package: Parser ABC + ParseResult, a get_parser registry, and adapters:
- local (default) — existing pymupdf/markitdown behavior, refactored behind the abstraction (behavior-preserving).
- mineru — MinerU over HTTP: hosted cloud (submit→poll→zip) or self-hosted server.
- mistral — Mistral OCR via the official mistralai SDK (sync).
- vlm — any vision LLM via the existing litellm dep (Gemini, GPT-4o, Claude, …).
Selection via .openkb/config.yamlparser: + a parsers: per-provider block; per-run CLI override openkb add --parser <name> (validated by click.Choice).
Optional deps via pip extras: openkb[mistral], openkb[mineru], openkb[parsers]; vlm needs none. Adapters lazy-import and raise actionable errors when an extra/key is missing.
Image handling unified via localize_images (writes basenames, rewrites links; reuses extract_base64_images).

Scope / non-goals

PageIndex is untouched. Long PDFs (≥ pageindex_threshold, default 20) still go to PageIndex — the parser setting governs shorter docs and non-PDF formats.
Image understanding (Feature: Optional image understanding / vision for inline and referenced images #74) not included, but the reusable parsers/vlm_client.py is factored so Feature: Optional image understanding / vision for inline and referenced images #74 can plug in later.

Test plan

Unit tests for every adapter + registry + localize_images + the converter refactor (HTTP/SDK fully mocked — no network). MinerU cloud poll+download flow covered.
Full suite green (616 passed; the only repo-wide failures are pre-existing trafilatura url_ingest tests, unrelated).
Real-PDF smoke test of the local path (examples/docs/attention-is-all-you-need.pdf → Markdown + image extraction).
Real online-parser e2e (MinerU/Mistral/Gemini) — requires API keys; not run in CI.

Review fixes included

Hardened from review: MinerU poll-loop guards (non-positive interval, empty result), anchored image-link rewrite, path-traversal-safe image filenames, VLM global-model fallback warning, narrowed LLM_API_KEY propagation (no longer defeats the Mistral key guard), and single-source-of-truth parser dispatch.

…bitrary filenames (#77)

)

…tighten tests (#77)

)

…edundant localize for local parser (#77)

…h/VALID_PARSERS (#77)

+"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
+from __future__ import annotations
+
+from pathlib import Path


…localize_images (#77)

…ction; drop redundant image rewrite (#77)

…de (#77)

…st hygiene (#77)

+
+
+def test_result_from_zip_does_not_rewrite_links(tmp_path):
+ import io, zipfile


+
+
+def test_cloud_empty_extract_result_then_done(monkeypatch, tmp_path):
+ import io, sys, types, zipfile


+
+
+def test_full_md_basename_preferred_over_endswith(tmp_path):
+ import io, zipfile


+
+
+def test_image_basename_collision_warns(tmp_path, caplog):
+ import io, zipfile, logging as _logging


+ zf.writestr("images/fig.png", b"A")
+ zf.writestr("sub/fig.png", b"B")
+ with caplog.at_level(_logging.WARNING):
+ result = _result_from_zip(buf.getvalue())


KylinMountain · 2026-06-01T06:41:15Z

Summary

Adds a pluggable parser abstraction so the file → Markdown step can be routed
through higher-accuracy online/self-hosted parsers, addressing #77 (markitdown gives
breadth but not precision on complex docs like papers). The local parser remains the
zero-config default with no new required dependencies.

New openkb/parsers/ package: Parser ABC + ParseResult, a get_parser registry, and adapters:
- local (default) — existing pymupdf/markitdown behavior, refactored behind the abstraction (behavior-preserving).
- mineru — MinerU over HTTP: hosted cloud (submit→poll→zip) or self-hosted server.
- mistral — Mistral OCR via the official mistralai SDK (sync).
- vlm — any vision LLM via the existing litellm dep (Gemini, GPT-4o, Claude, …).
Selection via .openkb/config.yamlparser: + a parsers: per-provider block; per-run CLI override openkb add --parser <name> (validated by click.Choice).
Optional deps via pip extras: openkb[mistral], openkb[mineru], openkb[parsers]; vlm needs none. Adapters lazy-import and raise actionable errors when an extra/key is missing.
Image handling unified via localize_images (writes basenames, rewrites links; reuses extract_base64_images).

Scope / non-goals

PageIndex is untouched. Long PDFs (≥ pageindex_threshold, default 20) still go to PageIndex — the parser setting governs shorter docs and non-PDF formats.
Image understanding (Feature: Optional image understanding / vision for inline and referenced images #74) not included, but the reusable parsers/vlm_client.py is factored so Feature: Optional image understanding / vision for inline and referenced images #74 can plug in later.

Test plan

Unit tests for every adapter + registry + localize_images + the converter refactor (HTTP/SDK fully mocked — no network). MinerU cloud poll+download flow covered.
Full suite green (616 passed; the only repo-wide failures are pre-existing trafilatura url_ingest tests, unrelated).
Real-PDF smoke test of the local path (examples/docs/attention-is-all-you-need.pdf → Markdown + image extraction).
Real online-parser e2e (MinerU/Mistral/Gemini) — requires API keys; not run in CI.

Review fixes included

Hardened from review: MinerU poll-loop guards (non-positive interval, empty result), anchored image-link rewrite, path-traversal-safe image filenames, VLM global-model fallback warning, narrowed LLM_API_KEY propagation (no longer defeats the Mistral key guard), and single-source-of-truth parser dispatch.

…bitrary filenames (#77)

)

…tighten tests (#77)

)

…edundant localize for local parser (#77)

…h/VALID_PARSERS (#77)

+"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
+from __future__ import annotations
+
+from pathlib import Path


…localize_images (#77)

…ction; drop redundant image rewrite (#77)

…de (#77)

…st hygiene (#77)

+
+
+def test_result_from_zip_does_not_rewrite_links(tmp_path):
+ import io, zipfile


+
+
+def test_cloud_empty_extract_result_then_done(monkeypatch, tmp_path):
+ import io, sys, types, zipfile


+
+
+def test_full_md_basename_preferred_over_endswith(tmp_path):
+ import io, zipfile


+
+
+def test_image_basename_collision_warns(tmp_path, caplog):
+ import io, zipfile, logging as _logging


+ zf.writestr("images/fig.png", b"A")
+ zf.writestr("sub/fig.png", b"B")
+ with caplog.at_level(_logging.WARNING):
+ result = _result_from_zip(buf.getvalue())


KylinMountain · 2026-06-01T06:41:15Z

Summary

Adds a pluggable parser abstraction so the file → Markdown step can be routed
through higher-accuracy online/self-hosted parsers, addressing #77 (markitdown gives
breadth but not precision on complex docs like papers). The local parser remains the
zero-config default with no new required dependencies.

New openkb/parsers/ package: Parser ABC + ParseResult, a get_parser registry, and adapters:
- local (default) — existing pymupdf/markitdown behavior, refactored behind the abstraction (behavior-preserving).
- mineru — MinerU over HTTP: hosted cloud (submit→poll→zip) or self-hosted server.
- mistral — Mistral OCR via the official mistralai SDK (sync).
- vlm — any vision LLM via the existing litellm dep (Gemini, GPT-4o, Claude, …).
Selection via .openkb/config.yamlparser: + a parsers: per-provider block; per-run CLI override openkb add --parser <name> (validated by click.Choice).
Optional deps via pip extras: openkb[mistral], openkb[mineru], openkb[parsers]; vlm needs none. Adapters lazy-import and raise actionable errors when an extra/key is missing.
Image handling unified via localize_images (writes basenames, rewrites links; reuses extract_base64_images).

Scope / non-goals

PageIndex is untouched. Long PDFs (≥ pageindex_threshold, default 20) still go to PageIndex — the parser setting governs shorter docs and non-PDF formats.
Image understanding (Feature: Optional image understanding / vision for inline and referenced images #74) not included, but the reusable parsers/vlm_client.py is factored so Feature: Optional image understanding / vision for inline and referenced images #74 can plug in later.

Test plan

Unit tests for every adapter + registry + localize_images + the converter refactor (HTTP/SDK fully mocked — no network). MinerU cloud poll+download flow covered.
Full suite green (616 passed; the only repo-wide failures are pre-existing trafilatura url_ingest tests, unrelated).
Real-PDF smoke test of the local path (examples/docs/attention-is-all-you-need.pdf → Markdown + image extraction).
Real online-parser e2e (MinerU/Mistral/Gemini) — requires API keys; not run in CI.

Review fixes included

Hardened from review: MinerU poll-loop guards (non-positive interval, empty result), anchored image-link rewrite, path-traversal-safe image filenames, VLM global-model fallback warning, narrowed LLM_API_KEY propagation (no longer defeats the Mistral key guard), and single-source-of-truth parser dispatch.

…bitrary filenames (#77)

)

…tighten tests (#77)

)

…edundant localize for local parser (#77)

…h/VALID_PARSERS (#77)

+"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
+from __future__ import annotations
+
+from pathlib import Path


…localize_images (#77)

…ction; drop redundant image rewrite (#77)

…de (#77)

…st hygiene (#77)

+
+
+def test_result_from_zip_does_not_rewrite_links(tmp_path):
+ import io, zipfile


+
+
+def test_cloud_empty_extract_result_then_done(monkeypatch, tmp_path):
+ import io, sys, types, zipfile


+
+
+def test_full_md_basename_preferred_over_endswith(tmp_path):
+ import io, zipfile


+
+
+def test_image_basename_collision_warns(tmp_path, caplog):
+ import io, zipfile, logging as _logging


+ zf.writestr("images/fig.png", b"A")
+ zf.writestr("sub/fig.png", b"B")
+ with caplog.at_level(_logging.WARNING):
+ result = _result_from_zip(buf.getvalue())


KylinMountain · 2026-06-01T06:41:15Z

Summary

Adds a pluggable parser abstraction so the file → Markdown step can be routed
through higher-accuracy online/self-hosted parsers, addressing #77 (markitdown gives
breadth but not precision on complex docs like papers). The local parser remains the
zero-config default with no new required dependencies.

New openkb/parsers/ package: Parser ABC + ParseResult, a get_parser registry, and adapters:
- local (default) — existing pymupdf/markitdown behavior, refactored behind the abstraction (behavior-preserving).
- mineru — MinerU over HTTP: hosted cloud (submit→poll→zip) or self-hosted server.
- mistral — Mistral OCR via the official mistralai SDK (sync).
- vlm — any vision LLM via the existing litellm dep (Gemini, GPT-4o, Claude, …).
Selection via .openkb/config.yamlparser: + a parsers: per-provider block; per-run CLI override openkb add --parser <name> (validated by click.Choice).
Optional deps via pip extras: openkb[mistral], openkb[mineru], openkb[parsers]; vlm needs none. Adapters lazy-import and raise actionable errors when an extra/key is missing.
Image handling unified via localize_images (writes basenames, rewrites links; reuses extract_base64_images).

Scope / non-goals

PageIndex is untouched. Long PDFs (≥ pageindex_threshold, default 20) still go to PageIndex — the parser setting governs shorter docs and non-PDF formats.
Image understanding (Feature: Optional image understanding / vision for inline and referenced images #74) not included, but the reusable parsers/vlm_client.py is factored so Feature: Optional image understanding / vision for inline and referenced images #74 can plug in later.

Test plan

Unit tests for every adapter + registry + localize_images + the converter refactor (HTTP/SDK fully mocked — no network). MinerU cloud poll+download flow covered.
Full suite green (616 passed; the only repo-wide failures are pre-existing trafilatura url_ingest tests, unrelated).
Real-PDF smoke test of the local path (examples/docs/attention-is-all-you-need.pdf → Markdown + image extraction).
Real online-parser e2e (MinerU/Mistral/Gemini) — requires API keys; not run in CI.

Review fixes included

Hardened from review: MinerU poll-loop guards (non-positive interval, empty result), anchored image-link rewrite, path-traversal-safe image filenames, VLM global-model fallback warning, narrowed LLM_API_KEY propagation (no longer defeats the Mistral key guard), and single-source-of-truth parser dispatch.

…bitrary filenames (#77)

)

…tighten tests (#77)

)

…edundant localize for local parser (#77)

…h/VALID_PARSERS (#77)

+"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
+from __future__ import annotations
+
+from pathlib import Path


…localize_images (#77)

…ction; drop redundant image rewrite (#77)

…de (#77)

…st hygiene (#77)

+
+
+def test_result_from_zip_does_not_rewrite_links(tmp_path):
+ import io, zipfile


+
+
+def test_cloud_empty_extract_result_then_done(monkeypatch, tmp_path):
+ import io, sys, types, zipfile


+
+
+def test_full_md_basename_preferred_over_endswith(tmp_path):
+ import io, zipfile


+
+
+def test_image_basename_collision_warns(tmp_path, caplog):
+ import io, zipfile, logging as _logging


+ zf.writestr("images/fig.png", b"A")
+ zf.writestr("sub/fig.png", b"B")
+ with caplog.at_level(_logging.WARNING):
+ result = _result_from_zip(buf.getvalue())


KylinMountain · 2026-06-01T06:41:15Z

Summary

Adds a pluggable parser abstraction so the file → Markdown step can be routed
through higher-accuracy online/self-hosted parsers, addressing #77 (markitdown gives
breadth but not precision on complex docs like papers). The local parser remains the
zero-config default with no new required dependencies.

New openkb/parsers/ package: Parser ABC + ParseResult, a get_parser registry, and adapters:
- local (default) — existing pymupdf/markitdown behavior, refactored behind the abstraction (behavior-preserving).
- mineru — MinerU over HTTP: hosted cloud (submit→poll→zip) or self-hosted server.
- mistral — Mistral OCR via the official mistralai SDK (sync).
- vlm — any vision LLM via the existing litellm dep (Gemini, GPT-4o, Claude, …).
Selection via .openkb/config.yamlparser: + a parsers: per-provider block; per-run CLI override openkb add --parser <name> (validated by click.Choice).
Optional deps via pip extras: openkb[mistral], openkb[mineru], openkb[parsers]; vlm needs none. Adapters lazy-import and raise actionable errors when an extra/key is missing.
Image handling unified via localize_images (writes basenames, rewrites links; reuses extract_base64_images).

Scope / non-goals

PageIndex is untouched. Long PDFs (≥ pageindex_threshold, default 20) still go to PageIndex — the parser setting governs shorter docs and non-PDF formats.
Image understanding (Feature: Optional image understanding / vision for inline and referenced images #74) not included, but the reusable parsers/vlm_client.py is factored so Feature: Optional image understanding / vision for inline and referenced images #74 can plug in later.

Test plan

Unit tests for every adapter + registry + localize_images + the converter refactor (HTTP/SDK fully mocked — no network). MinerU cloud poll+download flow covered.
Full suite green (616 passed; the only repo-wide failures are pre-existing trafilatura url_ingest tests, unrelated).
Real-PDF smoke test of the local path (examples/docs/attention-is-all-you-need.pdf → Markdown + image extraction).
Real online-parser e2e (MinerU/Mistral/Gemini) — requires API keys; not run in CI.

Review fixes included

Hardened from review: MinerU poll-loop guards (non-positive interval, empty result), anchored image-link rewrite, path-traversal-safe image filenames, VLM global-model fallback warning, narrowed LLM_API_KEY propagation (no longer defeats the Mistral key guard), and single-source-of-truth parser dispatch.

…bitrary filenames (#77)

)

…tighten tests (#77)

)

…edundant localize for local parser (#77)

…h/VALID_PARSERS (#77)

+"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
+from __future__ import annotations
+
+from pathlib import Path


…localize_images (#77)

…ction; drop redundant image rewrite (#77)

…de (#77)

…st hygiene (#77)

+
+
+def test_result_from_zip_does_not_rewrite_links(tmp_path):
+ import io, zipfile


+
+
+def test_cloud_empty_extract_result_then_done(monkeypatch, tmp_path):
+ import io, sys, types, zipfile


+
+
+def test_full_md_basename_preferred_over_endswith(tmp_path):
+ import io, zipfile


+
+
+def test_image_basename_collision_warns(tmp_path, caplog):
+ import io, zipfile, logging as _logging


+ zf.writestr("images/fig.png", b"A")
+ zf.writestr("sub/fig.png", b"B")
+ with caplog.at_level(_logging.WARNING):
+ result = _result_from_zip(buf.getvalue())


KylinMountain · 2026-06-01T06:41:15Z

Summary

Adds a pluggable parser abstraction so the file → Markdown step can be routed
through higher-accuracy online/self-hosted parsers, addressing #77 (markitdown gives
breadth but not precision on complex docs like papers). The local parser remains the
zero-config default with no new required dependencies.

New openkb/parsers/ package: Parser ABC + ParseResult, a get_parser registry, and adapters:
- local (default) — existing pymupdf/markitdown behavior, refactored behind the abstraction (behavior-preserving).
- mineru — MinerU over HTTP: hosted cloud (submit→poll→zip) or self-hosted server.
- mistral — Mistral OCR via the official mistralai SDK (sync).
- vlm — any vision LLM via the existing litellm dep (Gemini, GPT-4o, Claude, …).
Selection via .openkb/config.yamlparser: + a parsers: per-provider block; per-run CLI override openkb add --parser <name> (validated by click.Choice).
Optional deps via pip extras: openkb[mistral], openkb[mineru], openkb[parsers]; vlm needs none. Adapters lazy-import and raise actionable errors when an extra/key is missing.
Image handling unified via localize_images (writes basenames, rewrites links; reuses extract_base64_images).

Scope / non-goals

PageIndex is untouched. Long PDFs (≥ pageindex_threshold, default 20) still go to PageIndex — the parser setting governs shorter docs and non-PDF formats.
Image understanding (Feature: Optional image understanding / vision for inline and referenced images #74) not included, but the reusable parsers/vlm_client.py is factored so Feature: Optional image understanding / vision for inline and referenced images #74 can plug in later.

Test plan

Unit tests for every adapter + registry + localize_images + the converter refactor (HTTP/SDK fully mocked — no network). MinerU cloud poll+download flow covered.
Full suite green (616 passed; the only repo-wide failures are pre-existing trafilatura url_ingest tests, unrelated).
Real-PDF smoke test of the local path (examples/docs/attention-is-all-you-need.pdf → Markdown + image extraction).
Real online-parser e2e (MinerU/Mistral/Gemini) — requires API keys; not run in CI.

Review fixes included

Hardened from review: MinerU poll-loop guards (non-positive interval, empty result), anchored image-link rewrite, path-traversal-safe image filenames, VLM global-model fallback warning, narrowed LLM_API_KEY propagation (no longer defeats the Mistral key guard), and single-source-of-truth parser dispatch.

…bitrary filenames (#77)

)

…tighten tests (#77)

)

…edundant localize for local parser (#77)

…h/VALID_PARSERS (#77)

+"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
+from __future__ import annotations
+
+from pathlib import Path


…localize_images (#77)

…ction; drop redundant image rewrite (#77)

…de (#77)

…st hygiene (#77)

+
+
+def test_result_from_zip_does_not_rewrite_links(tmp_path):
+ import io, zipfile


+
+
+def test_cloud_empty_extract_result_then_done(monkeypatch, tmp_path):
+ import io, sys, types, zipfile


+
+
+def test_full_md_basename_preferred_over_endswith(tmp_path):
+ import io, zipfile


+
+
+def test_image_basename_collision_warns(tmp_path, caplog):
+ import io, zipfile, logging as _logging


+ zf.writestr("images/fig.png", b"A")
+ zf.writestr("sub/fig.png", b"B")
+ with caplog.at_level(_logging.WARNING):
+ result = _result_from_zip(buf.getvalue())


KylinMountain · 2026-06-01T06:41:15Z

Summary

Adds a pluggable parser abstraction so the file → Markdown step can be routed
through higher-accuracy online/self-hosted parsers, addressing #77 (markitdown gives
breadth but not precision on complex docs like papers). The local parser remains the
zero-config default with no new required dependencies.

New openkb/parsers/ package: Parser ABC + ParseResult, a get_parser registry, and adapters:
- local (default) — existing pymupdf/markitdown behavior, refactored behind the abstraction (behavior-preserving).
- mineru — MinerU over HTTP: hosted cloud (submit→poll→zip) or self-hosted server.
- mistral — Mistral OCR via the official mistralai SDK (sync).
- vlm — any vision LLM via the existing litellm dep (Gemini, GPT-4o, Claude, …).
Selection via .openkb/config.yamlparser: + a parsers: per-provider block; per-run CLI override openkb add --parser <name> (validated by click.Choice).
Optional deps via pip extras: openkb[mistral], openkb[mineru], openkb[parsers]; vlm needs none. Adapters lazy-import and raise actionable errors when an extra/key is missing.
Image handling unified via localize_images (writes basenames, rewrites links; reuses extract_base64_images).

Scope / non-goals

PageIndex is untouched. Long PDFs (≥ pageindex_threshold, default 20) still go to PageIndex — the parser setting governs shorter docs and non-PDF formats.
Image understanding (Feature: Optional image understanding / vision for inline and referenced images #74) not included, but the reusable parsers/vlm_client.py is factored so Feature: Optional image understanding / vision for inline and referenced images #74 can plug in later.

Test plan

Unit tests for every adapter + registry + localize_images + the converter refactor (HTTP/SDK fully mocked — no network). MinerU cloud poll+download flow covered.
Full suite green (616 passed; the only repo-wide failures are pre-existing trafilatura url_ingest tests, unrelated).
Real-PDF smoke test of the local path (examples/docs/attention-is-all-you-need.pdf → Markdown + image extraction).
Real online-parser e2e (MinerU/Mistral/Gemini) — requires API keys; not run in CI.

Review fixes included

Hardened from review: MinerU poll-loop guards (non-positive interval, empty result), anchored image-link rewrite, path-traversal-safe image filenames, VLM global-model fallback warning, narrowed LLM_API_KEY propagation (no longer defeats the Mistral key guard), and single-source-of-truth parser dispatch.

…bitrary filenames (#77)

)

…tighten tests (#77)

)

…edundant localize for local parser (#77)

…h/VALID_PARSERS (#77)

+"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
+from __future__ import annotations
+
+from pathlib import Path


…localize_images (#77)

…ction; drop redundant image rewrite (#77)

…de (#77)

…st hygiene (#77)

+
+
+def test_result_from_zip_does_not_rewrite_links(tmp_path):
+ import io, zipfile


+
+
+def test_cloud_empty_extract_result_then_done(monkeypatch, tmp_path):
+ import io, sys, types, zipfile


+
+
+def test_full_md_basename_preferred_over_endswith(tmp_path):
+ import io, zipfile


+
+
+def test_image_basename_collision_warns(tmp_path, caplog):
+ import io, zipfile, logging as _logging


+ zf.writestr("images/fig.png", b"A")
+ zf.writestr("sub/fig.png", b"B")
+ with caplog.at_level(_logging.WARNING):
+ result = _result_from_zip(buf.getvalue())


KylinMountain added 22 commits May 31, 2026 10:35

feat(parsers): add ParseResult and Parser ABC (#77)
7a5f781

feat(images): add localize_images helper for parser output (#77)
592d11d

fix(images): use replacement function in localize_images to handle ar…
cec3618
…bitrary filenames (#77)

feat(parsers): add LocalParser wrapping legacy extraction (#77)
455c746

feat(parsers): add registry + get_parser factory (#77)
0978cbf

refactor(converter): route file→markdown through parser abstraction (#77
ed3368d
)

docs(converter): refresh convert_document docstring for parser flow; …
27d314c
…tighten tests (#77)

feat(parsers): add reusable litellm vision client (#77)
36e8b4f

fix(parsers): use litellm file content part for PDFs in vlm_client (#77)
8b1d4eb

feat(parsers): add VLMParser (vision LLM via litellm) (#77)
2a93eec

feat(parsers): add MistralParser via mistralai SDK (#77)
2c7e693

feat(parsers): log skipped undecodable Mistral images (#77)
50b83bb

feat(parsers): add MineruParser (cloud + self-hosted HTTP) (#77)
e452a3a

test(parsers): cover MinerU cloud poll+download flow (#77)
a6074f6

feat(cli): add --parser override and default parser config (#77)
82958e4

feat(cli): validate --parser against valid set via click.Choice (#77)
995b90c

build: add optional parser extras (mistral, mineru, parsers) (#77)
2959a8d

docs(readme): document pluggable document parsers (#77)
33cee68

fix(parsers): harden MinerU poll loop and anchor image-link rewrite (#77
526db30
)

fix(parsers): sanitize image filenames against path traversal; skip r…
8af174f
…edundant localize for local parser (#77)

fix(parsers): warn on VLM global-model fallback; unify parser dispatc…
e424bb4
…h/VALID_PARSERS (#77)

fix(cli): only propagate LLM_API_KEY to the active provider key (#77)
a981b91

github-code-qualityBot found potential problems Jun 1, 2026
View reviewed changes

Comment threadtests/test_parsers_local.py
"""Tests for LocalParser — preserves legacy md/pdf/markitdown behavior."""
from __future__ import annotations

from pathlib import Path
Comment threadtests/test_parsers_mineru.py Fixed
Comment threadtests/test_parsers_mineru.py Fixed

KylinMountain added 4 commits June 1, 2026 16:38

fix(images): match image links by basename (dir-prefixed, titled) in …
6287dea
…localize_images (#77)

fix(parsers): harden MinerU cloud response handling, timeout, md sele…
6e111fb
…ction; drop redundant image rewrite (#77)

fix(parsers): warn that VLM is text-only and on silent parser downgra…
02daf52
…de (#77)

fix(parsers): delete uploaded Mistral OCR files; fix patch.stopall te…
b243505
…st hygiene (#77)

github-code-qualityBot found potential problems Jun 1, 2026
View reviewed changes

gwokhou mentioned this pull request Jun 23, 2026
[Feature] Share PDF page extraction across short PDFs and long PDF page JSON #135
Open