📚 Tabulus: Scientific PDF Table Extraction Pipeline

🔍 Overview

Tabulus is a modular multi-stage pipeline for extracting structured table data from scientific PDF documents.

The current rebuilt library supports PDF profiling, canonical table-crop export, table reconstruction, reference-table classification, GROBID-backed bibliography extraction, and deterministic reference matching. Paper-level scholarly reference resolution, resolved export, and full run orchestration remain planned in this checkout.

The project was developed as part of a Master's thesis investigating scientific table extraction, OCR benchmarking, bibliography-aware processing, and structured scholarly knowledge extraction.

---

✨ Features

📄 Scientific Table Extraction

• Automated table detection from scientific PDFs
• Table cropping and preprocessing
• Adapter-based table reconstruction from fixed canonical crops
• Raw prediction CSV generation before reference resolution

🔗 Bibliography-Aware Processing

• One deterministic regex/rule classifier applied to each reconstruction method's outputs
• Preserved separation between reconstruction predictions and reference routing
• GROBID bibliography extraction from full publications
• Offline table-cell-to-bibliography-position matching
• Clear planned boundary for paper-level scholarly reference resolution

📊 Research & Evaluation

• OCR benchmarking framework
• RMS-based table similarity evaluation
• Precision, Recall, and F1-score analysis
• Runtime benchmarking
• Reproducible evaluation workflows

🏗️ System Design

• Modular CLI and library architecture
• Explicit filesystem contracts between stages
• Separate ML environments for heavyweight adapters
• CPU and GPU reconstruction-adapter support
• Legacy service implementation retained separately from the rebuilt library

---

⚙️ Pipeline Workflow

Scientific PDF
|
+--> tabulus profile / MinerU
| |
| +--> MinerU table_body
| |
| +--> canonical MinerU table crops
| |
| v
| reconstruction adapters
| |
| v
| prediction CSVs
| |
| v
| Stage 3 reference-table classification
|
+--> Stage 4 GROBID bibliography extraction
|
v
references/bibliography.json
selected_reference_tables.json + bibliography.json
|
v
references/reference_matches.json (Stage 5)
|
v
Stage 6: paper-level scholarly reference resolution (planned)
|
v
Stage 7: join resolved identities to all relevant cells / export (planned)

---

📁 Repository Structure

tabulus/
│
├── assets/
│ ├── img/
│ └── logo.png
│
├── dataset/
│ └── README.md
│
├── evaluation/
│ ├── deplot/
│ ├── new_results/
│ ├── plots/
│ │ ├── reference_extraction/
│ │ ├── scripts/
│ │ └── table_extraction/
│ ├── scripts/
│ └── README.md
│
├── docs/
│ └── ...
│
├── src/
│ ├── tabulus/
│ │ ├── mineru/
│ │ ├── reference_tables/
│ │ ├── table_ocr/
│ │ └── cli.py
│ │
│ ├── legacy_tabulus/
│ │ └── ...
│ │
│ ├── ocr_models/
│ │ └── ...
│ │
│ └── README.md
│
├── tests/
│ └── ...
│
├── .gitignore
├── LICENSE
├── README.md
├── pyproject.toml
└── requirements.txt

---

🧩 Main Components

Component	Purpose
src/tabulus	Current installable Tabulus library and CLI
src/legacy_tabulus	Retained legacy thesis implementation
legacy/ocr_models	Historical OCR services, runners, and benchmarking components
docs	ReadTheDocs documentation
tests	Current library test suite
evaluation	Evaluation scripts, metrics, and visualizations
dataset	Benchmark dataset documentation and ground-truth structure
assets	Images and visual resources used in the documentation

Detailed documentation for each component is available in the corresponding README files.

---

🤖 External Tools And Models

The rebuilt Tabulus library currently uses MinerU for PDF profiling and a registry of Stage 2 reconstruction adapters for canonical MinerU crops. The complete supported-adapter table is maintained in the ReadTheDocs page:

docs/tutorial/08-table-ocr.md

GROBID is used by the rebuilt bibliography-extraction branch. Kreuzberg remains relevant only in retained historical or fallback-planning material; neither GROBID nor Kreuzberg is a Stage 2 reconstruction adapter.

---

🗄️ Dataset

The project uses a manually curated evaluation dataset containing:

• scientific publications,
• annotated tables,
• bibliography references,
• OCR outputs,
• DOI matching results,
• evaluation metrics.

The complete dataset exceeds 700 MB and is distributed separately.

See:

dataset/README.md

for details.

---

📈 Evaluation

A comprehensive evaluation framework is included for analyzing:

• table extraction quality,
• OCR robustness,
• bibliography extraction performance,
• reference matching coverage, consistency, and agreement,
• reference-resolution coverage and evidence quality after that stage is implemented,
• runtime efficiency.

Generated benchmark plots and visualizations are available in:

evaluation/plots/

See:

evaluation/README.md

for detailed documentation.

---

🚀 Running the Current Rebuilt Workflow

Install the current library from the repository checkout:

python -m pip install -e ".[dev]"

The commands below are verified in this checkout. For one PDF:

tabulus profile --pdf /path/to/paper.pdf --backend pipeline
tabulus reconstruct-tables \
--crops /path/to/tabulus-output/table-crops/<paper> \
--adapter <adapter> \
--device gpu:0
tabulus classify-reference-tables \
--reconstruction /path/to/tabulus-output/table-crops/<paper>/reconstructions/<adapter>
tabulus extract-bibliography \
--pdf /path/to/paper.pdf \
--out /path/to/artifact-root \
--grobid-url http://localhost:8070

For several PDFs in one folder:

tabulus profile \
--folder /path/to/papers \
--backend hybrid-engine \
--method auto \
--effort high
tabulus reconstruct-tables \
--crops-folder /path/to/papers/tabulus-output/table-crops \
--adapter <adapter> \
--device gpu:0
tabulus classify-reference-tables \
--crops-folder /path/to/papers/tabulus-output/table-crops \
--adapter <adapter>

See the ReadTheDocs installation pages for Windows CPU setup, GPU-server setup, and adapter-specific environments. The legacy Docker/service workflow is not the current rebuilt-library entry point.

---

📖 Documentation

Additional documentation is available in:

https://tabulus.readthedocs.io/
docs/
evaluation/
dataset/

Each README contains detailed setup instructions, implementation details, API documentation, evaluation procedures, and usage examples.

---

🎓 Research Context

This repository accompanies a Master's thesis focused on:

• scientific table extraction,
• OCR benchmarking,
• bibliography-aware table processing,
• DOI enrichment,
• structured scientific knowledge extraction,
• reproducible research workflows.

---

📑 Citation

If you use this repository in your research, please cite the associated Master's thesis.

Citation information will be added after publication.

---

📜 License

This project is provided for research and educational purposes.