CART-Doc is a compact document key information extraction (KIE) framework that selectively applies local visual processing to OCR regions. The model combines a DistilBERT text path, normalized OCR geometry, a preliminary BIO classifier, confidence-based top-K routing, MobileNetV3 crop encoding, and gated visual fusion.
cart_doc/ Python package
config/ Default experiment configuration
data/ Dataset loading and normalization
models/ CART-Doc and baseline models
scripts/ Single-run training/evaluation entry point
train/ Training, inference, metrics, and routing diagnostics
utils/ Logging and memory utilities
scripts/ Experiment orchestration and analysis
configs/ Dataset-format documentation
external/ Reference baseline helpers
tests/ Unit tests
Python 3.10 or newer is recommended.
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .
pytest -qFor GPU training, install a PyTorch build compatible with the local CUDA driver before installing the remaining dependencies if required by your system.
CORD v2 is loaded from naver-clova-ix/cord-v2 through Hugging Face Datasets. Images, OCR words, semantic categories, and bounding boxes are converted into the BIO representation used by the models.
BuDDIE visual/layout experiments require a multimodal manifest containing page images, OCR tokens, bounding boxes, BIO labels, entity dictionaries, and document classes. See configs/buddie_manifest.md for the required JSONL format.
Run the complete CORD v2 experiment suite:
./run_experiments.sh cord_v2 results/experimentsRun BuDDIE using a prepared manifest:
./run_experiments.sh buddie results/experiments \
--buddie-manifest-dir /path/to/buddie_manifestThe default experiment seeds are 42, 7, and 123. Routing budgets are selected on validation data from 64, 96, 128, and 192 before test evaluation.
python scripts/run_experiments.py --dataset cord_v2 --results-root results/experiments --stage audit
python scripts/run_experiments.py --dataset cord_v2 --results-root results/experiments --stage baselines
python scripts/run_experiments.py --dataset cord_v2 --results-root results/experiments --stage routing_train
python scripts/run_experiments.py --dataset cord_v2 --results-root results/experiments --stage routing_test
python scripts/run_experiments.py --dataset cord_v2 --results-root results/experiments --stage router_ablations
python scripts/run_experiments.py --dataset cord_v2 --results-root results/experiments --stage postprocessA single model can be trained directly, for example:
python -m cart_doc.scripts.train_model \
--model cart_doc \
--dataset cord_v2 \
--seed 42 \
--output-dir results/experiments \
--run-name confidence_k96 \
--route-k 96 \
--router-mode confidenceSupported model keys are distilbert, lilt, layoutlmv3, ltrf_doc, and cart_doc.
Supported CART-Doc router modes are confidence, learned, and hybrid.
The experiment runner separates model selection from test evaluation:
- audit dataset integrity and label coverage;
- train baseline models across the configured seeds;
- train routing-budget candidates using validation data;
- select and lock the routing budget;
- evaluate the routing sweep on the test split;
- train the selected CART-Doc configuration across the configured seeds;
- run text-layout, learned-router, and hybrid-router ablations;
- aggregate metrics, routing diagnostics, bootstrap comparisons, and qualitative examples.
The label vocabulary is constructed from training data. For CART-Doc, routing operates only over valid first-subtoken OCR units and uses K_d = min(K, n_d) for each document.
Each run writes to:
results/experiments/<dataset>/<model>/<run_name>/seed_<seed>/
Typical files include:
manifest.json
checkpoints/
logs/
metrics/
predictions/
artifacts/
Dataset-level summaries are written to:
results/experiments/<dataset>/reports/
They include multi-seed mean/standard-deviation tables, routing diagnostics, routing-budget sensitivity, paired bootstrap comparisons, per-document-type summaries, and qualitative routed examples.
scripts/audit_dataset.py— dataset integrity and label-space auditscripts/select_routing_budget.py— validation-based routing-budget selectionscripts/evaluate_routing_sweep.py— evaluate saved routing-budget candidatesscripts/aggregate_multiseed.py— aggregate metrics across seedsscripts/paired_bootstrap.py— document-paired bootstrap comparisonsscripts/generate_qualitative_cases.py— select routed correction and failure casesscripts/generate_latex_tables.py— export compact LaTeX table rowsscripts/verify_outputs.py— verify required experiment outputs
external/layoutlmv3_reference/ contains a helper for running Microsoft's LayoutLMv3 CORD example in a separate environment. The upstream implementation uses its own preprocessing and evaluation protocol, so those results should be interpreted separately from the common CART-Doc evaluator.
If you use this codebase, please cite the associated manuscript:
@inproceedings{oladele2026_cart_doc,
title = {CART-Doc: A Compact Design Space for Confidence-Routed Document Key Information Extraction},
author = {Daniel Ayo, Oladele and Malusi Sibiya},
booktitle = {The 19th International Conference on Multi-disciplinary Trends in Artificial Intelligence (MIWAI) 2026},
series = {Lecture Notes in Artificial Intelligence},
publisher = {Springer},
year = {2026}
}Citation metadata is provided in CITATION.cff.