Bulk ingestion: per-document rewrites of entity/concept pages scale ~O(N²) on large, topically-dense corpora — propose a two-phase (map/reduce) consolidation mode

[Megej]

Summary

When back-filling a large, topically-dense corpus in one pass, openkb add gets progressively slower because entity and concept pages are re-generated in full on every document. For "hot" entities/concepts that appear in most documents, the page grows with each ingest and is rewritten each time, so total work scales roughly O(N²) and per-document latency/token-cost climbs steadily. The incremental model is great for steady-state use (a few sources at a time); it's a poor fit for a one-time migration of hundreds–thousands of documents.

Environment

• openkb 0.4.5 (pip), Python 3.13
• Model: bedrock/us.anthropic.claude-sonnet-4-6 via LiteLLM
• Corpus: ~1,100 documents, topically dense (many documents about the same handful of suppliers, standards bodies, and topics); mixed PDF/DOCX/PPTX

Observed behavior

Ingesting one document at a time, update: / entity-update: calls on recurring pages grow without bound:

• A supplier entity referenced across dozens of documents: its update call grew from ~69s to ~154s (~4.6k → ~8.3k output tokens) over a single 20-document batch as the page accreted.
• An umbrella concept: one update reached ~310s / ~14k output tokens.
• Worst case, an entity representing the corpus author (present in nearly every document): ~290s / ~13.7k output tokens per update — until we excluded it via AGENTS.md.

Each update regenerates the whole (growing) page, so cost is dominated by re-writing the same accreting pages rather than by the new document's content. recompile re-runs the same per-document loop, so it doesn't help.

Why it matters

For the intended steady-state workflow (drop a source, review) this is invisible — pages stay small and stable. But for a one-time backfill of a large archive, per-document cost climbs into minutes and the token spend / wall-time make full ingestion impractical (our ~1,100-doc set extrapolates to tens of hours).

Proposed enhancement — a batch / two-phase consolidation mode

An ingestion mode optimized for bulk backfill:

1. Map (per document, parallelizable): extract entity/concept mentions from each document independently — cheap, and with no dependence on the growing wiki state.
2. Reduce (once per unique entity/concept): after the batch (or per N-document chunk), synthesize each entity/concept page a single time from all its collected mentions.

This turns O(N²) full-page rewrites into ~O(N) extraction + one synthesis per unique page. It also:

• makes entity canonicalization easier (all name variants are visible at once, so foo-bar vs bar-foo duplicates can be merged in the reduce);
• lets contradiction detection run over the full mention set rather than incrementally;
• can go hierarchical for very large corpora (chunk → summarize → merge), dovetailing with the roadmap item "hierarchical concept (topic) indexing for massive knowledge bases."

Possible CLI surface: openkb add <dir> --bulk (or --batch-consolidate), leaving the default per-document behavior unchanged for steady-state use.

Alternatives / current mitigations (all partial)

• Excluding "hot" entities (e.g., a corpus-author self-entity) via AGENTS.md.
• Splitting the corpus into per-topic knowledge bases so fewer documents hit the same pages.
• Keeping concepts atomic / avoiding umbrella pages (schema guidance — but the compiler still rewrites whole pages).
• Smaller batches + re-runs.

None address the core O(N²) rewrite cost for genuinely recurring entities/concepts.

Relation to roadmap

Aligns with "scale to large document collections" and "hierarchical concept (topic) indexing for massive knowledge bases." Happy to help test.