Skip to content

Repository files navigation

OKFgraph

Ladybug-backed knowledge graph with ONNX-optimized Jina v5 embeddings, multimodal image ingestion, delta-aware incremental imports, and schema migration.

Architecture: architecture.md v5.9 — all 16 gaps CLOSED.

OKFgraph is a Python library and CLI tool for building, querying, and managing knowledge graphs from Markdown/OKF documents. It combines graph traversal, hybrid semantic search, chunk-level retrieval, and — in v5.1 — delta detection with safe purge of deleted concepts into a single SQLite-backed system.


Features

CategoryFeatures
EmbeddingsONNX-optimized Jina v5 text model, SentenceTransformer omni model (lazy-loaded), Matryoshka truncation (32–1024 dims, default 512), numpy-only post-processing (no torch)
SearchHybrid RRF fusion (vector + FTS), graph traversal, chunk-level search with RRF, graph-aware reranking (hub scores), context expansion, direct lookup, image search via unified index
StorageLadybugDB — graph + vector + full-text search in a single SQLite file
ImagesThree ingestion modes (text / optional / omni), unified vector space for text + image, content-hash dedup, okf-asset:// protocol
Import/ExportOKF Markdown round-trip, batch import with single-transaction upsert, delta detection (SHA-256 hash skip), purge deleted concepts (--purge), filtered bulk export with graph enrichment (See Also + Cited By), auto-generated index.md files
PDF Ingestionokfgraph.ingest sub-module — pdf_oxide fast path + ONNX/Rapid heavy passes (RapidLaTeXOCR, RapidOCR, RapidLayout, RapidTable), four routing modes (NEVER/AUTO/SURGICAL/ALWAYS), Paddle-free
CLI14 commands + interactive REPL, --mode for image ingestion, --device cuda with auto-fallback, chunking flags (--chunk-overlap, --no-chunking), --purge for stale concept cleanup
LLM Tools16 OpenAI-compatible tool definitions for agent integration (search, traverse, chunks, graph enrichment, path finding, export, ingest_md, ingest_thoughts, ingest_pdf)
MCP ServerModel Context Protocol server (okf-mcp) — stdio transport, lifespan-managed router, 16 tools with read/write annotations, compatible with Claude Desktop, Cursor, Continue

Quick Start

Installation

pip install ladybug pydantic python-frontmatter pyyaml optimum[onnxruntime] transformers torch sentence-transformers Pillow

Programmatic Usage

fromokfgraphimportOKFRouter, ConceptModel# Initialize the routerrouter=OKFRouter(
db_path="okfgraph.db",
bundle_root="./my-knowledge-base",
embedding_dim=512, # Matryoshka: 32, 64, 128, 256, 512, 768, 1024device="cuda", # or "cpu" — defaults to CUDA when onnxruntime-gpu is available
)
# Import an OKF bundleids=router.import_bundle(mode="optional")
print(f"Imported {len(ids)} concepts")
# Hybrid searchresults=router.search_hybrid("machine learning basics", limit=5)
forrinresults:
print(f" [{r['relevance_score']:.3f}] {r['title']} ({r['type']})")
# Chunk-level search (RRF-fused vector + FTS)chunks=router.search_chunks("neural network architecture", limit=5)
forcinchunks:
print(f" [{c['rrf_score']:.3f}] {c['parent_title']} §{c['chunk_index']}")
# Search with graph contextenriched=router.search_with_context("graph neural networks", limit=3)
forrinenriched:
chunk=r["chunk"]
print(f" [{chunk['rrf_score']:.3f}] {chunk['parent_title']}")
print(f" ← linked by: {len(r['incoming_links'])} docs")
# Hub-score rerankingranked=router.search_chunks_with_hub_score("fundamentals", limit=10)
forrinranked:
print(f" [{r['final_score']:.3f}] hub={r['hub_score']:.2f}{r['parent_title']}")
# Reconstruct document from chunksoriginal=router.reconstruct_document("concepts/intro")
# Image search (text → image via unified index)images=router.search_images_with_text("diagram of a neural network", limit=10)
forimginimages:
print(f" [{img['relevance_score']:.3f}] {img['file_name']}")
# Graph traversalnodes=router.traverse("concepts", relationship="CONTAINS", direction="OUTGOING", depth=2)

CLI Usage

# Initialize database
okf init --bundle ./my-knowledge-base --dim 512
# Import a single file
okf import ./concepts/intro.md --mode optional
# Import an entire bundle
okf import --all --mode omni --allow-remote-images
# Import with delta detection + purge deleted concepts
okf import --all --purge
# Hybrid search
okf search "graph neural networks" --type section --limit 5
# Hybrid search with matched chunks
okf search "machine learning" --chunks
# Chunk-level search
okf search-chunks "neural network architecture"# Search with graph context
okf context "graph neural networks"# Hub-score reranking
okf hub-search "fundamentals"# Find path between concepts
okf path concepts/intro concepts/advanced
# List chunks for a concept
okf chunks concepts/intro
# Reconstruct document from chunks
okf reconstruct concepts/intro
# Image search
okf search-images "network architecture diagram"# List directory
okf list concepts
# Interactive shell
okf shell

MCP Server

OKFgraph includes a Model Context Protocol server that exposes all 16 tools over stdio transport. Compatible with Claude Desktop, Cursor, Continue, and any MCP client.

# Start MCP server (stdio transport)
okf-mcp --db-path ./okfgraph.db
# With GPU acceleration
okf-mcp --db-path ./okfgraph.db --device cuda
# With custom embedding dimension
okf-mcp --db-path ./okfgraph.db --embedding-dim 512

Claude Desktop config (claude_desktop_config.json):

{
"mcpServers": {
"okfgraph": {
"command": "python",
"args": ["-m", "okfgraph.mcp_server", "--db-path", "/path/to/okfgraph.db"]
}
}
}

Programmatic usage:

fromokfgraph.mcp_serverimportcreate_mcp_servermcp=create_mcp_server(
db_path="./okfgraph.db",
bundle_root="./my-knowledge-base",
device="cpu",
embedding_dim=1024,
)
mcp.run(transport="stdio")

Architecture

Embedding Engine

OKFgraph uses a dual-model approach with a unified vector space:

ModelRoleFrameworkLoaded
jina-embeddings-v5-text-small-retrievalText embeddings for conceptsONNX Runtime (optimum)Always
jina-embeddings-v5-omni-small-retrievalImage embeddings (vision tower)SentenceTransformerLazy (first use)

Both models output vectors in the same Matryoshka space, so text and image embeddings are directly comparable in the image_omni_idx index.

Image Ingestion Modes

ModeImage with alt-textImage without alt-textOmni model loaded?
textEmbed alt-text (lightweight)Embed filename + image-numberNever
optionalEmbed alt-text (lightweight)Embed image bytes (rich)Only for images without alt-text
omniEmbed image bytes (rich)Embed image bytes (rich)Always (if bytes available)

When omni is requested but raw bytes are unavailable (e.g. remote URLs not fetched), the system degrades gracefully to the text path.

Database Schema

Concept ImageAsset Directory BrokenLink Chunk FileHash
├── id ├── id ├── id ├── id ├── id ├── path (PK)
├── type ├── file_name └── (empty) ├── source_id ├── parent_doc_id ├── hash
├── title ├── mime_type ├── target_id ├── chunk_index ├── concept_id
├── body ├── alt_text └── timestamp ├── chunk_text
├── embedding ├── caption ├── block_type
└── extra ├── embed_route ├── start_offset
├── content_hash ├── end_offset
├── data (BLOB) └── embedding
└── embedding
CONTAINS: Directory → Directory / Concept
LINKS_TO: Concept → Concept
INCLUDES_ASSET: Concept → ImageAsset
PART_OF: Concept → Chunk

Chunking & Graph Retrieval

OKFgraph v5.0 adds document chunking with graph-aware retrieval:

FeatureDescription
Mordant chunkerRust-based Markdown parser that splits documents into semantic blocks (paragraphs, headings, code blocks, lists, tables, blockquotes)
Heading context injectionParagraph chunks get parent heading prepended to embedding payloads without mutating stored text
Structural boundariesHeadings, code blocks, lists, tables, blockquotes, and diagrams enforce hard boundaries — no overlap tails bleed across them
RRF-fused chunk searchVector + FTS fusion at chunk granularity with per-doc limits and graph filters
Hub-score rerankingChunks from authoritative documents (high incoming link count) rank higher
Graph context expansionSearch results enriched with incoming/outgoing links, directory ancestry, and sibling concepts
Path findingBFS shortest path between any two concepts in the knowledge graph
Document reconstruction~98% fidelity round-trip from chunks back to original Markdown
Hybrid search with chunkssearch_hybrid(query, include_chunks=True) attaches matched chunks to concept results

PDF Ingestion Engine (okfgraph.ingest)

The okfgraph.ingest sub-module provides a Paddle-free PDF → Markdown conversion pipeline using the RapidAI family of ONNX models. Four routing modes control when ONNX models are invoked:

ModeBehaviour
NEVERFast path only (pdf_oxide). No ONNX models loaded.
AUTOHeuristics per page → full ONNX pipeline only on flagged pages.
SURGICALFormula crops via RapidLaTeXOCR; full pipeline only for scans.
ALWAYSEvery page through the full ONNX layout + OCR pipeline.

Key features:

  • Zero hard dependencies — all RapidAI imports are guarded; the module loads cleanly without them
  • Lazy loading — born-digital PDFs never pay for OCR/layout/table models
  • Graceful degradation — if a model fails to load, the pipeline falls back to the fast path
  • Device → ort_providers coerciondevice="cuda" auto-resolves to ["CUDAExecutionProvider", "CPUExecutionProvider"] (accepts "gpu" as alias)
  • Output contract — single .md with inline/display LaTeX, fenced code, GFM tables, and okf-asset:// links

Key Design Decisions

  • Last-token pooling (not mean pooling) — required by Jina v5's training protocol
  • Delete-then-create upserts — LadybugDB vector indexes block SET on indexed properties
  • Sequential encoding — padded batch tokenization wastes attention compute with variable-length documents
  • Content-hash dedup — unchanged images are not re-embedded on re-import
  • okf-asset:// protocol — markdown references use UUID-based URIs instead of inline Base64

API Reference

OKFRouter

MethodDescription
__init__(db_path, bundle_root, model_id, omni_model_id, embedding_dim, cache_dir, device, allow_remote_images)Initialize router
import_from_okf(file_path, mode)Import a single OKF file
import_bundle(bundle_path, batch_size, mode, purge_deleted)Import entire bundle with batch encoding and delta detection
export_to_okf(concept_id, output_path)Export a single concept to OKF markdown
export_bundle(output_dir, directory_id, concept_type, tags)Export filtered bundle with graph enrichment
search_hybrid(query, concept_type, tags, parent_id, exclude_reserved, limit)Hybrid RRF search
search_images_with_text(text_query, use_text_model, limit)Text→image search via unified index
traverse(start_id, relationship, direction, depth, node_type)Graph traversal
list_directory(directory_id)List children of a directory
get_by_id(concept_id)Fetch full concept by ID
list_images(concept_id)List image assets attached to a concept
get_image_data(asset_id)Fetch image asset with raw bytes
list_broken_links()List orphaned links
repair_links()Repair broken links
model_info(model_id, cache_dir)Inspect model cache status (class method)
search_chunks(query, concept_type, tags, parent_id, limit, max_chunks_per_doc)RRF-fused chunk search
search_with_context(query, limit, context_hops)Chunk search + graph neighborhood expansion
search_chunks_with_hub_score(query, limit, hub_weight)Chunk search reranked by hub score
get_chunks(concept_id)List all chunks for a concept
reconstruct_document(document_id)Reconstruct original Markdown from chunks
find_path(start_id, end_id, max_length)BFS shortest path between concepts
expand_with_graph_context(chunk_ids)Discover related concepts via graph edges
rerank_with_hub_score(chunk_results)Adjust chunk scores by parent hub score
ingest_pdf(pdf_path, auto_import, output_dir, ...)Programmatic PDF ingestion (Gap #5b)
_purge_concept(concept_id)Safe cascading delete (concept, chunks, links, orphaned assets)
_changed_files(source_files)Delta detection — returns (changed, deleted) tuple
_file_hash(path)Compute SHA-256 hex digest of a file

IngestMode

fromokfgraph.imagesimportIngestModeIngestMode.TEXT# alt-text / filename fallback (no omni)IngestMode.OPTIONAL# omni only for images lacking alt-textIngestMode.OMNI# every image via omni model

Aliases: text-only, altTEXT; hybrid, autoOPTIONAL; full, multimodalOMNI.

ConceptModel

classConceptModel(BaseModel):
id: strtype: strtitle: Optional[str] =Nonedescription: Optional[str] =Noneresource: Optional[str] =Nonetags: List[str] =Field(default_factory=list)
timestamp: Optional[datetime] =Nonebody: str=""embedding: Optional[List[float]] =None# extra frontmatter keys captured via model_config.extra = "allow"

CLI Reference

CommandDescription
okf initInitialize database and schema
okf model-infoShow model cache status
okf import <files>Import one or more OKF files
okf import --allImport entire bundle recursively
okf search <query>Hybrid search (with --chunks for matched chunks)
okf search-images <query>Find images via unified vector index
okf search-chunks <query>Chunk-level RRF-fused search
okf context <query>Search with graph neighborhood expansion
okf hub-search <query>Chunk search reranked by hub score
okf traverse <id>Graph traversal
okf list [dir]List directory contents
okf get <id>Fetch full concept
okf export --allExport entire bundle
okf path <id1> <id2>Find shortest path between concepts
okf siblings <id>List sibling concepts in same directory
okf ancestry <id>Show directory hierarchy for a concept
okf chunks <id>List chunks for a concept
okf reconstruct <id>Reconstruct document from chunks
okf broken-linksList orphaned links
okf repair-linksRepair broken links
okf reindex [--if-dirty]Rebuild vector + FTS indexes
okf shellInteractive REPL

Global Options

OptionDefaultDescription
--dbokfgraph.dbDatabase file path
--bundle.Bundle root directory
--dim512Embedding dimension (32–1024)
--cache-dir~/.cache/huggingfaceHuggingFace model cache
--devicecudacpu or cuda
--omni-model-idjinaai/jina-embeddings-v5-omni-small-retrievalMultimodal model ID
--verbose, -vVerbose logging (Gap #10)
--quiet, -qSuppress non-essential output
--log-fileWrite logs to file
--profileEnable cProfile profiling (Gap #10)

Import Options

OptionDefaultDescription
--modetextImage ingestion mode: text, optional, omni
--allow-remote-imagesFetch http(s):// image URLs
--batch-size32Batch size for encoding
--purgeAlso purge concepts whose source files were deleted from disk

Performance

MetricTimeNotes
Single import (100 concepts)~140s~1400ms per concept
Batch import (100 concepts)~128s1.1x faster than single
Hybrid search107ms mean5 repetitions
Export (100 concepts)77ms0.8ms per concept

Batch speedup comes from DB-level optimizations (single transaction, bulk directory/link building), not from ONNX batching — sequential single-pass encoding avoids padding overhead with variable-length documents.


Testing

# Run all tests
pytest tests/ -v
# Run just image tests (no heavy deps)
python tests/test_images.py
# Run integration tests (requires installed models)
pytest tests/test_integration.py -v
Test SuiteTestsStatus
test_ingest.py39✅ All passing (Gap #5b: ingest_pdf method, version checking, engine degradation, asset staging)
test_router_misc.py33✅ All passing (Gap #12a: reindex, repair_links, meta/epoch, error isolation, context window, schema migration)
test_router.py29✅ All passing
test_chunking.py26✅ All passing
test_delta.py21✅ All passing
test_cli.py19✅ All passing
test_export_compliance.py13✅ All passing
test_integration.py11✅ All passing
test_images.py11✅ All passing
test_logging.py10✅ All passing (Gap #10: logging setup, timing instrumentation, profiling)
test_directory_hash.py10✅ All passing (Gap #1b: directory-level hash aggregation, purge of deleted subtrees)
test_search_browser.py9✅ All passing (PySide6 stubbed)
test_graph_enrichment.py9✅ All passing
test_chunk_search.py7✅ All passing
test_reconstruction.py5✅ All passing
test_okf_ingest_tool.py5✅ All passing
test_ingest_tool.py3✅ All passing
test_converter.py2✅ All passing (PySide6 stubbed)
Total262All passing (0 warnings)

Gap Analysis & Production Readiness

OKFgraph has undergone a comprehensive gap analysis (docs/gap-analysis.md) assessing production-readiness across 15 areas:

CategoryStatusNotes
Delta Detection✅ Closed (v5.1, v5.5)File-level SHA-256 + directory-level hash aggregation + purge
Schema Migration✅ Closed (v5.3)Versioned migrations (v1→v2→v3→v4)
PDF Ingestion✅ Closed (v5.3)CLI okf ingest command with --auto-import
Error Isolation✅ Closed (v5.1)Per-concept error isolation in import pipeline
Index Lifecycle✅ Closed (v5.1)Epoch-based dirty tracking + change-driven rebuild
Context Window✅ Closed (v5.1)90% threshold warning for oversized chunks
Missing Tests✅ Closed (v5.7)300 tests across 16 files (15 GPU tests)
RapidAI Pinning✅ Closed (v5.4)Version pins + runtime warning
Observability✅ Closed (v5.4)Structured logging + profiling hooks
PDF Ingest API✅ Closed (v5.4)OKFRouter.ingest_pdf() programmatic API
Open: Concurrency⚠️ OpenWAL mode + single-writer constraint needed

See architecture.md §15 for details on open gaps.


Project Structure

okfgraph/
├── okfgraph/
│ ├── __init__.py # Exports: ConceptModel, OKFRouter, cli_main, ChunkModel
│ ├── models.py # ConceptModel + ImageAssetModel + ChunkModel
│ ├── router.py # OKFRouter — ~2700 lines (chunking, graph enrichment, export)
│ ├── cli.py # CLI + interactive shell — ~800 lines
│ ├── tools.py # LLM tool definitions (13 tools)
│ ├── images.py # Image ingestion logic
│ ├── ingest/ # ONNX/Rapid PDF ingestion engine
│ │ ├── __init__.py # Public API exports
│ │ ├── config.py # ConverterConfig + RoutingMode
│ │ ├── engine.py # OnnxRapidEngine (lazy ONNX loaders)
│ │ ├── converter.py # HybridConverter (core pipeline)
│ │ ├── tables.py # HTML → GFM pipe-table converter
│ │ ├── assets.py # okf-asset:// staging logic
│ │ └── versions.py # RapidAI version pinning + runtime checks (Gap #15)
│ └── docs/
│ ├── chunking-and-graph-retrieval.md # Chunking specification
│ └── chunking-status.md # Implementation status
├── tests/
│ ├── test_chunking.py # 13 chunking unit tests
│ ├── test_reconstruction.py # 9 document reconstruction tests
│ ├── test_chunk_search.py # 9 chunk search tests
│ ├── test_graph_enrichment.py # 11 graph enrichment tests
│ ├── test_integration.py # 16 end-to-end tests
│ ├── test_export_compliance.py # 13 OKF export compliance tests
│ ├── test_ingest.py # 39 ONNX/Rapid ingestion tests (version checking, ingest_pdf, asset staging)
│ ├── test_delta.py # 21 delta detection & purge tests
│ ├── test_router_misc.py # 23 router unit tests (reindex, repair_links, meta/epoch, error isolation, context window)
│ ├── test_router.py # 29 smoke & cache & device tests
│ ├── test_logging.py # 10 logging & profiling tests
│ ├── test_converter.py # 2 converter staging tests (PySide6 stubbed)
│ ├── test_search_browser.py # 9 search browser tests (PySide6 stubbed)
│ └── fixtures/bundle/ # Test markdown fixtures
├── benchmarks/
│ └── benchmark_500.py
├── examples/
│ ├── hybrid_pdf_converter.py # PySide6 UI (legacy Paddle stack)
│ └── oxide_to_markdown_converter.py
├── docs/
│ ├── okf-export-compliance.md # OKF export specification
│ ├── ONNX_RAPID_IMPLEMENTATION.md # ONNX/Rapid migration guide
│ └── gap-analysis.md # Production-readiness gap analysis (v3.0)
├── architecture.md # Full architecture specification (v5.4)
├── pyproject.toml # Project metadata
└── requirements.txt # Dependencies

Requirements

Core (required)

  • Python 3.10+
  • ladybug>=0.17
  • pydantic>=2.0
  • python-frontmatter>=1.0
  • pyyaml>=6.0
  • optimum[onnxruntime]>=1.20
  • transformers>=4.57
  • numpy>=1.24 (replaces torch for post-processing)
  • mordant>=0.12 (Rust-based Markdown chunker)
  • sentence-transformers>=3.0
  • Pillow>=10.0
  • pytest>=8.0 (dev)

PDF Ingestion (optional — ONNX/Rapid stack)

  • pdf_oxide — fast native PDF→Markdown (MIT)
  • rapidocr — text detection + recognition (Apache-2.0)
  • rapid_latex_ocr — formula image → LaTeX (MIT)
  • rapid_layout — layout region detection (Apache-2.0)
  • rapid_table — table structure → HTML (Apache-2.0)
  • onnxruntime-gpu or onnxruntime-directml or onnxruntime (pick one for your hardware)

All RapidAI imports are guarded — the ingestion sub-module loads cleanly without them installed.


License

See LICENSE for details.


Contributing

Contributions welcome! Please open an issue or pull request. Key areas for contribution (per gap analysis):

  • P1: RapidAI version pinning — pin exact versions + runtime warning ✅ (closed)
  • P2: Structured logging — replace print() with stdlib logging ✅ (closed)
  • P2: PDF Ingest APIOKFRouter.ingest_pdf() programmatic API ✅ (closed)
  • P2: WAL mode — enable SQLite WAL for concurrent read access ✅ (closed, v5.8)
  • P2: URL allowlist — restrict --allow-remote-images to safe domains ✅ (closed, v5.8)
  • P2: TOML config — add okfgraph.toml + env var support ✅ (closed, v5.8)
  • P2: Soft-delete recovery — undo purge within 24h recovery window ✅ (closed, v5.8)
  • P2: Query latency tracking — log search query latency
  • P2: Embedding cache hit rates — track ONNX model cache hit/miss
  • P3: LLM tool definition — add ingest_pdf tool to tools.py ✅ (closed)
  • P3: CLI okf ingest parity — fix stage_images 5-arg call + mordant linting ✅ (closed)
  • P3: ingest_thoughts linting — defensive in-memory lint via _lint_converted_md_str() ✅ (closed)
  • P3: Tool parameter parity — fix search_hybrid type filter, ingest_pdf bool default, traverse defaults ✅ (closed)
  • GPU performance benchmarking at scale (10k+ concepts)
  • Real-world OKF bundle testing
  • --skip-embedding flag for faster imports without ONNX encoding
  • ONNX/Rapid end-to-end PDF tests (see docs/ONNX_RAPID_IMPLEMENTATION.md testing checklist)
  • Office file conversion via office_oxide
  • Delta detection: directory-level hash aggregation (skip entire subtrees)
  • Delta detection: soft-delete with recovery window (undo purge within N hours)
  • okf index-status command — report epoch, dirty state, last rebuild time
  • okf schema --version command — inspect schema version

About

OKF knowledge graph: LadybugDB + Jina v5 ONNX embeddings, hybrid RRF search, graph traversal, and v4.0 multimodal image ingestion with three ingestion modes.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages