Production-resilient TypeScript engine for PDF text chunking, Google Gemini vector embeddings, and local ChromaDB semantic vector search.
vector-doc-engine is a lightweight library and CLI tool designed for document processing pipelines. It splits documents cleanly at natural text boundaries, generates embeddings with Google Gemini, and indexes them into ChromaDB for fast similarity retrieval.
- Recursive Text Chunking: Splits paragraphs, lines, and sentences cleanly before applying character limits.
- Resilient API Layer: Uses exponential backoff with delay retries to handle rate limits and temporary network drops.
- Streaming Batch Processing: Processes chunks in steady batches to keep memory usage minimal on large documents.
- Similarity Threshold Filtering: Filters search results against a similarity floor to exclude off-topic matches.
Install as a project dependency:
npm install git+https://github.com/DileepWick/vector-doc-engine.gitOr via GitHub Packages registry:
npm install @dileepwick/vector-doc-engineCreate a .env file in your root directory:
CHROMA_URL=http://localhost:8000GEMINI_API_KEY=your_gemini_api_keyGEMINI_EMBED_MODEL=gemini-embedding-2-previewStart a local ChromaDB instance:
docker run -p 8000:8000 chromadb/chromaView Code Examples
Ingest PDF Document
import{ingestPdfDocument}from"@dileepwick/vector-doc-engine";constresult=awaitingestPdfDocument({filePath: "./data/documents/sem-reg.pdf",});console.log(`Ingested ${result.totalChunks} chunks.`);Perform Vector Search Query
import{queryVectorSearch}from"@dileepwick/vector-doc-engine";constmatches=awaitqueryVectorSearch({query: "What are the main key takeaways?",topK: 3,minSimilarity: 0.35,});matches.forEach((match,idx)=>{console.log(`[${idx+1}] Score: ${match.score.toFixed(4)} | Excerpt: "${match.doc}"`);});View CLI Commands
Build Package
npm run buildIngest Document via CLI
npx ts-node src/ingest.ts sem-reg.pdfQuery Vector Search via CLI
npx ts-node src/ask.ts "What are the key takeaways?"vector-doc-engine/
├── src/
│ ├── index.ts # Public API exports
│ ├── ingest.ts # Document ingestion API
│ ├── ask.ts # Vector search query API
│ ├── chunker.ts # Text chunking utility
│ └── embedder.ts # Gemini API embedder with retries
├── data/documents/ # Default document storage
├── dist/ # Built JavaScript binaries & declarations
├── docs/ # Technical documentation & guides
└── tests/ # Unit test suites