Skip to content
BranchesTags

Repository files navigation

ragctl

Production-ready document processing CLI for RAG applications

Process documents, extract text with advanced OCR, chunk intelligently, and prepare data for RAG systems - all from the command line with ragctl.

VersionPyPIStatusTestsCoverageLicensePRs WelcomePython

What is ragctl?

ragctl is a command-line tool for processing documents into chunks ready for Retrieval-Augmented Generation (RAG) systems. It handles the dirty work of document ingestion, OCR, and intelligent chunking so you can focus on building your RAG application.

Key capabilities:

  • Universal document loading (PDF, DOCX, images, HTML, Markdown, etc.)
  • Advanced OCR with automatic fallback (EasyOCR → PaddleOCR → pytesseract)
  • Intelligent semantic chunking using LangChain
  • Production-ready batch processing with auto-retry
  • Multiple export formats (JSON, JSONL, CSV)
  • Direct ingestion into Qdrant vector store

Features

Universal Document Processing

  • Supported formats: PDF, DOCX, ODT, TXT, HTML, Markdown, Images (JPEG, PNG)
  • Smart OCR cascade:
    1. EasyOCR (best quality, multi-language)
    2. PaddleOCR (fast, good for complex layouts)
    3. pytesseract (fallback, most tolerant)
  • Quality detection: Automatically rejects unreadable documents
  • Multi-language: French, English, German, Spanish, Italian, Portuguese, and more

Intelligent Chunking

  • Semantic chunking: Context-aware text splitting using LangChain RecursiveCharacterTextSplitter
  • Multiple strategies:
    • semantic - Smart splitting by meaning (default)
    • sentence - Split by sentences
    • token - Fixed token-based splitting
  • Configurable: Token limits (50-2000), overlap (0-500), model selection
  • Rich metadata: Source file, chunk index, token count, strategy, timestamps

Production-Ready Batch Processing

  • Automatic retry: Up to 3 attempts with exponential backoff (1s, 2s, 4s...)
  • Interactive error handling:
    • interactive - Prompt user on each error (default)
    • auto-continue - Continue on errors (CI/CD mode)
    • auto-stop - Stop on first error (validation mode)
    • auto-skip - Skip failed files automatically
  • Complete history: Every run saved to ~/.ragctl/history/
  • Retry capability: ragctl retry to rerun failed files only
  • Per-file output: One chunk file per document for better traceability

Flexible Export & Storage

  • Export formats: JSON, JSONL (streaming), CSV (Excel-compatible)
  • Vector store integration: Direct ingestion into Qdrant
  • No database required: Pure file-based export for easy sharing

Configuration System

  • Hierarchical config: CLI flags > Environment variables > YAML file > Defaults
  • Example config: config.example.yml with detailed documentation
  • Easy customization: Override any setting via command line

Quick Start

Installation

From PyPI (Recommended)

# Install from PyPI
pip install ragctl
# Verify installation
ragctl --version

From Source

# Clone repository
git clone git@github.com:datallmhub/ragctl.git
cd ragctl
# Install with pip
pip install -e .# Verify installation
ragctl --version

Basic Usage

# Process a single document
ragctl chunk document.pdf --show
# Process with advanced OCR for scanned documents
ragctl chunk scanned.pdf --advanced-ocr -o chunks.json
# Batch process a folder
ragctl batch ./documents --output ./chunks/
# Preview files before processing (dry-run)
ragctl batch ./documents --dry-run
# Batch with auto-retry for CI/CD
ragctl batch ./documents --output ./chunks/ --auto-continue
# Quiet mode (errors only)
ragctl batch ./documents -q
# Verbose mode (debug info)
ragctl chunk document.pdf -v

Usage Examples

Single Document Processing

# Simple text file
ragctl chunk document.txt --show
# PDF with semantic chunking (default)
ragctl chunk report.pdf -o report_chunks.json
# Scanned image with OCR
ragctl chunk contract.jpeg --advanced-ocr --show
# Custom chunking parameters
ragctl chunk document.pdf \
--strategy semantic \
--max-tokens 500 \
--overlap 100 \
-o output.jsonl

Batch Processing

# Process all files in a directory
ragctl batch ./documents --output ./chunks/
# Preview files without processing (dry-run)
ragctl batch ./documents --dry-run
# Output:# Would process 15 files:# ├── report.pdf (2.3 MB)# ├── contract.docx (156 KB)# └── notes.txt (12 KB)# Total: 15 files, 45.2 MB# Process only PDFs recursively
ragctl batch ./documents \
--pattern "*.pdf" \
--recursive \
--output ./chunks/
# CI/CD mode - continue on errors (quiet mode)
ragctl batch ./documents \
--output ./chunks/ \
--auto-continue \
--quiet \
--save-history
# Verbose mode for debugging
ragctl batch ./documents -v --output ./chunks/
# Per-file output (default):# chunks/# ├── doc1_chunks.jsonl (25 chunks)# ├── doc2_chunks.jsonl (42 chunks)# └── doc3_chunks.jsonl (18 chunks)# Single-file output (all chunks combined):
ragctl batch ./documents \
--output ./all_chunks.jsonl \
--single-file

Retry Failed Files

# Show last failed run
ragctl retry --show
# Retry all failed files from last run
ragctl retry
# Retry specific run by ID
ragctl retry run_20251028_133403

Vector Store Integration

# Ingest chunks into Qdrant
ragctl ingest chunks.jsonl \
--collection my-docs \
--url http://localhost:6333
# Get system info
ragctl info

Evaluate Chunking Quality

# Evaluate chunking strategy
ragctl eval document.pdf \
--strategies semantic sentence token \
--metrics coverage overlap coherence
# Compare strategies with visualization
ragctl eval document.pdf --compare --output eval_results.json

Documentation

DocumentDescription
Getting StartedInstallation and first steps
CLI GuideComplete command reference
SecuritySecurity features and best practices
Full DocumentationComplete documentation index

Configuration

Create ~/.ragctl/config.yml or use CLI flags:

# OCR settingsocr:
use_advanced_ocr: falseenable_fallback: true# Chunking settingschunking:
strategy: semanticmax_tokens: 400overlap: 50# Output settingsoutput:
format: jsonlinclude_metadata: truepretty_print: true

Configuration hierarchy: CLI flags > Environment variables > YAML config > Defaults

Testing

# Run all tests
make test# Run CLI tests
make test-cli
# Quick validation
ragctl --version
ragctl chunk tests/data/sample.txt --show

Test Coverage: 496 tests, 41% coverage

Performance

Processing Speed

  • Text documents: ~100-200 docs/minute
  • PDFs with OCR: ~5-10 docs/minute (depends on page count)
  • Batch processing: Parallel-ready with retry mechanism

Quality Metrics

  • OCR accuracy: 95%+ with EasyOCR on clear scans
  • Chunk quality: 90% readability threshold enforced
  • Semantic coherence: LangChain's RecursiveCharacterTextSplitter optimized for context

CLI Commands

CommandDescription
ragctl chunkProcess a single document
ragctl batchBatch process multiple files
ragctl retryRetry failed files from history
ragctl ingestIngest chunks into Qdrant
ragctl evalEvaluate chunking quality
ragctl infoSystem information

Run ragctl COMMAND --help for detailed options.

Troubleshooting

Common Issues

NumPy incompatibility

# For OCR support, use NumPy 1.x
pip install "numpy<2.0"

Missing system dependencies

# Ubuntu/Debian
sudo apt-get install tesseract-ocr poppler-utils
# macOS
brew install tesseract poppler

"Document unreadable" errors

  • Try lowering quality threshold: --ocr-threshold 0.2
  • Use advanced OCR: --advanced-ocr
  • Check document is not corrupted

Import errors

# Reinstall dependencies
pip install -e .

More help: Getting Started Guide

Development

# Install dev dependencies
make install-dev
# Format code
make format
# Run linters
make lint
# Install pre-commit hooks
make pre-commit-install
# Run all CI checks
make ci-all

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Support

Acknowledgments

Built with:

Version: 0.1.5 | Status: Beta | License: MIT

About

A powerful CLI tool to manage, test, and optimize RAG pipelines. Streamline your Retrieval-Augmented Generation workflows from terminal.

pypi.org/project/ragctl/0.1.5

Topics

chatbotllmocrpythonragvector-database

Resources

Readme
MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

23 stars

Watchers

1 watching

Forks

6 forks
Report repository

Releases

Contributors

Languages