🔄 OmniRAG

Developed by Kristopher Clark · Solutions Architect at ZEDEDA

A high-performance, local-first AI Retrieval-Augmented Generation (RAG) tool built entirely in Rust with Redis for state management. OmniRAG synchronizes a local directory of files with an Open WebUI knowledge base — hashing files via SHA-256, tracking state atomically in Redis, dynamically prepending configurable context metadata, and interfacing with the Open WebUI REST API. Deployed as a 1:1 Pod architecture (one Rust container + one Redis container per Knowledge Base), OmniRAG is designed for edge computing environments where data sovereignty, offline capability, and low-latency AI inference are critical.

---

⚡ Quick Start

git clone https://github.com/krisclarkdev/omnirag.git &&cd omnirag
# ⚠️ Edit docker-compose.yml before starting:# 1. Change ./alpha-docs to the path to your documents folder# 2. Change ./redis-alpha to where you want Redis data persisted (or leave as-is)
docker compose pull
docker compose up -d
# Open http://localhost:3000 → Configure → Trigger Sync

---

📑 Table of Contents

• ✨ Features
• 🚀 Quick Start
• 🖥️ Production Deployment
• 🏗️ Architecture: The 1:1 Pod
• 🧩 Best Practices: Chunking
• 🤖 REST API (Automation)
• 💻 CLI
• 🌐 Web UI
• 🐳 Docker
• 📐 Hardware Requirements & Scaling
• 🧪 Testing
• 🔄 CI/CD
• 📁 Source Files
• 🧪 End-to-End Test Results
• 📄 License

---

✨ Features

• Three-phase sync — Reconciles state, ingests new/updated files, and cleans up orphans
• Pre-sync reconciliation — Queries Open WebUI before syncing to heal state and prevent duplicates; skips ghost files
• SHA-256 change detection — Only re-uploads files whose contents actually changed
• Context-dirty flag — Context changes via Web UI or CLI trigger re-upload on next sync without file modification
• Configurable context header — Set a custom label per deployment (e.g., "Solutions Architect Context", "Project Lore")
• Dynamic KB picker — Fetches available Knowledge Bases from Open WebUI and presents a dropdown selector
• Convert to Markdown — Optionally wraps text files in language-specific code fences and uploads as .md for better LLM chunking
• Configurable concurrency — Tune max simultaneous uploads (default 5, 0 = unlimited)
• Context Manager dialog — Searchable, paginated modal for managing per-file context strings
• Redis Functions (Lua) — Atomic server-side operations for state management
• Redis AOF persistence — Native append-only file for durable state across container restarts
• File filtering — Extension whitelist, OS file ignore list, and .ragignore support
• API retry with backoff — Exponential backoff (3 attempts) on all Open WebUI API calls
• Binary-safe context injection — Prepends metadata to text files only; binary files (PDF) are uploaded raw to prevent corruption
• JSON REST API — Automation-ready endpoints for cron, n8n, webhooks, and external triggers
• Swagger / OpenAPI — Interactive API docs at /docs with auto-generated OpenAPI 3.0 spec
• Embedded Web UI — HTMX + Tailwind glassmorphism dark-mode dashboard with contextual help icons
• CLI tools — set-context, get-context, sync, serve
• 1:1 Pod architecture — Each Knowledge Base gets its own isolated Rust + Redis container pair
• CI/CD ready — GitHub Actions workflow for automated Docker builds and GHCR publishing

---

🚀 Quick Start

Deploy with Docker Compose

# 1. Clone
git clone https://github.com/krisclarkdev/omnirag.git &&cd omnirag
# 2. ⚠️ Edit docker-compose.yml:# - Change ./alpha-docs to the path to your documents folder# - Change ./redis-alpha to where you want Redis data persisted (or leave as-is)# 3. Pull and deploy
docker compose pull
docker compose up -d
# 4. Configure via http://localhost:3000# 5. Click "▶ Trigger Sync" in the UI

Pull from GitHub Container Registry (GHCR)

Once the GitHub Actions workflow runs on your main branch, the image is published to GHCR:

services:
redis:
image: redis:7-alpinecommand: redis-server --appendonly yesvolumes:
- ./redis-data:/datarestart: unless-stoppedomnirag:
image: ghcr.io/krisclarkdev/omnirag:mainports:
- "3000:3000"volumes:
- ./my-documents:/ragenvironment:
- REDIS_URL=redis://redis:6379/0depends_on:
- redisrestart: unless-stopped

---

🖥️ Production Deployment

Once the GitHub Actions workflow completes on main, the Docker image is published to GHCR. Deploy to your server with:

# 1. SSH into your production server
ssh user@your-server
# 2. Create working directory
mkdir -p ~/omnirag &&cd~/omnirag
# 3. Create a docker-compose.yml that pulls from GHCR
cat > docker-compose.yml << 'EOF'services: redis: image: redis:7-alpine command: redis-server --appendonly yes volumes: - ./redis-data:/data restart: unless-stopped omnirag: image: ghcr.io/krisclarkdev/omnirag:main ports: - "3000:3000" volumes: - /path/to/your/documents:/rag:ro environment: - REDIS_URL=redis://redis:6379/0 depends_on: - redis restart: unless-stoppedEOF# 4. ⚠️ Edit docker-compose.yml — replace /path/to/your/documents# with the absolute path to your local documents folder, e.g.:# - /home/kris/knowledge-docs:/rag:ro# 5. Pull and start
docker compose pull
docker compose up -d
# 6. Verify
docker compose ps
docker compose logs -f omnirag
# 7. Open http://your-server:3000 → Configure → Trigger Sync

Updating to Latest Version

cd~/omnirag
docker compose pull
docker compose up -d

Optional: Reverse Proxy (Nginx)

server{listen443ssl;server_name omnirag.yourdomain.com;location / {proxy_passhttp://127.0.0.1:3000;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;}}

---

🏗️ Architecture: The 1:1 Pod

Kristopher Clark designed OmniRAG around a strict 1:1 Pod architecture, leveraging Rust for memory-safe, zero-cost-abstraction performance and Redis with AOF persistence for durable, atomic state management. The system stores its global configuration (config:global) and all per-file sync state in Redis. Because of this, each directory/Knowledge Base pair MUST have its own dedicated Redis container. Sharing a Redis container across multiple OmniRAG instances will result in config collisions and corrupted vector states.

The system enforces this strict 1:1 Pod model:

┌─── Pod (docker-compose) ─────────────────────────────────┐
│ │
│ ┌────────────────┐ ┌────────────────────────────┐ │
│ │ redis:7-alpine │ ◄──► │ omnirag (Rust) │ │
│ │ :6379 (AOF) │ │ │ │
│ └────────────────┘ │ ┌────────┐ ┌──────────┐ │ │
│ │ │ │ Axum │ │ Sync │ │ │
│ ┌──────▼──────┐ │ │ Web UI │ │ Engine │ │ │
│ │ ./redis-*/ │ │ │ + REST │ │ │ │ │
│ │ appendonly │ │ └────────┘ └──────────┘ │ │
│ │ .aof │ ▲ │ │ │
│ └─────────────┘ │ ▼ │ │
│ /rag ◄───────────────┘ Open WebUI API │ │
└──────────────────────────────────────────────────────────┘

Multi-Knowledge Base Deployment

To sync multiple directories to separate Knowledge Bases, run multiple isolated Pods in a single docker-compose.yml:

services:
# ── Pod 1: first project ─────────────────────────────redis-alpha:
image: redis:7-alpinecommand: redis-server --appendonly yesvolumes:
- ./redis-alpha:/dataomnirag-alpha:
build: .ports:
- "3000:3000"volumes:
- ./alpha-docs:/ragenvironment:
- REDIS_URL=redis://redis-alpha:6379/0depends_on:
- redis-alpha# ── Pod 2: second project ────────────────────────────redis-beta:
image: redis:7-alpinecommand: redis-server --appendonly yesvolumes:
- ./redis-beta:/dataomnirag-beta:
build: .ports:
- "3001:3000"volumes:
- ./beta-docs:/ragenvironment:
- REDIS_URL=redis://redis-beta:6379/0depends_on:
- redis-beta

Each Pod gets its own Redis data directory, port, and document volume. They are completely isolated — no shared state.

Data Flow

1. Reconcile — Query Open WebUI KB file list, heal Redis state to prevent duplicates (skips ghost files not on disk)
2. Walk/rag directory recursively (filtered by extension whitelist + .ragignore)
3. Hash each file's contents (SHA-256)
4. Compare against stored hash in Redis
5. Upload new/changed files via Open WebUI API with retry (context prepended for text files only; binary files uploaded raw)
6. Clean up orphaned files (deleted from disk → deleted from Open WebUI)

Redis Data Model

File State — Key: <sha256(absolute_path)>_<filename>

Field	Description
absolute_path	Full filesystem path
content_hash	SHA-256 of file contents
openwebui_file_id	UUID from Open WebUI
context_text	User-defined context string
context_dirty	"true" if context was changed and needs re-upload

Config — Key: config:global

Field	Default
target_directory	/rag
openwebui_url	(set via UI)
openwebui_token	(set via UI)
openwebui_knowledge_id	(set via UI)
redis_url	redis://127.0.0.1:6379/0
context_header_label	File Context

Redis Persistence

Redis runs with AOF (Append-Only File) enabled (--appendonly yes). The AOF log is written to /data/appendonly.aof inside the Redis container, which is bind-mounted to ./redis-<name> on the host. This ensures all state survives container restarts natively — no external backup layer needed.

Redis Functions (Lua)

Five server-side functions loaded via FUNCTION LOAD REPLACE at startup:

Function	Purpose
get_formatted_context	Returns context as Markdown blockquote
check_file_exists	Key existence check
verify_file_hash	Content hash comparison
upsert_sync_state	Atomic field update (preserves context_text)
get_cleanup_batch	SCAN-based batch retrieval for orphan cleanup

Three-Phase Sync

Phase 0 — Reconciliation:

Queries the Open WebUI API for all files in the target knowledge base. Matches each remote file against local Redis state by filename. If a file exists remotely but the UUID isn't tracked locally (e.g., crash between upload and state save), it heals the Redis entry to prevent duplicate uploads.

Ghost file detection: If a file exists in Open WebUI but the corresponding local file has been deleted from disk while the container was offline, the reconciliation phase skips healing ([GHOST]). Phase 2 will naturally detect these as orphans and clean them up from Open WebUI, avoiding unnecessary state churn.

Phase 1 — Ingestion (5 concurrent uploads via semaphore):

Condition	Action
File not in Redis	[NEW] Upload → poll status → attach to KB → save state
File in Redis, hash matches, context clean	[SKIP] No action
File in Redis, hash matches, context dirty	[UPDATE] Re-upload with new context (set via Web UI or CLI)
File in Redis, hash differs	[UPDATE] Delete old → upload new → save state

Phase 2 — Orphan Cleanup:

Condition	Action
Redis key but file missing from disk	[ORPHAN:missing] Delete from Open WebUI → remove Redis key
Redis key but file now matches .ragignore / OS ignore / disallowed extension	[ORPHAN:filtered] Delete from Open WebUI → remove Redis key

File Filtering

Files are filtered at three levels before processing:

Filter	Behavior
Hidden/OS files	.DS_Store, Thumbs.db, desktop.ini, .gitkeep always skipped
Extension whitelist	Only processes: .md, .txt, .pdf, .csv, .json, .yaml, .yml, .toml, .xml, .html, .htm, .rst, .log, .cfg, .ini, .conf, .py, .rs, .go, .js, .ts, .sh, .bat, .ps1
.ragignore	Place a .ragignore file in the root of /rag with one pattern per line (supports * wildcards, directory names, and exact paths). Lines starting with # are comments.

Example .ragignore:

# Skip the drafts folder
drafts
# Skip all log files in archives
archives/*.log
# Skip a specific file
notes/scratch.txt

Context Injection (Binary-Safe)

Context strings set via set-context are prepended to files as a Markdown header during upload. This happens in-memory only — local files are never modified.

File Type	Behavior
Text (.md, .txt, .csv, .json, .yaml, .py, .rs, etc.)	Context prepended as Markdown blockquote
Binary (.pdf, and any non-text extension)	Uploaded raw — no context injection to prevent file corruption

API Resiliency

All Open WebUI API calls use exponential backoff retry (3 attempts: 500ms → 1s → 2s):

• 5xx / network errors → retry
• 429 (rate limit) → retry
• 4xx client errors → fail immediately (no retry)
• Failed file → logged, skipped, continues sync (retried on next trigger)

Open WebUI API Integration

Operation	Endpoint	When
List KB files	GET /api/v1/knowledge/{kb}	Phase 0 reconciliation
Upload	POST /api/v1/files/	New or updated file
Poll status	GET /api/v1/files/{id}/process/status	After upload
Attach to KB	POST /api/v1/knowledge/{kb}/file/add	After processing complete
Delete	DELETE /api/v1/files/{id}	Update (old version) or orphan

---

🧩 Best Practices: Chunking

The context string prepended by OmniRAG becomes part of the document text sent to Open WebUI for vector embedding. To ensure the context stays attached to the first logical section during chunking:

• Chunk size: Set to ≥1500 tokens in your Open WebUI settings. This ensures the context blockquote and the first section of the document land in the same vector chunk.
• Chunk overlap: Set to ≥150 tokens. This creates redundancy at chunk boundaries, so context fragments aren't lost if a split happens nearby.
• No horizontal rule: The context format deliberately omits --- (Markdown horizontal rule) because modern text splitters (LangChain, LlamaIndex) treat --- as a hard chunk boundary, which would orphan the context from the document text.

Tip: If you're using Open WebUI's default settings, 1500 chunk size with 150 overlap is a reliable starting point for context-injected documents.

---

🤖 REST API (Automation)

OmniRAG exposes a JSON REST API under /api/v1/ designed for cron jobs, n8n workflows, webhooks, and any external automation. No authentication is required — secure access via network policy or reverse proxy.

Endpoints

Endpoint	Method	Description
/api/v1/health	GET	Liveness probe — returns service name and version
/api/v1/sync	POST	Trigger a sync — returns 202 Accepted or 409 Conflict
/api/v1/sync/status	GET	Poll current sync status as JSON
/api/v1/openapi.json	GET	OpenAPI 3.0 specification (JSON)

GET /api/v1/health

curl http://localhost:3000/api/v1/health

{
"status": "ok",
"service": "omnirag",
"version": "0.1.0"
}

POST /api/v1/sync

curl -X POST http://localhost:3000/api/v1/sync

202 Accepted — sync started:

{
"status": "triggered",
"message": "Sync started"
}

409 Conflict — sync already in progress:

{
"status": "already_running",
"message": "A sync operation is already in progress"
}

GET /api/v1/sync/status

curl http://localhost:3000/api/v1/sync/status

{
"status": "completed",
"detail": "Completed successfully"
}

Status values: idle, running, completed, error, unknown.

Automation Examples

Cron (every 15 minutes):

*/15****curl -s -X POST http://localhost:3000/api/v1/sync > /dev/null

n8n HTTP Request Node:

• Method: POST
• URL: http://omnirag:3000/api/v1/sync
• Authentication: None
• Response Format: JSON

Docker Healthcheck:

services:
omnirag:
image: ghcr.io/<owner>/omnirag:mainhealthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3000/api/v1/health"]interval: 30stimeout: 5sretries: 3

Bash Trigger + Wait:

#!/bin/bash# Trigger sync and poll until complete
curl -s -X POST http://localhost:3000/api/v1/sync
whiletrue;do
STATUS=$(curl -s http://localhost:3000/api/v1/sync/status | jq -r .status)echo"Status: $STATUS"
[ "$STATUS"="completed" ] || [ "$STATUS"="error" ] &&break
sleep 5
done

---

💻 CLI

# Start the web server
omnirag serve
# Run sync manually (all 3 phases)
omnirag sync
# Set context for a file (prepended on upload)
omnirag set-context /rag/doc.md "Architecture reference for EVE-OS edge deployments."# Get formatted context
omnirag get-context /rag/doc.md
# > **Solutions Architect Context:**# > Architecture reference for EVE-OS edge deployments.# ---

CLI Flags

Flag	Env Var	Default	Description
--redis-url	REDIS_URL	redis://127.0.0.1:6379/0	Redis connection

---

🌐 Web UI

Glassmorphism dark-mode dashboard served at http://localhost:3000:

• Configuration panel — Set Open WebUI URL, API token, and Knowledge Base ID
• Sync control — Trigger sync and monitor status in real-time (polls every 3s)
• HTMX-powered — Dynamic updates without page reloads

HTMX Endpoints (Web UI)

Endpoint	Method	Purpose
/	GET	Full page
/api/config	GET	Config form partial
/api/config	POST	Save config
/api/sync	POST	Trigger sync (returns HTML)
/api/sync/status	GET	Poll status (returns HTML)
/api/contexts	GET	Context Manager file list
/api/contexts	POST	Update file context_text
/docs	GET	Swagger UI (interactive API docs)
/api/v1/openapi.json	GET	OpenAPI 3.0 specification

---

🐳 Docker

Dockerfile

Multi-stage build:

1. Builder (rust:1.85-bookworm) — Compiles release binary
2. Runtime (debian:bookworm-slim) — Installs redis-server, gosu, ca-certificates

Boot Sequence

entrypoint.sh
├── chown -R redis:redis /data
├── gosu redis redis-server --appendonly yes --dir /data --daemonize yes
├── Wait for Redis PONG (readiness check)
└── exec omnirag serve
├── Connect to Redis
├── Load Lua functions
└── Start Axum on :3000

Volume permissions: The entrypoint runs initially as root to chown the /data bind-mount, then uses gosu to drop privileges and start Redis as the redis user. This prevents "permission denied" errors when host directories have strict permissions.

docker-compose.yml

services:
omnirag:
image: ghcr.io/krisclarkdev/omnirag:mainextra_hosts:
- "host.docker.internal:host-gateway"ports:
- "3000:3000"volumes:
- ./my-documents:/rag:ro # Your files (change path as needed)environment:
- REDIS_URL=redis://redis:6379/0
- RUST_LOG=info,omnirag=debugrestart: unless-stopped

Networking: Connecting to Open WebUI

OmniRAG runs inside a Docker container and needs to reach your Open WebUI API. This requires special configuration when both services run on the same host.

Same-Host Deployment (Open WebUI on the same machine)

From inside a Docker container, localhost refers to the container itself — not the host machine. To reach services on the Docker host:

1. Add extra_hosts to your docker-compose.yml (required on Linux):

omnirag-alpha:
image: ghcr.io/krisclarkdev/omnirag:mainextra_hosts:
- "host.docker.internal:host-gateway"

2. Set the Open WebUI URL in the OmniRAG config to:

http://host.docker.internal:<PORT>

where <PORT> is the port Open WebUI listens on (e.g., 3000, 8080).

Tip: To find which port Open WebUI uses, run curl -s http://localhost:<PORT>/api/v1/knowledge/ from the host — the port that returns JSON is the correct one.

Remote Open WebUI (different machine)

If Open WebUI runs on a different server, use its hostname or IP directly:

https://openwebui.yourdomain.com

No extra_hosts needed in this case.

Port Customization

If port 3000 is already in use on your host (e.g., by Open WebUI), change the left side of the port mapping:

ports:
- "3001:3000"# Access OmniRAG at http://your-server:3001

Multi-Container Deployment (One Container per Collection)

OmniRAG uses a single global config per container — each instance syncs one local directory to one Open WebUI collection (Knowledge Base). This is by design: to sync multiple directories to different collections, run one Pod per collection:

services:
# ── Pod 1 ────────────────────────────────────────────redis-alpha:
image: redis:7-alpinecommand: redis-server --appendonly yesvolumes:
- ./redis-alpha:/dataomnirag-alpha:
image: ghcr.io/krisclarkdev/omnirag:mainextra_hosts:
- "host.docker.internal:host-gateway"ports:
- "3000:3000"volumes:
- ./alpha-docs:/rag:roenvironment:
- REDIS_URL=redis://redis-alpha:6379/0
- RUST_LOG=info,omnirag=debugdepends_on:
- redis-alpharestart: unless-stopped# ── Pod 2 ────────────────────────────────────────────redis-beta:
image: redis:7-alpinecommand: redis-server --appendonly yesvolumes:
- ./redis-beta:/dataomnirag-beta:
image: ghcr.io/krisclarkdev/omnirag:mainextra_hosts:
- "host.docker.internal:host-gateway"ports:
- "3001:3000"volumes:
- ./beta-docs:/rag:roenvironment:
- REDIS_URL=redis://redis-beta:6379/0
- RUST_LOG=info,omnirag=debugdepends_on:
- redis-betarestart: unless-stopped

Each Pod gets its own Redis data volume, independent configuration, and a separate Web UI port. Configure each via its own UI (localhost:3000, localhost:3001, etc.) with the target collection's Knowledge Base ID. Pods are fully isolated — they share nothing and can be started, stopped, or scaled independently.

Logging Configuration

OmniRAG uses the RUST_LOG environment variable for log verbosity:

Level	What you see
RUST_LOG=info	Sync phases, file actions, errors
RUST_LOG=info,omnirag=debug	All of the above + API calls, filter decisions
RUST_LOG=warn	Only warnings and errors

# View container logs
docker logs -f omnirag
# Override at runtime
docker run -e RUST_LOG=debug ...

---

📐 Hardware Requirements & Scaling

The following T-shirt sizing guide provides realistic estimates based on OmniRAG's resource profile:

• Redis RAM: ~300 bytes per tracked file (path + SHA-256 hash + UUID + context string)
• App RAM: Up to 5 concurrent file buffers in memory (text files only; binary files passed through)
• CPU: SHA-256 hash computation for every file during sync
• Storage: Redis AOF file grows proportionally to total state size

Tier	Files	Container RAM	CPU	AOF Storage
XS (Tiny)	< 1,000	128 MB	0.5 vCPU	< 5 MB
S (Small)	1K – 5K	256 MB	1 vCPU	5 – 25 MB
M (Medium)	5K – 20K	512 MB	1 vCPU	25 – 100 MB
L (Large)	20K – 100K	1 GB	2 vCPU	100 – 500 MB
XL (Enterprise)	100K+	2 GB+	4 vCPU	500 MB+

XL Tier Bottleneck Analysis

At the Enterprise tier (100K+ files), the primary bottleneck is Phase 1 directory walk + SHA-256 hashing, not memory. Redis comfortably holds 100K entries in ~30 MB of RAM. However:

• SHA-256 hashing of 100K+ files on each sync cycle becomes CPU-bound. Each hash requires a full file read, so I/O throughput of the /rag volume also matters.
• Open WebUI API rate limits may throttle uploads during initial ingestion. The 5-concurrent-upload semaphore prevents overwhelming the API, but the first-ever sync of 100K files will take hours.
• Redis AOF rewrite can momentarily spike memory to 2× during background BGREWRITEAOF. The 2 GB recommendation accounts for this.

Recommendation for XL: Run on dedicated infrastructure with SSD-backed /rag volume. Consider increasing the semaphore limit (currently 5) for faster initial ingestion if your Open WebUI instance can handle it.

---

🧪 Testing

Test Suite

OmniRAG includes a comprehensive test suite covering all core components:

# Run all tests
cargo test# Run with output
cargo test -- --nocapture

Test Module	Tests	Coverage	Location
API Client	10	Upload, delete, KB operations, exponential backoff retry (5xx/429/4xx), poll status	src/api.rs (inline)
Web REST API	9	Health endpoint, sync trigger (202/409), status phases (idle/running/completed/error), method validation	tests/web_api_tests.rs
Sync Engine	15	Extension whitelist, OS file filtering, .ragignore parsing, binary-safe context injection, full pipeline	tests/sync_tests.rs
Hashing & Config	11	SHA-256 identity, deterministic keys, empty file hash, config serialization roundtrip	tests/hashing_config_tests.rs

API Retry Test Matrix

Scenario	Expected	Verified
5xx → 5xx → 200	Retry twice, succeed	✅
429 → 429 → 200	Retry twice, succeed	✅
400 (client error)	Fail immediately, no retry	✅
500 → 500 → 500	Fail after 3 attempts	✅

REST API Test Matrix

Scenario	Expected	Verified
GET /api/v1/health	200 + service info	✅
POST /api/v1/sync (idle)	202 Accepted	✅
POST /api/v1/sync (running)	409 Conflict	✅
GET /api/v1/sync/status (all phases)	Correct phase string	✅
GET /api/v1/sync (wrong method)	405 Method Not Allowed	✅

---

🔄 CI/CD

GitHub Actions workflow at .github/workflows/docker-publish.yml:

Trigger	Action
Push to main	Build + push to GHCR
Pull Request	Build only (verify compilation)
Release tag (v*)	Build + push with semver tags

The workflow uses Docker Buildx with GitHub Actions cache for fast rebuilds.

---

📁 Source Files

File	Purpose
src/sync.rs	Three-phase sync, file filtering, .ragignore, binary-safe context
src/api.rs	Open WebUI API client with retry + inline tests
src/web.rs	Axum routes (HTMX + JSON REST API), HTML rendering
src/main.rs	CLI parsing, Redis bootstrap
src/redis_client.rs	Lua function loading + FCALL wrappers
src/lua/rag_helpers.lua	5 Redis Functions
src/config.rs	AppConfig struct + Redis load/save
src/hashing.rs	SHA-256 key/content hashing

Dependencies

tokio · axum · clap · redis · reqwest · walkdir · sha2 · serde · serde_json · hex · dotenvy · tower-http · tracing · tracing-subscriber

Dev:wiremock · tempfile · tower · http-body-util · serde_urlencoded

---

🧪 End-to-End Test Results

Test Environment

Component	Value
Deployment	Docker Compose 1:1 Pod (omnirag-alpha + redis-alpha)
Test Directory	alpha-docs/ mounted at /rag
Open WebUI	Local Open WebUI instance
Target KB	"Test" (auto-discovered via Fetch KBs)

Test Files

File	Type	Expected Behavior
architecture.md	Markdown	Upload as-is
data_processor.py	Python	Convert → data_processor.md (code fence)
test_config.json	JSON	Convert → test_config.md (code fence)
release_notes.txt	Plain text	Upload as-is (.txt not converted)
secret_notes.txt	Plain text	Excluded by .ragignore
.ragignore	Config	Patterns: drafts, *.tmp, secret_notes.txt

Results

Test Case	Status	Notes
KB Dropdown Picker	✅ PASS	Fetched 3 KBs from API, "Test" selected via dropdown
Help Icon Modals	✅ PASS	All 6 fields have working ? icons with dialog content
Config Save/Load	✅ PASS	All fields persisted to Redis and restored on reload
Convert to Markdown	✅ PASS	.py → .md with Python fence, .json → .md with JSON fence
Markdown Passthrough	✅ PASS	.md and .txt files uploaded without conversion
Binary Passthrough	✅ PASS	PDFs uploaded without context injection
.ragignore Filtering	✅ PASS	secret_notes.txt excluded, 3 patterns loaded
Phase 1 Ingestion	✅ PASS	4 files uploaded as [NEW]
Phase 2 Cleanup	✅ PASS	No orphans found
Open WebUI Verification	✅ PASS	4 files visible in "Test" KB with correct names/sizes
Swagger Docs Link	✅ PASS	Header link navigates to /docs
Footer Credits	✅ PASS	GitHub link + "Built by Kristopher Clark"
Max Concurrent Uploads	✅ PASS	Configurable via UI, default 5

Docker Logs (Abbreviated)

omnirag-alpha-1 | Loaded 3 patterns from .ragignore
omnirag-alpha-1 | Found 4 files to process (skipped: 0 unsupported ext, 1 ragignored)
omnirag-alpha-1 | [NEW] data_processor.py
omnirag-alpha-1 | [NEW] release_notes.txt
omnirag-alpha-1 | [NEW] test_config.json
omnirag-alpha-1 | [NEW] architecture.md
omnirag-alpha-1 | Uploaded 'data_processor.md' → file_id=...
omnirag-alpha-1 | Sync complete.

---

👤 About the Author

Kristopher Clark is a Solutions Architect at ZEDEDA, specializing in edge computing, EVE-OS, Kubernetes, and AI at the edge. He built OmniRAG with Rust and Redis to solve a real-world need: enabling private, high-performance retrieval-augmented generation entirely on local infrastructure — no cloud API dependencies, no data leaving the premises.

Platform	Link
🌐 Website	kristopherclark.com
🐙 GitHub	krisclarkdev
🌍 Wikidata	Q138685504

---

📄 License

Apache License 2.0 — see LICENSE for details.