On-Premise AI Infrastructure — Your Data Never Leaves Your Network
GDPR · EU AI Act · Architecture · Quick Start · Full Docs
NetAI Stack SE is built for organizations that cannot afford cloud data leakage. Law firms, SMEs, and compliance-driven businesses run this stack entirely on-premise on Ubuntu 24.04 LTS with Intel Arc GPUs.
- Zero cloud inference — All LLM queries execute locally on your Intel Arc Pro B50
- Zero telemetry — No data leaves your network unless you explicitly configure it
- 100% on-premise — Air-gap capable, no external dependencies at runtime
The stack includes automated PII detection and redaction via Microsoft Presidio:
| Feature | Detail |
|---|---|
| PII-Guard | Intercepts all web search queries before they reach SearXNG |
| Detected Entities | Names, locations, IBANs, phone numbers, emails, passports, German NRP |
| Languages | German (de_core_news_md) + English |
| Redaction | Replaces PII with [REDACTED_NAME], [REDACTED_LOCATION], etc. |
| Audit Logging | All redaction events logged as structured JSON for compliance oversight |
| Compliance Headers | X-PIGuard-Redacted, X-PIGuard-Compliance on every proxied response |
Full transparency documentation and prompt injection protection via Mezzo-Prompt-Guard-v2-Base:
| Feature | Detail |
|---|---|
| Security-Guard | Filters all inference requests for prompt injection before they reach the LLM |
| Safety Model | Mezzo-Prompt-Guard-v2-Base (IQ4_XS, ~450 MB VRAM) on isolated iGPU |
| Detection | Jailbreaks, system prompt extraction, malicious code, PII extraction attempts |
| Fallback | Heuristic regex-based classifier when llama-server is unavailable |
| Human-in-the-Loop | All blocked requests logged with confidence scores and triggered categories |
| Dual-GPU Partitioning | Compliance services offloaded to Alder Lake iGPU to preserve dGPU VRAM for the main LLM |
User → Caddy (:443) → LibreChat → Security-Guard → Cascade LLM ─┬─ small model (low complexity)
├─ large model (high complexity)
└─ reroute (confidence < 0.7)
→ SearXNG via PII-Guard (Presidio redaction)
→ Hermes Agent (Telegram + Dashboard)
→ Hermes WebUI
→ Beszel (monitoring)
→ LightRAG (knowledge graph)
→ SuperTonic TTS / Parakeet STT
See docs/architecture.md for the complete system architecture, hardware partitioning, Caddy routes, API endpoints, and data flow diagrams.
- Ubuntu 24.04 LTS (kernel 6.8+)
- Intel Arc Pro B50 (Battlemage) GPU
- Model files in
models/(Qwen3.6-35B, LFM2.5, Mezzo-Prompt-Guard)
cp .env.example .env
# Edit .env: DOMAIN, ADMIN_EMAIL, ADMIN_PASSWORD, SSL_CERT_PATH, SEARXNG_SECRET,# LIBRECHAT_JWT_SECRET, JWT_REFRESH_SECRET, BESZEL_KEY, BESZEL_TOKEN./setup.sh # GPU drivers, Docker, device detection, model validation
docker compose up -d # Start all services| Endpoint | Description |
|---|---|
https://<DOMAIN>/ | LibreChat — main chat interface |
https://<DOMAIN>/menu/ | Main application menu |
https://<DOMAIN>/search/ | SearXNG — web search (via PII-Guard) |
https://<DOMAIN>/agent/ | Hermes Agent Dashboard |
https://<DOMAIN>/agent-api/ | Hermes Agent API (OpenAI-compatible) |
https://<DOMAIN>/hermes-webui/ | Hermes Web UI |
https://<DOMAIN>/beszel/ | Beszel — system monitoring |
https://<DOMAIN>/knowledge/ | LightRAG — knowledge graph |
https://<DOMAIN>/stack-health/ | Stack Health — container status |
https://<DOMAIN>/pii-guard/ | PII-Guard — compliance dashboard |
https://<DOMAIN>/security-guard/ | Security-Guard — safety dashboard |
https://<DOMAIN>/tts/ | SuperTonic TTS API |
https://<DOMAIN>/speech-stt/ | Parakeet STT API |
https://<DOMAIN>/inference/ | llama.cpp raw API (no guard) |
https://<DOMAIN>/api/v1/ | Public Agents API (LibreChat API key auth) |
bash scripts/setup-beszel.shOpen https://<DOMAIN>/ and register the first user (becomes admin).
pytest tests/ -m "not slow" -v # Fast smoke test (~30s)
pytest tests/ -v # Full suite (~2-3 min)For interactive setup — model selection, GPU detection, PII redaction mode, domain & certificates, benchmarks — use the configuration TUI:
./scripts/netai-setup.shThis provides menus for:
- Hardware Detection — auto-detect GPU, configure SYCL backend
- Model Configuration — switch main/auxiliary LLM profiles, tune parameters
- Privacy & Features — toggle PII redaction (redacted/unredacted), model presets
- Domain & Certificates — set domain, copy Let's Encrypt certs
- System Status — container health, GPU info, disk usage, network checks
- Benchmarks — run GPU performance benchmarks across all loaded models
| Service | Purpose |
|---|---|
| LibreChat | Chat frontend with RAG, MCP, multi-model support |
| Cascade LLM | Routes requests by complexity + confidence between small/large models |
| Security-Guard | Prompt injection protection (Mezzo-Prompt-Guard on iGPU) |
| PII-Guard | GDPR PII redaction (Microsoft Presidio) |
| SearXNG | Privacy-respecting meta search engine |
| Hermes Agent | Telegram bot + dashboard with tool calling |
| Hermes WebUI | Full web interface for Hermes |
| LightRAG | Graph-enhanced RAG with entity extraction |
| Beszel | Lightweight system monitoring |
| SuperTonic TTS | Text-to-Speech (OpenAI-compatible) |
| Parakeet STT | Speech-to-Text (OpenAI-compatible) |
Full API endpoint reference: see docs/architecture.md.
- Uses
ghcr.io/ggml-org/llama.cpp:server-intelimage (SYCL backend) - Kernel 6.8+ required for native Xe/i915 support on Battlemage
ONEAPI_DEVICE_SELECTOR=*:gpupassed to inference containers- SYCL backend only supports discrete Intel Arc GPUs (Xe-HPG+)
- Security-Guard uses the iGPU (
/dev/dri/card0) separately — no dGPU VRAM consumed - No NVIDIA/CUDA dependencies
370+ tests across 27 files. See AGENTS.md for the full test matrix.
pytest tests/ -m "not slow" -v # Smoke test
pytest tests/ -m "pii_guard" -v # GDPR compliance
pytest tests/ -m "security_guard" -v # AI Act compliance
pytest tests/ -m "rag" -v # LightRAG pipeline
pytest tests/ -m "multimodal" -v # Vision workflowsThis deployment configuration is provided as-is for B2B on-premise deployments. Model weights and upstream container images are subject to their respective licenses.