🚀 Markus Engine — v3.0.0

A blazing fast, memory-safe, pure Rust AI Model Manager & Inference Engine.

Zero llama.cpp dependencies. Fully standalone. Built on HuggingFace's candle.

---

📑 Table of Contents

1. What is Markus?
2. Why Pure Rust?
3. System Architecture
4. Installation
5. Usage & Commands
6. Detailed Inference Flow
7. Configuration
8. Roadmap & Future Vision

---

🧠 What is Markus?

Markus is an advanced, independent Large Language Model (LLM) manager designed to run quantized GGUF models directly on your local CPU/GPU hardware. By dropping heavy C/C++ build pipelines and Python wrappers, Markus operates as a single ~15MB binary that is extremely fast, predictable, and memory safe.

Core Features

• Native GGUF Parsing: Zero-copy extraction of model architectures, weights, and tokenizers.
• AVX/SIMD Optimized: Automatically utilizes CPU extensions (AVX-512) for massive performance gains.
• Dynamic Chat Templating: Automatically detects if a model requires LLaMA-3 (<|start_header_id|>), Mistral ([INST]), or ChatML formats to ensure perfect output coherency.
• OpenAI Compatible API: Drop-in replacement for OpenAI endpoints via the axum HTTP server.

---

🦀 Why Pure Rust?

Historically, LLM execution pipelines have relied heavily on llama.cpp wrapped in Python bindings. While performant, this creates massive dependency headaches—users need GCC versions, CMake, build-essential, and gigabytes of compiled binaries.

Markus v3 completely eliminates this.

• Memory Safety: Rust’s borrow checker guarantees no segfaults or memory leaks.
• Microscopic Footprint: No giant shared libraries.
• Portability: If it compiles, it runs. No missing .so or .dll files.

---

🏗️ System Architecture

Markus is divided into a highly modular Cargo workspace. Here is a high-level overview of how the internal components interact:

graph TD
subgraph Interfaces
A[Terminal UI] --> |Key Events| C
B[CLI Commands] --> |Args| C
W[Web / API Clients] --> |HTTP requests| S
end
subgraph Markus Engine
C[markus-cli]
S[markus-server<br/>Axum HTTP]
T[markus-tui<br/>Ratatui]
C --> T
C --> S
C --> |Init| Core[markus-core]
S --> |Prompt| Core
T --> |Chat Stream| Core
end
subgraph Inference Backend
Core --> |GGUF Parsing| F[Zero-Copy Loader]
Core --> |Forward Pass| ML[Candle ML Framework]
ML --> |Threadpool| R[Rayon Parallel Execution]
R --> Hardware[CPU AVX-512 / GPU]
end

Loading

---

⚡ Installation

To install Markus globally so you can run it from any directory, just copy and paste the one-liner for your OS. The installer will clone the repo, compile with maximum hardware optimizations (target-cpu=native), and place the binary in your PATH.

Linux & macOS

curl -sSL https://raw.githubusercontent.com/Precise-Goals/markus/main/install.sh | bash

Windows (PowerShell)

irm https://raw.githubusercontent.com/Precise-Goals/markus/main/install.ps1 | iex

(Note: You will need Rust and git installed).

---

🕹️ Usage & Commands

Once installed, you can simply type markus from any directory on your machine.

1. Launch the Interactive TUI (Default)

markus

Navigating the TUI: Use arrow keys (or j/k) to move, Enter to select, and q to quit. The TUI includes isolated widgets for chatting, browsing local models, and monitoring system RAM/CPU.

2. Command Line Mode (CLI)

You can bypass the TUI and execute commands directly:

• List local models:markus list
• Chat with a model in terminal:markus run 1 or markus run qwen3
• Download a model:markus pull llama3.2
• Start OpenAI API Server:markus serve 1 --port 8080
• Show system hardware status:markus status
• Clear system RAM / drop caches:markus freemem

---

🔄 Detailed Inference Flow

When you send a prompt to Markus, it uses asynchronous multi-threading to tokenize, evaluate, and stream the response back in real-time without blocking the user interface.

sequenceDiagram
participant User
participant Pipeline as markus-core
participant Tokenizer as MarkusTokenizer
participant Candle as Candle Backend
User->>Pipeline: Send Prompt (e.g. "Explain AI")
activate Pipeline
Pipeline->>Tokenizer: Request encoding
Note right of Tokenizer: Applies architecture-specific<br/>Chat Template (LLaMA-3, Mistral)
Tokenizer-->>Pipeline: Token IDs (Vec<u32>)
Pipeline->>Candle: Initialize Tensor & KV Cache
loop Token Generation
Pipeline->>Candle: Forward Pass (Tokens)
Candle-->>Pipeline: Raw Logits
Pipeline->>Pipeline: Apply Temp, Top-K & Repeat Penalty
Pipeline->>Tokenizer: Decode highest prob Token ID
Tokenizer-->>User: Stream Token String Fragment
end
deactivate Pipeline

Loading

---

⚙️ Configuration

Markus stores its configuration in ~/.config/markus/config.toml (or standard AppData paths on Windows/Mac). You can edit this file to customize engine behavior:

# markus-engine configurationthreads = 32# Number of Rayon threads to use for inferencectx_size = 4096# Maximum context window sizegpu_layers = 0# Number of layers to offload to GPU (0 = pure CPU)temperature = 0.7# Default sampling temperaturerepeat_penalty = 1.1# Penalty for repeating tokenssystem_prompt = "You are a helpful AI assistant."

To view or edit from the CLI, use: markus config show or markus config edit.

---

🚀 Roadmap & Future Vision

While Markus v3.0.0 serves as an ultra-fast local inference manager, the ultimate vision for Markus extends far beyond basic chat.

Upcoming Milestone: We are planning to evolve Markus into a Flexible, Servable Autonomous Agent Platform.

Similar to systems like agy, Markus will soon be capable of:

• Hosting self-directed autonomous agent workflows.
• Executing system-level tools, reading local repositories, and managing codebases securely.
• Operating as a completely self-hosted, private agentic backend that developers can deploy on their own infrastructure without relying on cloud APIs.

By building the foundation entirely in memory-safe Rust, the future agentic layer will be fast, secure, and infinitely scalable.

---

License: MIT