A from-scratch Llama-family serving engine implementing block-based KV caching, continuous batching, streamed OpenAI-compatible generation, INT8 weight quantization, and GPU kernel profiling—without using an external generation or serving framework.
ForgeLLM is a systems portfolio project: the transformer forward pass,
prefill/decode loop, cache allocator, sampler, scheduler, cancellation path, and
telemetry live in this repository. PyTorch provides tensor primitives and GEMMs;
tokenizers, safetensors, and Hugging Face Hub provide file-format and artifact
integration. vLLM and Transformers appear only in isolated benchmark baselines.
Status: the CPU toy-model path is CI-tested and has a committed smoke sweep. Real-model CUDA results are intentionally not committed until produced by the target GPU; the repository never substitutes estimated or invented numbers.
Serving-engine internals — KV-cache management, continuous batching, and scheduling under concurrent load — are the exact mechanics behind every production LLM deployment, making this directly relevant to any tech/AI role building or operating inference infrastructure. The engineering discipline generalizes further: the same batching/scheduling/cache-eviction problem shows up in any resource-constrained serving system, and the project's own refusal to report GPU numbers it hasn't actually produced on target hardware is the kind of evidence discipline that matters equally in quant/finance infrastructure claims and in consulting engagements where a client needs to trust a performance number, not just hear one.
flowchart LR
C["OpenAI client"] --> A["FastAPI · auth · rate limit"]
A --> Q["Bounded admission queue"]
Q --> S["Static / continuous scheduler"]
S --> E["Explicit prefill + decode engine"]
E --> T["Tokenizer"]
E --> K["Block KV allocator"]
E --> P["Seeded sampler"]
E --> M["Llama / Mistral / Qwen2 model"]
M --> B["PyTorch · Triton · CUDA extension"]
B --> G["GPU"]
S --> O["Prometheus + live dashboard"]
At each decode boundary, completed and cancelled sequences leave, queued work enters, and the remaining sequences form the next GPU batch. Long prompts are split into bounded chunks and interleaved with decode. Per-request RNGs make sampling reproducible even when batch membership changes.
- Llama-family decoder: RoPE, RMSNorm, grouped-query attention, SwiGLU, causal masking, safetensors loading, and tied embeddings.
- Explicit autoregressive prefill/decode with greedy, temperature, top-k, top-p, EOS, stop strings, token limits, and deterministic seeds.
- Physical KV blocks with logical per-request block tables, worst-case admission reservation, reclamation, capacity errors, and occupancy/byte telemetry.
- Decode-only block-table attention reads physical KV blocks directly; the Triton kernel uses online softmax without constructing padded batch-wide K/V.
- No-batch, static-batch, and iteration-boundary continuous-batch schedulers; chunked prefill, token budgets, FCFS, and aged shortest-prompt-first policies.
- Streaming/non-streaming
POST /v1/chat/completions, disconnect cancellation, API keys, token-bucket limits, queue limits, timeouts, structured errors, tracing IDs, graceful shutdown, health/readiness, and Prometheus metrics. - Reference PyTorch, fused Triton RMSNorm, and a separately compiled C++/CUDA RMSNorm, with correctness and latency benchmarks.
- Symmetric per-output-channel INT8 weights. Quantization is ours; GEMM is PyTorch after dequantization, so this is a memory experiment, not an optimized INT8 execution claim.
- Automated load matrix, machine-readable JSON/CSV, plots, and honest external controls for naive Transformers and vLLM.
The toy model validates the complete service without downloading weights. Use float32 on CPU:
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,bench]"
forge-llm serve --model toy --device cpu --dtype float32In another terminal:
curl http://localhost:8000/v1/chat/completions \
-H "Authorization: Bearer dev-key" \
-H "Content-Type: application/json" \
-d '{"model":"toy","messages":[{"role":"user","content":"Hello"}],"max_tokens":16,"stream":true}'Then open http://localhost:8000/dashboard.
The default key field is dev-key; production deployments must set
FORGE_ENVIRONMENT=production and a non-default FORGE_API_KEYS. Production
mode refuses to start with the development key. Terminate TLS at a trusted
reverse proxy or ingress.
Serve a supported open checkpoint (TinyLlama is the recommended first GPU target):
FORGE_API_KEYS=replace-me forge-llm serve \
--model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
--device cuda --dtype float16 --scheduler continuousCUDA container:
docker build -f Dockerfile.cuda -t forge-llm:cuda .
docker run --gpus all -p 8000:8000 -e FORGE_MODEL="$MODEL" \
-e FORGE_ENVIRONMENT=production -e FORGE_API_KEYS=replace-me forge-llm:cudaThe manually dispatched GPU acceptance workflow targets an isolated,
self-hosted NVIDIA runner and uploads kernel, API, and optional full-matrix
evidence tied to the exact commit SHA.
Supported architecture families are llama, mistral, and qwen2. Unsupported
RoPE scaling is rejected at load time rather than silently producing wrong
tokens.
The repository commits a labeled CPU toy smoke run; GPU result slots remain empty by design. Run the same matrix on your target GPU:
python -m benchmarks.run_matrix --experiment all \
--model TinyLlama/TinyLlama-1.1B-Chat-v1.0
python -m benchmarks.plot_results benchmarks/results/matrix.csv
python -m benchmarks.kernel_benchmark
python -m benchmarks.paged_attention_benchmark| Experiment | Variants | Recorded outputs |
|---|---|---|
| Batching | batch=1 vs continuous | tok/s, req/s, p50/p95/p99, TTFT |
| Scheduling | static vs continuous | queue time, throughput, tail latency |
| Precision | FP16/BF16 vs INT8 weights | GPU memory, throughput, latency |
| KV cache | enabled vs full recomputation | generation throughput, memory |
| Saturation | 1/2/4/8/16/32 clients | saturation point and tail latency |
| Context | increasing prompt lengths | prefill throughput and TTFT |
benchmarks/load_test.py writes per-request data plus summaries. Environment
metadata includes GPU/VRAM, CPU/RAM, CUDA, PyTorch, model, input/output settings,
and concurrency. See benchmark methodology before making
comparisons.
This is an API/scheduler regression measurement, not an LLM/GPU performance claim: 2-layer random toy model, CPU FP32, 24 streamed requests per point, 8-token limit, Windows 11, 8 logical CPUs, PyTorch 2.13.0 CPU.
| Clients | Generation tok/s | E2E p50 | E2E p95 | E2E p99 | TTFT p95 |
|---|---|---|---|---|---|
| 1 | 184.7 | 37.8 ms | 47.1 ms | 49.1 ms | 16.2 ms |
| 2 | 288.3 | 51.9 ms | 65.8 ms | 67.7 ms | 22.7 ms |
| 4 | 317.3 | 94.2 ms | 111.1 ms | 112.2 ms | 41.4 ms |
| 8 | 376.0 | 164.4 ms | 178.6 ms | 180.2 ms | 72.4 ms |
Throughput was still increasing at 8 clients in this smoke sweep while tail
latency rose, so this bounded run does not claim a CPU saturation point. Raw
per-request JSON and the CSV summary are in benchmarks/results/; reproduce all
live checks with scripts/live_verify.ps1 on Windows.
src/forge/ engine, scheduler, API, kernels, observability
csrc/ minimal C++/CUDA systems extension
tests/ unit, integration, API, concurrency, cache, kernel tests
benchmarks/ load matrix, plots, kernel test, external baselines
dashboard/ dependency-free real-time operations view
docs/ architecture, ADRs, report, profiling, limitations
examples/ streaming client
scripts/ build and operational helpers
- Technical report
- Architecture and request sequence
- Benchmark and profiler methodology
- Architecture decisions
- Known limitations and roadmap
The block allocator is inspired by the idea of paged virtual memory, not by vLLM internals. Its gather path favors legibility and correctness and is expected to lose to vLLM's paged-attention kernels. The project is designed to reveal that gap with profiles and measurements, then make the next optimization obvious—not to obscure it with an unfair benchmark.
Apache-2.0 licensed. Contributions should include a correctness test and, for performance claims, the raw machine-readable result.