A Helm chart for deploying LLMs. This stack is a lightweight, transparent framework to allow you to deploy any model on any inference engine with minimal configuration. It is designed to be flexible, allowing you to easily switch between different models and inference engines without changing your client code.
The spirit of this project is to produce a framework that solves distributed serving of LLMs in a non-specific way to the inference engine running the weights.
We achieve this by deploying the Onwards AI Gateway with configurable LLM model groups. This chart provides a transparent, unified interface for routing requests to multiple inference engines like vLLM, SGLang, TensorRT-LLM, and others. It also allows you to set human readable model names that map to backend services, making it easy to switch between models without changing client code.
If you want to go beyond what's available here for high-throughput deployments, contact us at hello@doubleword.ai.
The stack consists of:
- Onwards Gateway: The API gateway that routes requests to different inference engines.
- Model Groups: A grouping of kubernetes resources that represent a deployment of an inference engine. Each model group can only have one active model at a time, but can have custom numbers of replicas.
- Inference Engines: Backend services like vLLM, SGLang, TensorRT-LLM that handle the actual inference requests.
# Add the Helm repository (OCI format)
helm pull oci://ghcr.io/doublewordai/inference-stack
# Install with default configuration
helm install my-inference-stack oci://ghcr.io/doublewordai/inference-stack# Clone the repository
git clone https://github.com/doublewordai/inference-stack.git
cd inference-stack
# Install with default values
helm install my-inference-stack .# Or install with custom values
helm install my-inference-stack . -f my-values.yamlThe most important configuration is defining your model groups. Each model group represents a deployment of an inference engine:
modelGroups:
# vLLM deployment serving Llama modelsvllm-llama:
enabled: trueimage: vllm/vllm-openaitag: latestmodelAlias:
- "llama"
- "llama-3.1-8b-instruct"modelName: "meta-llama/Meta-Llama-3.1-8B-Instruct"command:
- "vllm"
- "serve"
- "--model"
- "meta-llama/Meta-Llama-3.1-8B-Instruct"# SGLang deployment serving Qwen models sglang-qwen:
enabled: trueimage: lmsysorg/sglangtag: latestmodelAlias:
- "qwen"
- "qwen-2.5-7b-instruct"modelName: "Qwen/Qwen2.5-7B-Instruct"command:
- "python"
- "-m"
- "sglang.launch_server"
- "--model-path"
- "Qwen/Qwen2.5-7B-Instruct"For GPU-enabled inference:
modelGroups:
vllm-llama:
enabled: trueresources:
limits:
nvidia.com/gpu: 2memory: 16Girequests:
nvidia.com/gpu: 1memory: 8GinodeSelector:
accelerator: nvidia-tesla-v100tolerations:
- key: nvidia.com/gpuoperator: Existseffect: NoSchedulemodelGroups:
vllm-llama:
# Single persistent volumepersistentVolumes:
- name: model-cachesize: 100GistorageClass: fast-ssdmountPath: /root/.cache/huggingfaceaccessModes:
- ReadWriteOnce# Multiple persistent volumes (advanced)# persistentVolumes:# - name: model-cache# size: 50Gi# mountPath: /root/.cache/huggingface# - name: model-weights# size: 200Gi# mountPath: /models# storageClass: fast-ssdmodelGroups:
vllm-llama:
# Rolling update strategy for zero-downtime deploymentsstrategy:
type: RollingUpdaterollingUpdate:
maxSurge: 1# Allow 1 extra pod during updatesmaxUnavailable: 0# Never take pods down (blue-green)# Scale replicas for high availabilityreplicaCount: 3ingress:
enabled: trueclassName: nginxannotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"hosts:
- host: inference.example.compaths:
- path: /pathType: Prefixtls:
- secretName: inference-tlshosts:
- inference.example.comThis framework can be used to deploy any set inference engines, we offer these as examples to get you started quickly. The examples/ directory contains ready-to-use configurations:
single-vllm.yaml- Single vLLM model deployment with persistent cachingsingle-sglang.yaml- SGLang deployment optimized for structured generationsingle-tensorrt-llm.yaml- TensorRT-LLM with model compilation init containermulti-engine.yaml- Complete multi-engine setup (vLLM + SGLang + TensorRT-LLM)
Use any example as a starting point, for example:
helm install my-stack . -f examples/single-vllm.yaml# values.yamlmodelGroups:
# Code generation with vLLMvllm-codegen:
enabled: trueimage: vllm/vllm-openaitag: latestmodelAlias:
- "codegen"
- "code-generation"modelName: "Salesforce/codegen-2B-multi"command:
- "vllm"
- "serve"
- "--model"
- "Salesforce/codegen-2B-multi"# Chat models with SGLangsglang-chat:
enabled: trueimage: lmsysorg/sglangtag: latestmodelAlias:
- "chat"
- "qwen-chat"modelName: "Qwen/Qwen2.5-7B-Instruct"command:
- "python"
- "-m"
- "sglang.launch_server"
- "--model-path"
- "Qwen/Qwen2.5-7B-Instruct"# High-performance with TensorRT-LLMtensorrt-llm:
enabled: trueimage: nvcr.io/nvidia/tritonservertag: 24.01-trtllm-python-py3modelAlias:
- "optimized"
- "fast-inference"modelName: "mistralai/Mistral-7B-Instruct-v0.2"command:
- "tritonserver"
- "--model-repository=/models"Once deployed, you can use the standard OpenAI API format:
# Get available models
curl -X GET http://inference.example.com/v1/models
# Generate completion
curl -X POST http://inference.example.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{ "model": "llama-3.1-8b-chat", "messages": [{"role": "user", "content": "Hello!"}] }'- Helm 3.13+
- Kubernetes 1.20+
- (Optional) GPU nodes with NVIDIA device plugin
# Install test dependencies
helm plugin install https://github.com/helm-unittest/helm-unittest.git
# Run linting
helm lint .# Run unit tests
helm unittest .# Test template rendering
helm template test-release .# Test with different configurations
helm template test-release . \
--set modelGroups.vllm-llama.enabled=true \
--set ingress.enabled=true \
--debug
# Install locally
helm install test-release . \
--set modelGroups.vllm-llama.enabled=true \
--dry-runThis project is licensed under the MIT License - see the LICENSE file for details.
Reach out at: hello@doubleword.ai!