QuantLLM

Load, quantise, serve, benchmark, and push any LLM -- in one line.

Quick Start • CLI • API • Examples • Custom Workflow

---

fromquantllmimportturbomodel=turbo("meta-llama/Llama-3.2-3B")
model.generate("Explain quantum computing simply")
model.export("gguf", "model.gguf")
model.push("your-username/my-model")

That is the entire API. One function loads any HuggingFace model, auto-detects your hardware, picks optimal quantization, enables Flash Attention, and configures memory. No config objects, no boilerplate.

Quick Start

pip install quantllm # core
pip install "quantllm[full]"# everything (GGUF, ONNX, MLX, server)

fromquantllmimportturbo# Load with auto-quantizationmodel=turbo("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
# Generate, chat, streammodel.generate("What is Python?")
model.chat([{"role": "user", "content": "Hello"}])
fortokeninmodel.generate("Count to 5", stream=True):
print(token, end="")
# Export to any formatmodel.export("gguf", "model.gguf", quantization="Q4_K_M")
model.export("onnx", "model.onnx")
model.export("safetensors", "./safetensors/")
# Push to HuggingFace Hub (auto-generates model card)model.push("your-username/my-quantized-model", license="apache-2.0")

Everything is optional. Skip the model and it auto-detects TinyLlama-1.1B. Skip the quantization and it picks Q4_K_M. Skip the config and SmartConfig reads your hardware and picks optimal settings.

What It Does

Operation	One line	What happens
Load	turbo("org/model")	Detects GPU/CPU, picks bits (2-16), dtype, group size, flash attention, offloading
Quantise	Built into load	BnB 4/8-bit or HQQ 2-8 bit, no calibration data needed
Generate	model.generate(...)	Text generation, chat, streaming
Export	model.export("gguf")	GGUF, ONNX, MLX, SafeTensors -- no SDK required
Fine-tune	model.finetune(data)	LoRA on quantized models, any dataset format
Serve	quantllm serve ...	OpenAI-compatible API, auto-detects GGUF backend
Benchmark	quantllm bench ...	Tokens/sec, latency, VRAM -- with hardware comparison
Push	model.push("user/repo")	Auto-generated model card, any format

CLI

quantllm # Full CLI
version # Show version
info # Show system info
convert <model> -o ... # Convert to GGUF
finetune <model> ... # Fine-tune with LoRAexport<model> -f ... # Export to any format
serve <model> --port 8080 # Start inference server
bench <model> ... # Run benchmarks
compare # Hardware auto-benchmark
models list # List curated models
models search <query># Search models
register <id> ... # Register model on HF Hub

quantllm info
quantllm serve TinyLlama-1.1B --port 8080
quantllm bench TinyLlama-1.1B --max-tokens 128 --save
quantllm models list --family llama

API Quick Reference

turbo(model_id, bits=None, config=None, device=None, quantize=True, verbose=False)

Param	Default	Description
model_id	"TinyLlama-1.1B"	HF model name or local path
bits	auto	2-16; auto-selected by SmartConfig.detect()
config	{}	Export/push config: {format, quantization, output_dir, push_format}
device	auto	"cuda:0" or "cpu"
quantize	True	Whether to apply BitsAndBytes quantization
verbose	False	Show loading details

SmartConfig (auto-decided)

Setting	Decision logic
bits	_choose_bits(): model size vs GPU memory, 3x headroom for training, 1.5x for inference
quant_type	_choose_quant_type(): Q4_K_M default, varies by bits
group_size	_choose_group_size(): 64 for >30B, 128 otherwise
dtype	bf16 (Ampere+) > fp16 (CUDA) > fp32 (CPU)
device	CUDA if available, else CPU
use_flash_attention	Compute capability >= 8.0
use_fused_kernels	Ampere+ GPU
compile_model	GPU with >= 16 GB VRAM
cpu_offload	Enabled when model exceeds available VRAM

model.push(repo_id, format=None, private=False, license="mit", token=None)

Pushes to HuggingFace Hub. Auto-exports before push. Generates a model card with: quantization details, performance benchmarks, usage instructions, and metadata.

model.finetune(data, epochs=3, batch_size=4, learning_rate=2e-4, lora_r=8, lora_alpha=16, output_dir="./finetuned")

Fine-tunes with LoRA. Accepts list of dicts ({"text": "..."}), JSON file path, or HuggingFace dataset name.

Quantization

Quantization is automatic on load. You can also control it explicitly:

# BitsAndBytes (4-bit or 8-bit, via transformers)model=turbo("meta-llama/Llama-3.2-3B", bits=4)
# HQQ (2-8 bit, native, no calibration data)fromquantllm.quantimportHQQQuantizer, HQQConfigmodel=turbo("TinyLlama-1.1B", bits=16) # load unquantizedquantizer=HQQQuantizer(HQQConfig(nbits=4, group_size=64))
qmodel=quantizer.quantize_model(model.model)

HQQ Performance (tested on H100, 7B model):

Bits	Time	Memory	Quality
2-bit	1.8s	1.86 GB	Functional
4-bit	1.0s	3.72 GB	Coherent
8-bit	1.7s	7.45 GB	Coherent

Inference Server

quantllm serve meta-llama/Llama-3.2-3B --port 8080

Auto-detects GGUF files and uses llama-cpp-python; otherwise uses Transformers.

fromopenaiimportOpenAIclient=OpenAI(base_url="http://localhost:8080/v1")
response=client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Hello!"}],
stream=True,
)
forchunkinresponse:
print(chunk.choices[0].delta.contentor"", end="")

Hardware Requirements

GPU VRAM	Models
6-8 GB	1-7B (4-bit)
12-24 GB	7-30B (4-bit)
24-80 GB	70B+

Tested on RTX 3060/3070/3080/3090/4070/4080/4090, A100, H100, Apple M1-M4.

Custom Workflow

Complete end-to-end pipeline demonstrating QuantLLM on a new/public model, run on Modal GPU (H100).

1. Load & Auto-Quantize

fromquantllmimportturbomodel=turbo("Qwen/Qwen2.5-7B-Instruct") # 7.62B params, BnB 8-bit on H100

2. Test Quantization (BnB + HQQ)

# BitsAndBytes runtime quantizationmodel_bnb4=turbo("Qwen/Qwen2.5-7B-Instruct", bits=4)
model_bnb8=turbo("Qwen/Qwen2.5-7B-Instruct", bits=8)
# HQQ native quantization (export/optimization)fromquantllm.quantimportHQQQuantizer, HQQConfigimporttorchweight=torch.randn(256, 512)
forbitsin [2, 4, 8]:
config=HQQConfig(bits=bits, group_size=64, optimize=True)
hqq_layer=HQQLinear(weight, config)

3. Export to All Formats

# GGUF (llama.cpp) - multiple quant typesmodel.export("gguf", "model.Q4_K_M.gguf", quantization="Q4_K_M")
model.export("gguf", "model.Q5_K_M.gguf", quantization="Q5_K_M")
model.export("gguf", "model.Q8_0.gguf", quantization="Q8_0")
# SafeTensors (HuggingFace native)model.export("safetensors", "./exports/safetensors/")
# ONNX (Optimum)model.export("onnx", "./exports/onnx/", quantization="int8")
# MLX (Apple Silicon)model.export("mlx", "./exports/mlx/", quantization="4bit")

4. Push to HF Hub with Model Cards

model.push("QuantLLM/Qwen2.5-7B-Instruct-GGUF", format="gguf", quantization="Q4_K_M")
model.push("QuantLLM/Qwen2.5-7B-Instruct-SafeTensors", format="safetensors")
model.push("QuantLLM/Qwen2.5-7B-Instruct-ONNX", format="onnx", quantization="int8")
model.push("QuantLLM/Qwen2.5-7B-Instruct-MLX", format="mlx", quantization="4bit")

5. Register in QuantLLM Registry

fromquantllm.registryimportregister_modelregister_model("QuantLLM/Qwen2.5-7B-Instruct-GGUF", family="qwen", params=7.0, verified=True, recommended="Q4_K_M")
register_model("QuantLLM/Qwen2.5-7B-Instruct-SafeTensors", family="qwen", params=7.0, verified=True, recommended="int8")
# ... etc

6. Validate from Registry

quantllm models list
# QuantLLM/Qwen2.5-7B-Instruct-GGUF qwen 7.0 4.0 4.5 Yes
model = turbo("QuantLLM/Qwen2.5-7B-Instruct-GGUF")
model.generate("Test")

Full Workflow Script

python examples/09_custom_workflow.py

Or run on Modal GPU:

modal run modal_custom_workflow.py::run_workflow

Examples

Full, runnable examples in examples/:

python examples/01_quickstart.py # Load, generate, chat, stream, export
python examples/05_hqq_quantization.py # HQQ 2-8 bit quantization
python examples/07_benchmark.py # Live benchmark + comparison
python examples/08_full_pipeline.py # Load -> Generate -> Chat -> Export -> Push -> Serve
python examples/09_custom_workflow.py # Complete custom model workflow

---

MIT License • GitHub • Report Bug