Lightweight LLM Fine-tuning for On-Device Deployment

From fine-tuning to edge deployment — the complete, lightweight solution

🌐 Homepage · Installation · Examples · Sample App

Fine-tune → Export → Convert → Deploy

---

🎯 What is Nimbo?

Nimbo is a lightweight, end-to-end LLM fine-tuning framework designed specifically for on-device deployment.

Unlike heavy frameworks like Transformers or Unsloth that pack hundreds of features, Nimbo focuses on what you actually need — nothing more, nothing less.

┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Fine-tune │ -> │ Export │ -> │ Convert │ -> │ Deploy │
│ (LoRA) │ │ (Merge) │ │ (ONNX/etc) │ │ (Sample App) │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘

Why Nimbo?

Pain Point	Heavy Frameworks	Nimbo
"I just want to fine-tune and deploy"	500+ dependencies, complex setup	Minimal deps, just works
"My target is mobile/edge devices"	Server-focused, no export tools	On-device first design
"I need a working demo fast"	DIY everything	Sample apps included
"Training is too slow"	Generic implementation	Triton-optimized kernels

---

🚀 End-to-End Pipeline

Step 1: Fine-tune (3 lines of code)

fromnimboimportNimbotrainer=Nimbo("microsoft/phi-2", dataset="your_data")
trainer.train()
trainer.save() # Merged model ready

Step 2: Export for On-Device Deployment

# Export to CoreML for iOS/macOS (Apple Neural Engine optimized)fromnimbo.exportimportLlamaConverter, LlamaConfig, LlamaForCausalLM# Load and convert modelconfig=LlamaConfig.from_json("./model/config.json")
model=LlamaForCausalLM(config)
model.load_pretrained_weights("./model")
# Convert to CoreML with 4-bit LUT quantizationconverter=LlamaConverter(
model=model,
context_length=512,
lut_bits=4, # 4-bit quantization (supported: 4, 6, 8)
)
coreml_model=converter.convert(split_part="monolithic")
coreml_model.save("model.mlpackage")

Step 3: Deploy with Sample Apps

Ready-to-use iOS sample application included:

# Open the sample app in Xcode
open SampleApp/NimboChat

📱 NimboChat — SwiftUI chat app with CoreML inference on Apple Neural Engine

---

⚡ Performance: Triton Kernel Acceleration

Nimbo includes custom Triton GPU kernels for up to 8x faster training:

Kernel	Speedup	Description
RMSNorm	7-8x	Fused normalization
SwiGLU	3-5x	Fused activation
RoPE	2x	Fused rotary embeddings

Benchmark: EXAONE 4.0 1.2B (A100 80GB)

┌─────────────────────────────────────────────────────────────┐
│ Training Speed Comparison │
├─────────────────┬──────────────┬──────────────┬─────────────┤
│ Metric │ Baseline │ + Triton │ Speedup │
├─────────────────┼──────────────┼──────────────┼─────────────┤
│ Forward Pass │ 119.79 ms │ 91.79 ms │ 1.3x │
│ Throughput │ 12,395 tok/s │ 14,116 tok/s │ +14% │
└─────────────────┴──────────────┴──────────────┴─────────────┘

Enable with one line:

fromnimbo.kernelsimportpatch_modelpatch_model(model) # 181 layers optimized automatically

---

🪶 Lightweight by Design

Nimbo vs. Others

Feature	Nimbo	Transformers	Unsloth
Install size	~50MB	~500MB+	~200MB+
Dependencies	Minimal	100+	50+
On-device export	✅	❌	❌
Sample apps	✅	❌	❌
Triton kernels	✅	❌	✅
Learning curve	5 min	Hours	30 min

Core Philosophy

• Essential features only — No bloat, no unused code
• On-device first — Every feature considers edge deployment
• Zero-to-deploy — From idea to working app, not just a model file
• Developer friendly — Simple API, sensible defaults

---

📦 Installation

# Lightweight install
pip install git+https://github.com/Nimbo-code/Nimbo.git
# With all export formats
pip install "nimbo[export] @ git+https://github.com/Nimbo-code/Nimbo.git"# Development
git clone https://github.com/Nimbo-code/Nimbo.git &&cd Nimbo
pip install -e ".[dev]"

---

🔧 Environment Setup (Fine-tuning to CoreML)

Complete guide for running the full pipeline: Fine-tune → Merge → CoreML Convert → Deploy

Prerequisites

• Python 3.9+ (3.10 recommended)
• macOS (CoreML conversion requires macOS with Xcode Command Line Tools)
• GPU (optional, for fine-tuning — NVIDIA with CUDA or Apple Silicon MPS)

Step 1: Create Virtual Environment

git clone https://github.com/Nimbo-code/Nimbo.git &&cd Nimbo
python3 -m venv .venv
source .venv/bin/activate

Step 2: Install Dependencies

# Core (fine-tuning)
pip install torch transformers datasets peft trl accelerate
# CoreML conversion
pip install coremltools safetensors numpy pyyaml tqdm scikit-learn
# HuggingFace model download (optional)
pip install huggingface_hub
# Install Nimbo itself (editable mode)
pip install -e .

Or install everything at once:

pip install -e ".[all]"
pip install coremltools safetensors scikit-learn

Step 3: Full Pipeline

# 1. Fine-tunefromnimboimportNimbotrainer=Nimbo("meta-llama/Llama-3.2-1B-Instruct", dataset="your_data")
trainer.train()
trainer.save() # Saves merged model to ./final_merged# 2. Convert to CoreML (split model, per-component quantization)fromnimbo.export.coremlimportconvert_hf_to_coremlresult=convert_hf_to_coreml(
'final_merged',
'coreml_models/output',
lut_bits=6, # Decoder: 6-bit LUTlut_embeddings_bits=-1, # Embeddings: float16 (no quantization)lut_lmhead_bits=6, # LM Head: 6-bit LUTsplit_model=True,
num_chunks=1,
)
# 3. Output: .mlpackage files + meta.yaml + tokenizer files

Step 4: Compile & Deploy to iPhone

# Compile each .mlpackage to .mlmodelc
xcrun coremlc compile coreml_models/output/model_embeddings.mlpackage coreml_models/output/
xcrun coremlc compile coreml_models/output/model_FFN_PF_lut6.mlpackage coreml_models/output/
xcrun coremlc compile coreml_models/output/model_lm_head_lut6.mlpackage coreml_models/output/
# Transfer to iPhone via Xcode, Finder, or Files app

Package Versions (Tested)

Package	Version
torch	2.7+
transformers	5.0+
coremltools	8.2+
safetensors	0.7+
peft	0.18+
trl	0.28+

---

🎯 Supported Models

Triton-Optimized Models (Accelerated Training)

Architecture	Models	Triton Kernels	On-Device
LLaMA 3.2	1B, 3B Instruct	✅ Full	✅ Recommended
EXAONE	3.5/4.0 (1.2B-32B)	✅ Full	✅
LLaMA	2 (7B-70B), 3 (8B, 70B)	✅ Full	✅
Phi	Phi-2, Phi-3, Phi-3.5	✅ Full	✅
Qwen2	0.5B, 1.5B, 7B	✅ Full	✅
Mistral	7B	✅ Full	✅

Other Compatible Models

Architecture	Models	On-Device
Gemma	Gemma, Gemma 2	✅
Mixtral	8x7B	⚠️ Large

On-Device Recommendation: LLaMA 3.2 1B/3B, Phi-2, EXAONE 1.2B, Qwen2-1.5B

---

💡 Examples

Basic Fine-tuning

fromnimboimportNimbotrainer=Nimbo(
base_model_name="microsoft/phi-2",
dataset="your_dataset",
output_dir="./output",
)
trainer.train()
trainer.save()

QLoRA (4-bit) for Consumer GPUs

fromnimboimportNimbo, QuantizationConfigtrainer=Nimbo(
base_model_name="meta-llama/Llama-2-7b-hf",
dataset="your_dataset",
quantization_config=QuantizationConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype="bfloat16",
),
)
# Fine-tune 7B model on 8GB VRAM!

Custom Training Config

fromnimboimportNimbo, LoRAConfig, TrainingConfigtrainer=Nimbo(
base_model_name="LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct",
dataset="your_dataset",
lora_config=LoRAConfig(r=16, lora_alpha=32),
training_config=TrainingConfig(
learning_rate=1e-4,
num_train_epochs=3,
gradient_checkpointing=True,
),
)

OLoRA (Orthogonal LoRA) for Better Stability

fromnimboimportNimbo, LoRAConfig# OLoRA uses orthogonal initialization via QR decomposition# Better training stability compared to standard LoRAtrainer=Nimbo(
base_model_name="microsoft/phi-2",
dataset="your_dataset",
lora_config=LoRAConfig(
r=16,
lora_alpha=32,
init_lora_weights="olora", # Orthogonal initialization
),
)
# Other options:# - init_lora_weights="pissa" # Principal Singular Values Adaptation# - init_lora_weights="loftq" # Quantization-aware initialization# - use_rslora=True # Rank-Stabilized LoRA (scales alpha by sqrt(r))# - use_dora=True # Weight-Decomposed LoRA

Response-Only Fine-tuning (Instruction Tuning)

fromnimboimportNimbo, TrainingConfig# Only compute loss on response/completion tokens# Instruction/input tokens are masked (labels=-100)trainer=Nimbo(
base_model_name="microsoft/phi-2",
dataset="your_instruction_dataset", # prompt-completion formattraining_config=TrainingConfig(
train_on_responses_only=True, # Only train on completionslearning_rate=2e-4,
),
)
trainer.train()
trainer.save()

Export to CoreML (iOS/macOS)

fromnimbo.exportimportLlamaConverter, LlamaConfig, LlamaForCausalLM# Load model configuration and weightsconfig=LlamaConfig.from_json("./model/config.json")
model=LlamaForCausalLM(config)
model.load_pretrained_weights("./model")
# Create converter with optimizationsconverter=LlamaConverter(
model=model,
context_length=512, # Max sequence lengthlut_bits=4, # LUT quantization (4-bit, 6-bit, or 8-bit)batch_size=64, # Batch size for prefill mode
)
# Convert to monolithic CoreML modelcoreml_model=converter.convert(split_part="monolithic")
coreml_model.save("llama_monolithic.mlpackage")
# Or convert as separate components for flexible deploymentembeddings=converter.convert(split_part="1") # Embeddingstransformer=converter.convert(split_part="2") # FFN layerslm_head=converter.convert(split_part="3") # LM head

Supported split_part options:

• "monolithic" - Single file (inference mode)
• "monolithic_prefill" - Single file (prefill mode)
• "1" - Embeddings only
• "2" - Transformer FFN layers
• "2_prefill" - Transformer prefill mode
• "3" - LM head only
• "123" - All components as separate files

Export to ONNX (Coming Soon)

fromnimboimportNimbotrainer=Nimbo("microsoft/phi-2", dataset="data")
trainer.train()
trainer.save()
# Export for deploymenttrainer.export(
format="onnx",
output_path="./deploy/model.onnx",
quantize=True, # INT8 quantization for edge
)

Streaming Inference

fromnimboimportNimboInferencemodel=NimboInference("./output/final_merged")
fortokeninmodel.stream("Once upon a time"):
print(token, end="", flush=True)

---

🗺️ Roadmap

• LoRA/QLoRA fine-tuning
• OLoRA (Orthogonal LoRA) and advanced variants (RSLoRA, DoRA, PiSSA)
• Response-only fine-tuning (completion_only_loss)
• Triton kernel acceleration
• EXAONE 4.0 optimization
• LLaMA 3.2 (1B, 3B) Triton optimization
• CoreML export for iOS/macOS (ANE optimized, LUT quantization)
• Sample iOS app (SwiftUI) — NimboChat
• ONNX export with quantization
• ONNX Runtime sample app

---

📊 API Reference

Core Classes

Class	Description
Nimbo	Main trainer for fine-tuning
NimboInference	Lightweight inference engine
LlamaConverter	CoreML export for LLaMA models

Export Module (nimbo.export)

Class	Description
LlamaConverter	Convert LLaMA to CoreML (ANE optimized)
LlamaConfig	Configuration for ANE-optimized model
LlamaForCausalLM	ANE-optimized LLaMA implementation
BaseConverter	Abstract base for custom converters

Configuration

Config	Purpose
LoRAConfig	LoRA hyperparameters
TrainingConfig	Training settings
QuantizationConfig	QLoRA settings

CoreML Export Options

Option	Description
context_length	Maximum sequence length (default: 512)
lut_bits	LUT quantization: 4, 6, or 8 bits
batch_size	Batch size for prefill mode (default: 64)
split_part	Model splitting strategy
argmax_in_model	Compute argmax inside model

---

🛠️ Development

git clone https://github.com/Nimbo-code/Nimbo.git
cd Nimbo
pip install -e ".[dev]"# Run tests
pytest tests/ -v
# Format
black src/ && isort src/

---

📜 License

Apache License 2.0 — Use freely for personal and commercial projects.

---

Nimbo — Fine-tune once, deploy everywhere

_{Made for developers who ship to production, not just notebooks}