Skip to content
BranchesTags

Repository files navigation

unsloth logo

Finetune Llama 3.3, Mistral, Phi-4, Qwen 2.5 & Gemma 2x faster with 80% less memory!

✨ Finetune for Free

All notebooks are beginner friendly! Add your dataset, click "Run All", and you'll get a 2x faster finetuned model which can be exported to GGUF, Ollama, vLLM or uploaded to Hugging Face.

Unsloth supportsFree NotebooksPerformanceMemory use
Llama 3.2 (3B)▶️ Start for free2x faster70% less
GRPO (R1 reasoning)▶️ Start for free2x faster80% less
Phi-4 (14B)▶️ Start for free2x faster70% less
Llama 3.2 Vision (11B)▶️ Start for free2x faster50% less
Llama 3.1 (8B)▶️ Start for free2x faster70% less
Gemma 2 (9B)▶️ Start for free2x faster70% less
Qwen 2.5 (7B)▶️ Start for free2x faster70% less
Mistral v0.3 (7B)▶️ Start for free2.2x faster75% less
Ollama▶️ Start for free1.9x faster60% less
DPO Zephyr▶️ Start for free1.9x faster50% less

⚡ Quickstart

  • Install with pip (recommended) for Linux devices:
pip install unsloth

For Windows install instructions, see here.

🦥 Unsloth.ai News

Click for more news

🔗 Links and Resources

TypeLinks
📚 Documentation & WikiRead Our Docs
Twitter (aka X)Follow us on X
💾 InstallationPip install
🔮 Our ModelsUnsloth Releases
✍️ BlogRead our Blogs
RedditJoin our Reddit page

⭐ Key Features

  • All kernels written in OpenAI's Triton language. Manual backprop engine.
  • 0% loss in accuracy - no approximation methods - all exact.
  • No change of hardware. Supports NVIDIA GPUs since 2018+. Minimum CUDA Capability 7.0 (V100, T4, Titan V, RTX 20, 30, 40x, A100, H100, L40 etc) Check your GPU! GTX 1070, 1080 works, but is slow.
  • Works on Linux and Windows
  • Supports 4bit and 16bit QLoRA / LoRA finetuning via bitsandbytes.
  • If you trained a model with 🦥Unsloth, you can use this cool sticker!

💾 Install Unsloth

You can also see our documentation for more detailed installation and updating instructions here.

Pip Installation

Install with pip (recommended) for Linux devices:

pip install unsloth

See here for advanced pip install instructions.

Windows Installation

Warning

Python 3.13 does not support Unsloth. Use 3.12, 3.11 or 3.10

  1. Install NVIDIA Video Driver: You should install the latest version of your GPUs driver. Download drivers here: NVIDIA GPU Drive.

  2. Install Visual Studio C++: You will need Visual Studio, with C++ installed. By default, C++ is not installed with Visual Studio, so make sure you select all of the C++ options. Also select options for Windows 10/11 SDK. For more detailed instructions, see here.

  3. Install CUDA Toolkit: Follow the instructions to install CUDA Toolkit.

  4. Install PyTorch: You will need the correct version of PyTorch that is compatibile with your CUDA drivers, so make sure to select them carefully. Install PyTorch.

  5. Install Unsloth:

pipinstall"unsloth[windows] @ git+https://github.com/unslothai/unsloth.git"

Notes

To run Unsloth directly on Windows:

  • Install Triton from this Windows fork and follow the instructions here (be aware that the Windows fork requires PyTorch >= 2.4 and CUDA 12)
  • In the SFTTrainer, set dataset_num_proc=1 to avoid a crashing issue:
trainer=SFTTrainer(
dataset_num_proc=1,
...
)

Advanced/Troubleshooting

For advanced installation instructions or if you see weird errors during installations:

  1. Install torch and triton. Go to https://pytorch.org to install it. For example pip install torch torchvision torchaudio triton
  2. Confirm if CUDA is installated correctly. Try nvcc. If that fails, you need to install cudatoolkit or CUDA drivers.
  3. Install xformers manually. You can try installing vllm and seeing if vllm succeeds. Check if xformers succeeded with python -m xformers.info Go to https://github.com/facebookresearch/xformers. Another option is to install flash-attn for Ampere GPUs.
  4. Double check that your versions of Python, CUDA, CUDNN, torch, triton, and xformers are compatible with one another. The PyTorch Compatibility Matrix may be useful.
  5. Finally, install bitsandbytes and check it with python -m bitsandbytes

Conda Installation (Optional)

⚠️Only use Conda if you have it. If not, use Pip. Select either pytorch-cuda=11.8,12.1 for CUDA 11.8 or CUDA 12.1. We support python=3.10,3.11,3.12.

conda create --name unsloth_env \
python=3.11 \
pytorch-cuda=12.1 \
pytorch cudatoolkit xformers -c pytorch -c nvidia -c xformers \
-y
conda activate unsloth_env
pip install unsloth
If you're looking to install Conda in a Linux environment, read here, or run the below 🔽
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

Advanced Pip Installation

⚠️Do **NOT** use this if you have Conda. Pip is a bit more complex since there are dependency issues. The pip command is different for torch 2.2,2.3,2.4,2.5 and CUDA versions.

For other torch versions, we support torch211, torch212, torch220, torch230, torch240 and for CUDA versions, we support cu118 and cu121 and cu124. For Ampere devices (A100, H100, RTX3090) and above, use cu118-ampere or cu121-ampere or cu124-ampere.

For example, if you have torch 2.4 and CUDA 12.1, use:

pip install --upgrade pip
pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git"

Another example, if you have torch 2.5 and CUDA 12.4, use:

pip install --upgrade pip
pip install "unsloth[cu124-torch250] @ git+https://github.com/unslothai/unsloth.git"

And other examples:

pip install "unsloth[cu121-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu118-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu118-torch240] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu121-torch230] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu121-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu121-torch250] @ git+https://github.com/unslothai/unsloth.git"
pip install "unsloth[cu124-ampere-torch250] @ git+https://github.com/unslothai/unsloth.git"

Or, run the below in a terminal to get the optimal pip installation command:

wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python -

Or, run the below manually in a Python REPL:

try: importtorchexcept: raiseImportError('Install torch via `pip install torch`')
frompackaging.versionimportVersionasVv=V(torch.__version__)
cuda=str(torch.version.cuda)
is_ampere=torch.cuda.get_device_capability()[0] >=8ifcuda!="12.1"andcuda!="11.8"andcuda!="12.4": raiseRuntimeError(f"CUDA = {cuda} not supported!")
ifv<=V('2.1.0'): raiseRuntimeError(f"Torch = {v} too old!")
elifv<=V('2.1.1'): x='cu{}{}-torch211'elifv<=V('2.1.2'): x='cu{}{}-torch212'elifv<V('2.3.0'): x='cu{}{}-torch220'elifv<V('2.4.0'): x='cu{}{}-torch230'elifv<V('2.5.0'): x='cu{}{}-torch240'elifv<V('2.6.0'): x='cu{}{}-torch250'else: raiseRuntimeError(f"Torch = {v} too new!")
x=x.format(cuda.replace(".", ""), "-ampere"ifis_ampereelse"")
print(f'pip install --upgrade pip && pip install "unsloth[{x}] @ git+https://github.com/unslothai/unsloth.git"')

📜 Documentation

  • Go to our official Documentation for saving to GGUF, checkpointing, evaluation and more!
  • We support Huggingface's TRL, Trainer, Seq2SeqTrainer or even Pytorch code!
  • We're in 🤗Hugging Face's official docs! Check out the SFT docs and DPO docs!
  • If you want to download models from the ModelScope community, please use an environment variable: UNSLOTH_USE_MODELSCOPE=1, and install the modelscope library by: pip install modelscope -U.

unsloth_cli.py also supports UNSLOTH_USE_MODELSCOPE=1 to download models and datasets. please remember to use the model and dataset id in the ModelScope community.

fromunslothimportFastLanguageModelimporttorchfromtrlimportSFTTrainer, SFTConfigfromdatasetsimportload_datasetmax_seq_length=2048# Supports RoPE Scaling interally, so choose any!# Get LAION dataseturl="https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl"dataset=load_dataset("json", data_files= {"train" : url}, split="train")
# 4bit pre quantized models we support for 4x faster downloading + no OOMs.fourbit_models= [
"unsloth/Meta-Llama-3.1-8B-bnb-4bit", # Llama-3.1 2x faster"unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
"unsloth/Meta-Llama-3.1-70B-bnb-4bit",
"unsloth/Meta-Llama-3.1-405B-bnb-4bit", # 4bit for 405b!"unsloth/Mistral-Small-Instruct-2409", # Mistral 22b 2x faster!"unsloth/mistral-7b-instruct-v0.3-bnb-4bit",
"unsloth/Phi-3.5-mini-instruct", # Phi-3.5 2x faster!"unsloth/Phi-3-medium-4k-instruct",
"unsloth/gemma-2-9b-bnb-4bit",
"unsloth/gemma-2-27b-bnb-4bit", # Gemma 2x faster!"unsloth/Llama-3.2-1B-bnb-4bit", # NEW! Llama 3.2 models"unsloth/Llama-3.2-1B-Instruct-bnb-4bit",
"unsloth/Llama-3.2-3B-bnb-4bit",
"unsloth/Llama-3.2-3B-Instruct-bnb-4bit",
"unsloth/Llama-3.3-70B-Instruct-bnb-4bit"# NEW! Llama 3.3 70B!
] # More models at https://huggingface.co/unslothmodel, tokenizer=FastLanguageModel.from_pretrained(
model_name="unsloth/Llama-3.2-1B",
max_seq_length=max_seq_length,
load_in_4bit=True,
)
# Do model patching and add fast LoRA weightsmodel=FastLanguageModel.get_peft_model(
model,
r=16,
target_modules= ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha=16,
lora_dropout=0, # Supports any, but = 0 is optimizedbias="none", # Supports any, but = "none" is optimized# [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!use_gradient_checkpointing="unsloth", # True or "unsloth" for very long contextrandom_state=3407,
max_seq_length=max_seq_length,
use_rslora=False, # We support rank stabilized LoRAloftq_config=None, # And LoftQ
)
trainer=SFTTrainer(
model=model,
train_dataset=dataset,
tokenizer=tokenizer,
args=SFTConfig(
dataset_text_field="text",
max_seq_length=max_seq_length,
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
warmup_steps=10,
max_steps=60,
logging_steps=1,
output_dir="outputs",
optim="adamw_8bit",
seed=3407,
),
)
trainer.train()
# Go to https://github.com/unslothai/unsloth/wiki for advanced tips like# (1) Saving to GGUF / merging to 16bit for vLLM# (2) Continued training from a saved LoRA adapter# (3) Adding an evaluation loop / OOMs# (4) Customized chat templates

💡 Reinforcement Learning

RL including DPO, GRPO, PPO, Reward Modelling, Online DPO all work with Unsloth. We're in 🤗Hugging Face's official docs! We're on the SFT docs and the DPO docs! List of RL notebooks:

  • ORPO notebook: Link
  • DPO Zephyr notebook: Link
  • KTO notebook: Link
  • SimPO notebook: Link
Click for DPO code
importosos.environ["CUDA_VISIBLE_DEVICES"] ="0"# Optional set GPU device IDfromunslothimportFastLanguageModelimporttorchfromtrlimportDPOTrainer, DPOConfigmax_seq_length=2048model, tokenizer=FastLanguageModel.from_pretrained(
model_name="unsloth/zephyr-sft-bnb-4bit",
max_seq_length=max_seq_length,
load_in_4bit=True,
)
# Do model patching and add fast LoRA weightsmodel=FastLanguageModel.get_peft_model(
model,
r=64,
target_modules= ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha=64,
lora_dropout=0, # Supports any, but = 0 is optimizedbias="none", # Supports any, but = "none" is optimized# [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!use_gradient_checkpointing="unsloth", # True or "unsloth" for very long contextrandom_state=3407,
max_seq_length=max_seq_length,
)
dpo_trainer=DPOTrainer(
model=model,
ref_model=None,
train_dataset=YOUR_DATASET_HERE,
# eval_dataset = YOUR_DATASET_HERE,tokenizer=tokenizer,
args=DPOConfig(
per_device_train_batch_size=4,
gradient_accumulation_steps=8,
warmup_ratio=0.1,
num_train_epochs=3,
logging_steps=1,
optim="adamw_8bit",
seed=42,
output_dir="outputs",
max_length=1024,
max_prompt_length=512,
beta=0.1,
),
)
dpo_trainer.train()

🥇 Performance Benchmarking

We tested using the Alpaca Dataset, a batch size of 2, gradient accumulation steps of 4, rank = 32, and applied QLoRA on all linear layers (q, k, v, o, gate, up, down):

ModelVRAM🦥 Unsloth speed🦥 VRAM reduction🦥 Longer context😊 Hugging Face + FA2
Llama 3.3 (70B)80GB2x>75%13x longer1x
Llama 3.1 (8B)80GB2x>70%12x longer1x

Context length benchmarks

Llama 3.1 (8B) max. context length

We tested Llama 3.1 (8B) Instruct and did 4bit QLoRA on all linear layers (Q, K, V, O, gate, up and down) with rank = 32 with a batch size of 1. We padded all sequences to a certain maximum sequence length to mimic long context finetuning workloads.

GPU VRAM🦥Unsloth context lengthHugging Face + FA2
8 GB2,972OOM
12 GB21,848932
16 GB40,7242,551
24 GB78,4755,789
40 GB153,97712,264
48 GB191,72815,502
80 GB342,73328,454

Llama 3.3 (70B) max. context length

We tested Llama 3.3 (70B) Instruct on a 80GB A100 and did 4bit QLoRA on all linear layers (Q, K, V, O, gate, up and down) with rank = 32 with a batch size of 1. We padded all sequences to a certain maximum sequence length to mimic long context finetuning workloads.

GPU VRAM🦥Unsloth context lengthHugging Face + FA2
48 GB12,106OOM
80 GB89,3896,916


Citation

You can cite the Unsloth repo as follows:

@software{unsloth,
author = {Daniel Han, Michael Han and Unsloth team},
title = {Unsloth},
url = {http://github.com/unslothai/unsloth},
year = {2023}
}

Thank You to

About

Finetune Llama 3.3, DeepSeek-R1 & Reasoning LLMs 2x faster with 70% less memory! 🦥

unsloth.ai

Resources

Readme
License

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages