Efficient Python Bindings for `llama.cpp` library

Efficient Python bindings for ggml-org'sllama.cpp library. This package provides:

Low-level access to C API via ctypes interface.
- llama_cpp_lib
- mtmd_cpp_lib
- ggml_cpp_lib
  - Note: Synchronize ggml's ctypes calls as needed, but won't fully implement it, because most of it is called at the lower level in the upstream llama.cpp.
High-level Python API for text completion
FAQ

The new documentation will be maintained in the docs/wiki directory based on the LLM Wiki approach. Interested volunteers are welcome to participate in its maintenance and updates :)

Discussions

Starting March 2026, I am excited to announce that we have officially enabled the Discussions tab for llama-cpp-python!

You can access it right here: GitHub Discussions.

Why Discussions? & Updates on Documentation As the project has evolved, our existing documentation (docs) has unfortunately become a bit bloated and outdated. To provide you with more timely and clear information:

New Feature Releases: Moving forward, whenever a new feature is rolled out, I will publish a dedicated standalone article in the Discussions section. These posts will include detailed explanations, usage guides, and important caveats.
This approach will serve as a more agile and interactive "live documentation" while we figure out the best way to refactor the old docs.

Join the Community I warmly welcome all of you to use this new space. Let's build together:

💬 Discuss & Share: Have a question, an idea, or a cool use case? Share it with the community!
🛠️ Maintain & Test: Help us test new features, troubleshoot issues, and collaboratively maintain the repository.
📚 Learn & Grow: I hope everyone can benefit from this project, learn from each other, and gain valuable insights.

Thank you for your continuous support!

Installation

For a structured source-install and backend build guide, see docs/wiki/install.md.

Requirements:

Python 3.9+
C compiler
- Linux: gcc or clang
- Windows: Visual Studio 2022 Build Tools or MinGW
- MacOS: Xcode
CMake 3.21+
Git

To install the package, run:

Method 1:

pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

Method 2:

git clone https://github.com/JamePeng/llama-cpp-python --recursive
cd llama-cpp-python
python -m pip install -U pip
pip install .

This will also build llama.cpp from source and install it alongside this python package.

If this fails, add --verbose to the pip install see the full cmake build log.

Installation Configuration

llama.cpp supports a number of hardware acceleration backends to speed up inference as well as backend specific options. See the llama.cpp build docs for a full list.

All llama.cpp cmake build options can be set via the CMAKE_ARGS environment variable or via the --config-settings / -C cli flag during installation.

Environment Variables

# Linux and Mac
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" \
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

# Windows powershell$env:CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

# Windows command promptset CMAKE_ARGS = "-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

Sanity Checking
Use this line to check if installation was successful before moving further.
python.exe -c "from llama_cpp import Llama; print('llama-cpp import OK')"

CLI / requirements.txt

They can also be set via pip install -C / --config-settings command and saved to a requirements.txt file:

pip install --upgrade pip # ensure pip is up to date
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git" \
-C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

# requirements.txt
llama-cpp-python -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

Supported Backends

Below are some common backends, their build commands and any additional environment variables required.

CUDA

Installing a CUDA-supported version requires the CUDA Toolkit environment to be installed first.

Note: Please select and install according to your system environment and local graphics card model to ensure that the compilation is based on the optimal local environment.

See here: https://developer.nvidia.com/cuda-toolkit-archive

More Information see: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#cuda

Then, set the GGML_CUDA=on environment variable before installing:

# Linux
CMAKE_ARGS="-DGGML_CUDA=on" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

# Windows$env:CMAKE_ARGS="-DGGML_CUDA=on"
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

Note: Programmatic Dependent Launch (PDL) is a CUDA optimization for newer NVIDIA GPUs (CC >= 90; does not include Ada). It enables stream-level dependency-driven concurrent execution of CUDA kernels within the same stream, achieving similar kernel launch overhead reduction as CUDA Graphs. If you have a newer NVIDIA GPU (e.g. Hoppper, Blackwell and above), you can achieve significant speedups and latency reduction in token generation across nearly all models when compiling with -DGGML_CUDA_PDL=ON.

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with CUDA support. Make sure your system meets the following requirements:

CUDA version: 12.4, 12.6, 12.8, or 13.1
Python version: 3.10, 3.11, 3.12, 3.13, or 3.14
Starting with 0.3.39-preview, Windows and Linux x64 wheels are built with GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS.

This means CPU backends are shipped as dynamically loaded runtime libraries under:

site-packages/llama_cpp/lib

Supported CPU backend variants may include:

ggml-cpu-x64
ggml-cpu-sse42
ggml-cpu-sandybridge
ggml-cpu-ivybridge
ggml-cpu-piledriver
ggml-cpu-haswell
ggml-cpu-skylakex
ggml-cpu-cannonlake
ggml-cpu-cascadelake
ggml-cpu-cooperlake
ggml-cpu-icelake
ggml-cpu-alderlake
ggml-cpu-sapphirerapids
ggml-cpu-zen4

The old Basic and AVX2 wheel variants are no longer required for the new dynamic-backend wheels. GGML can load the compatible CPU backend at runtime, which improves CPU instruction-set compatibility across different x64 machines.

Before 0.3.39-preview:

Basic: compiled without AVX instructions for maximum compatibility.
AVX2: compiled with AVX2 instructions for newer CPUs.

Check the releases page: https://github.com/JamePeng/llama-cpp-python/releases

OpenBLAS (CPU)

To install with OpenBLAS, set the GGML_BLAS and GGML_BLAS_VENDOR environment variables before installing:

CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

OpenVINO

Install OpenVINO Runtime

Follow the guide to install OpenVINO Runtime from an archive file: Linux | Windows

Linux:

📦 Click to expand OpenVINO installation from an archive file on Ubuntu

wget https://raw.githubusercontent.com/ravi9/misc-scripts/main/openvino/ov-archive-install/install-openvino-from-archive.sh
chmod +x install-openvino-from-archive.sh
./install-openvino-from-archive.sh

Verify OpenVINO is initialized properly:

echo$OpenVINO_DIR

Supported Devices

OpenVINO backend supports the following hardware:

Intel CPUs
Intel GPUs (integrated and discrete)
Intel NPUs

Although OpenVINO supports a wide range of Intel hardware, the llama.cpp OpenVINO backend has been validated specifically on AI PCs such as the Intel® Core™ Ultra Series 1 and Series 2.

More Information see: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md

To install with OpenVINO, set the GGML_OPENVINO=ON environment variable before installing:

# Linuxsource /opt/intel/openvino/setupvars.sh
# Windows"C:\Program Files (x86)\Intel\openvino_2026.0\setupvars.bat"# Build
CMAKE_ARGS="-DGGML_OPENVINO=ON" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

Metal

On MacOS, Metal is enabled by default(GGML_METAL=ON). Using Metal makes the computation run on the GPU.

To disable the Metal build at compile time use the CMAKE_ARGS="-DGGML_METAL=OFF" cmake option.

When built with Metal support, you can explicitly disable GPU inference with the n-gpu-layers=0 parameter.

pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with Metal support. As long as your system meets some requirements:

CPU Arch: arm64
MacOS Version is 11.0 or later
Python Version is 3.10, 3.11, 3.12, 3.13 or 3.14

Check the releases page: https://github.com/JamePeng/llama-cpp-python/releases

HIP (ROCm)

Linux ROCm
This provides GPU acceleration on HIP-supported AMD GPUs. Make sure to have ROCm installed.
You can download it from your Linux distro's package manager or from here: ROCm Quick Start (Linux).
To install with HIP / ROCm support for AMD cards, set the GGML_HIP=ON environment variable before installing:
```
CMAKE_ARGS="-DGGML_HIP=ON -DGPU_TARGETS=gfx1030" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"
```
Note: GPU_TARGETS is optional, omitting it will build the code for all GPUs in the current system.
More details see here: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip

Windows ROCm

Note: Install TheRock ROCm, activate your venv, then run in PowerShell. Replace gfx1200 with your GPU architecture.

cmd /c '"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat" >nul 2>&1 && set'|ForEach-Object { if ($_-match'^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1],$matches[2],'Process') } }
rocm-sdk init
$ROCM_DEVEL="$env:VIRTUAL_ENV\Lib\site-packages\_rocm_sdk_devel"$ROCM_CORE="$env:VIRTUAL_ENV\Lib\site-packages\_rocm_sdk_core"$ROCM_GFX= (Get-Item"$env:VIRTUAL_ENV\Lib\site-packages\_rocm_sdk_libraries_gfx*").FullName
$env:HIP_PATH=$ROCM_DEVEL$env:ROCM_PATH=$ROCM_DEVEL$env:HIP_DEVICE_LIB_PATH="$ROCM_CORE\lib\llvm\amdgcn\bitcode"$env:PATH="$ROCM_DEVEL\bin;$ROCM_DEVEL\lib\llvm\bin;$ROCM_GFX\bin;$env:PATH"$env:CMAKE_GENERATOR="Ninja"$env:HIP_PLATFORM="amd"$env:CC="$ROCM_DEVEL\lib\llvm\bin\clang.exe"$env:CXX="$ROCM_DEVEL\lib\llvm\bin\clang++.exe"$env:HIP_CLANG_PATH="$ROCM_DEVEL\lib\llvm\bin"$R=$ROCM_DEVEL-replace'\\','/'$env:CMAKE_ARGS="-DGGML_HIP=ON -DGGML_HIPBLAS=on -DGPU_TARGETS=gfx1200 -DCMAKE_HIP_ARCHITECTURES=gfx1200 -DCMAKE_C_COMPILER=`"$R/lib/llvm/bin/clang.exe`" -DCMAKE_CXX_COMPILER=`"$R/lib/llvm/bin/clang++.exe`" -DHIP_LIBRARIES=`"$R/lib/amdhip64.lib`" -DCMAKE_PREFIX_PATH=`"$R`""
pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"--no-cache-dir

Vulkan

For Windows User: Download and install the Vulkan SDK with the default settings.
For Linux User:
- First, follow the official LunarG instructions for the installation and setup of the Vulkan SDK in the Getting Started with the Linux Tarball Vulkan SDK guide.
- After completing the first step, ensure that you have used the source command on the setup_env.sh file inside of the Vulkan SDK in your current terminal session. Otherwise, the build won't work. Additionally, if you close out of your terminal, you must perform this step again if you intend to perform a build. However, there are ways to make this persistent. Refer to the Vulkan SDK guide linked in the first step for more information about any of this.
For Mac User:
- Generally, follow LunarG's Getting Started with the MacOS Vulkan SDK guide for installation and setup of the Vulkan SDK. There are two options of Vulkan drivers on macOS, both of which implement translation layers to map Vulkan to Metal. They can be hot-swapped by setting the VK_ICD_FILENAMES environment variable to point to the respective ICD JSON file. Check the box for "KosmicKrisp" during the LunarG Vulkan SDK installation.
- Set environment variable for the LunarG Vulkan SDK after installation (and optionally add to your shell profile for persistence):
```
source /path/to/vulkan-sdk/setup-env.sh
```

More Information see: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#vulkan

Then install with Vulkan support by set the GGML_VULKAN=on environment variable before installing:

CMAKE_ARGS="-DGGML_VULKAN=on" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

SYCL

Supported OS

OS	Status	Verified
Linux	Support	Ubuntu 22.04, Fedora Silverblue 39, Arch Linux
Windows	Support	Windows 11

Intel GPU

SYCL backend supports Intel GPU Family:

Intel Data Center Max Series
Intel Flex Series, Arc Series
Intel Built-in Arc GPU
Intel iGPU in Core CPU (11th Generation Core CPU and newer, refer to oneAPI supported GPU).

On older Intel GPUs, you may try OpenCL although the performance is not optimal, and some GPUs may not support OpenCL nor have any GPGPU capabilities.

More Information see here: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md

To install with SYCL support, set the GGML_SYCL=on environment variable before installing:

# Export relevant ENV variablessource /opt/intel/oneapi/setvars.sh
# Option 1: Use FP32 (recommended for better performance in most cases)
CMAKE_ARGS="-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"# Option 2: Use FP16
CMAKE_ARGS="-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_SYCL_F16=ON" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

RPC

To install with RPC support, set the GGML_RPC=on environment variable before installing:

source /opt/intel/oneapi/setvars.sh CMAKE_ARGS="-DGGML_RPC=on" pip install "llama-cpp-python @ git+https://github.com/JamePeng/llama-cpp-python.git"

Install Notes

Optimization Options (Optional)

💡 Tip: If you want to save compilation time, you can skip building of llama.cpp with the standalone examples, tools, tests, and server by adding the following flags, as they are not required for Python bindings:

-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF

CUDA compiler warning suppression is optional

CUDA nvcc compiler may print many template-related warnings from ggml-cuda, such as:

warning #177-D
warning #221-D
warning #550-D

These usually generate a huge amount of noisy diagnostics rather than build blockers. They constantly flood logs and consume CPU printing performance.

For cleaner CI/local logs, you can pass:

-DCMAKE_CUDA_FLAGS="--diag-suppress=177 --diag-suppress=221 --diag-suppress=550"

Notes for `GGML_BACKEND_DL` + `GGML_CPU_ALL_VARIANTS` builds

When building wheels with `GGML_BACKEND_DL=ON` and `GGML_CPU_ALL_VARIANTS=ON`, GGML CPU backends are built as separate dynamic libraries, such as:

ggml-cpu-x64.dll
ggml-cpu-haswell.dll
ggml-cpu-alderlake.dll
ggml-cpu-zen4.dll

These backend libraries must be packaged together under:

site-packages/llama_cpp/lib

The runtime must also explicitly load them with:

ggml_backend_load_all_from_path()

Windows notes

For full x64 CPU variant coverage, LLVM/Clang is recommended. MSVC may skip some variants such as zen4, cooperlake, or sapphirerapids.

If GGML_OPENMP=ON is used, the LLVM OpenMP runtime must also be packaged next to the backend DLLs:

libomp140.x86_64.dll

Without this file, ggml-cpu-*.dll may fail to load dynamically at runtime.

Wheel packaging checklist

Enable GGML_BACKEND_DL=ON
Enable GGML_CPU_ALL_VARIANTS=ON
Use GGML_NATIVE=OFF for portable wheels
Install all ggml-cpu-* backend libraries into llama_cpp/lib
Package required runtime dependencies such as libomp140.x86_64.dll
Remove development-only files such as .lib, cmake/, and pkgconfig/

Upgrading and Reinstalling

To upgrade and rebuild llama-cpp-python add --upgrade --force-reinstall --no-cache-dir flags to the pip install command to ensure the package is rebuilt from source.

High-level API

API Reference

The high-level API provides a simple managed interface through the Llama class.

Below is a short example demonstrating how to use the high-level API to for basic text completion:

fromllama_cppimportLlamallm=Llama(
model_path="./models/7B/llama-model.gguf",
# n_gpu_layers=-1, # Uncomment to use GPU acceleration# seed=1337, # Uncomment to set a specific seed# n_ctx=2048, # Uncomment to increase the context window
)
output=llm(
"Q: Name the planets in the solar system? A: ", # Promptmax_tokens=32, # Generate up to 32 tokens, set to None to generate up to the end of the context windowstop=["Q:", "\n"], # Stop generating just before the model would generate a new questionecho=True# Echo the prompt back in the output
) # Generate a completion, can also call create_completionprint(output)

By default llama-cpp-python generates completions in an OpenAI compatible format:

{
"id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"object": "text_completion",
"created": 1679561337,
"model": "./models/7B/llama-model.gguf",
"choices": [
{
"text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
"index": 0,
"logprobs": None,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 14,
"completion_tokens": 28,
"total_tokens": 42
}
}

Text completion is available through the __call__ and create_completion methods of the Llama class.

Pulling models from Hugging Face Hub

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method.

You'll need to install the huggingface_hub package to use this feature (pip install --upgrade huggingface_hub).

llm=Llama.from_pretrained(
repo_id="Qwen/Qwen2.5-0.5B-Instruct-GGUF",
filename="qwen2.5-0.5b-instruct-q4_k_m.gguf",
verbose=False
)

By default from_pretrained will download the model to the huggingface cache directory, you can then manage installed model files with the hf tool.

Chat Completion

The high-level API also provides a simple interface for chat completion.

Chat completion requires that the model knows how to format the messages into a single prompt. The Llama class does this using pre-registered chat formats (ie. chatml, llama-2, gemma, etc) or by providing a custom chat handler object.

The model will will format the messages into a single prompt using the following order of precedence:

Use the chat_handler if provided
Use the chat_format if provided
Use the tokenizer.chat_template from the gguf model's metadata (should work for most new models, older models may not have this)
else, fallback to the llama-2 chat format

Set verbose=True to see the selected chat format.

fromllama_cppimportLlamallm=Llama(
model_path="path/to/llama-2/llama-model.gguf",
chat_format="llama-2"
)
llm.create_chat_completion(
messages= [
{"role": "system", "content": "You are an assistant who perfectly describes images."},
{
"role": "user",
"content": "Describe this image in detail please."
}
]
)

Chat completion is available through the create_chat_completion method of the Llama class.

For OpenAI API v1 compatibility, you use the create_chat_completion_openai_v1 method which will return pydantic models instead of dicts.

JSON and JSON Schema Mode

To constrain chat responses to only valid JSON or a specific JSON Schema use the response_format argument in create_chat_completion.

JSON Mode

The following example will constrain the response to valid JSON strings only.

fromllama_cppimportLlamallm=Llama(model_path="path/to/model.gguf", chat_format="chatml")
llm.create_chat_completion(
messages=[
{
"role": "system",
"content": "You are a helpful assistant that outputs in JSON.",
},
{"role": "user", "content": "Who won the world series in 2020"},
],
response_format={
"type": "json_object",
},
temperature=0.7,
)

JSON Schema Mode

To constrain the response further to a specific JSON Schema add the schema to the schema property of the response_format argument.

fromllama_cppimportLlamallm=Llama(model_path="path/to/model.gguf", chat_format="chatml")
llm.create_chat_completion(
messages=[
{
"role": "system",
"content": "You are a helpful assistant that outputs in JSON.",
},
{"role": "user", "content": "Who won the world series in 2020"},
],
response_format={
"type": "json_object",
"schema": {
"type": "object",
"properties": {"team_name": {"type": "string"}},
"required": ["team_name"],
},
},
temperature=0.7,
)

Function Calling

The high-level API supports OpenAI compatible function and tool calling. This is possible through the functionary pre-trained models chat format or through the generic chatml-function-calling chat format.

fromllama_cppimportLlamallm=Llama(model_path="path/to/chatml/llama-model.gguf", chat_format="chatml-function-calling")
llm.create_chat_completion(
messages= [
{
"role": "system",
"content": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. The assistant calls functions with appropriate input when necessary"
},
{
"role": "user",
"content": "Extract Jason is 25 years old"
}
],
tools=[{
"type": "function",
"function": {
"name": "UserDetail",
"parameters": {
"type": "object",
"title": "UserDetail",
"properties": {
"name": {
"title": "Name",
"type": "string"
},
"age": {
"title": "Age",
"type": "integer"
}
},
"required": [ "name", "age" ]
}
}
}],
tool_choice={
"type": "function",
"function": {
"name": "UserDetail"
}
}
)

Functionary v2

The various gguf-converted files for this set of models can be found here. Functionary is able to intelligently call functions and also analyze any provided function outputs to generate coherent responses. All v2 models of functionary supports parallel function calling. You can provide either functionary-v1 or functionary-v2 for the chat_format when initializing the Llama class.

Due to discrepancies between llama.cpp and HuggingFace's tokenizers, it is required to provide HF Tokenizer for functionary. The LlamaHFTokenizer class can be initialized and passed into the Llama class. This will override the default llama.cpp tokenizer used in Llama class. The tokenizer files are already included in the respective HF repositories hosting the gguf files.

fromllama_cppimportLlamafromllama_cpp.llama_tokenizerimportLlamaHFTokenizerllm=Llama.from_pretrained(
repo_id="meetkai/functionary-small-v2.2-GGUF",
filename="functionary-small-v2.2.q4_0.gguf",
chat_format="functionary-v2",
tokenizer=LlamaHFTokenizer.from_pretrained("meetkai/functionary-small-v2.2-GGUF")
)

NOTE: There is no need to provide the default system messages used in Functionary as they are added automatically in the Functionary chat handler. Thus, the messages should contain just the chat messages and/or system messages that provide additional context for the model (e.g.: datetime, etc.).

Continuing Assistant Responses (Prefill)

llama-cpp-python supports native Assistant Prefill for seamless message continuation. You can now simply use the assistant_prefill=True parameter in the create_chat_completion function.

This safely renders the N-1 conversation history using standard Jinja templates (preserving exact control tokens) and flawlessly appends your partial text directly to the prompt.

fromllama_cppimportLlamallm=Llama(model_path="path/to/model.gguf")
# An interrupted/partial conversationmessages= [
{"role": "user", "content": "What are the first 5 planets in the solar system?"},
{"role": "assistant", "content": "The first 5 planets in our solar system are:\n1. Mercury\n2."}
]
# Seamlessly continue the generationresponse=llm.create_chat_completion(
messages=messages,
max_tokens=50,
assistant_prefill=True# <--- Enables seamless continuation
)
prefilled_text=messages[-1]["content"]
# The model will flawlessly continue from " Venus\n3. Earth..."generated_text=response["choices"][0]["message"]["content"]
print(prefilled_text+generated_text)

Dynamic LoRA Routing & Control Vectors (Multi-Tenant Serving)

Historically, llama-cpp-python only supported "static loading" where a LoRA was permanently baked into the context during initialization. Switching personas required reloading the entire model or duplicating it in VRAM.

llama-cpp-python now supports Just-In-Time (JIT) dynamic adapter routing. Instead of statically binding a single LoRA to a model during initialization (which locks the instance to a single task), you can now preload multiple adapters into VRAM and seamlessly apply them on-the-fly per request.

This architecture unlocks true Multi-Tenant Serving:

Zero-Latency Switching: Compute graph weights are atomically modified in C++ memory instantly before evaluation.
VRAM Efficiency: You only load the heavy base model once. Multiple LoRAs share the same base model memory.
Thread-Safe & Contamination-Free: Strict internal state debouncing ensures that weights are perfectly cleaned between requests, guaranteeing zero persona contamination.

Dynamic LoRA Example

fromllama_cppimportLlama# 1. Load the pure base model oncellm=Llama(model_path="path/to/llama-3-8b.gguf")
# 2. Preload multiple LoRAs into VRAMllm.load_lora("python_coder", "path/to/python-coder-lora.gguf")
llm.load_lora("translator", "path/to/spanish-translator-lora.gguf")
# 3. User A: Coding Task (Instantly applies the coder LoRA)response_a=llm.create_chat_completion(
messages=[{"role": "user", "content": "Write a fast inverse square root in C."}],
active_loras=[{"name": "python_coder", "scale": 1.0}]
)
# 4. User B: Translation Task (Zero-latency switch to the translator LoRA)response_b=llm.create_chat_completion(
messages=[{"role": "user", "content": "Explain quantum physics in Spanish."}],
active_loras=[{"name": "translator", "scale": 0.85}] # Apply at 85% strength
)
# 5. User C: General Query (Automatically wipes graph weights for a clean base model state)response_c=llm.create_chat_completion(
messages=[{"role": "user", "content": "What is the capital of France?"}]
)
# 6. Cleanup (Optional: manually free VRAM for specific LoRAs)llm.unload_lora("python_coder")

Control Vector Injection (Representation Engineering)

In addition to LoRA, the API supports dynamic injection of Control Vectors (CVec). This allows you to steer the model's behavior, emotion, or alignment by directly modifying the activation values at specific hidden layers, without needing .gguf weight files.

response=llm.create_chat_completion(
messages=[{"role": "user", "content": "Tell me a story about a futuristic city."}],
control_vector={
"data": [...], # A flattened 1D list of floats representing the vector"layer_start": 15, # Apply starting from this layer (inclusive)"layer_end": 32# Apply up to this layer (inclusive)
}
)

Note(JamePeng): Ensure your data array length exactly matches embedding_length * layer_end. The C++ backend maps the buffer continuously starting from layer 1, so early skipped layers must be zero-padded in your array.

Sampling Configuration & Usage (LlamaSamplingParams)

The Llama class provides extensive control over the llama.cpp sampling chain during text generation. You can configure state-of-the-art sampling algorithms, dynamic temperature, and advanced repetition penalties directly via the generate, create_completion, or __call__ methods.

Core Sampling Parameters

These are the most common parameters used to control the randomness and focus of the model's output.

temperature (float, default: 0.80): Controls the randomness of the generation. Higher values (e.g., 1.0) make output more random, while lower values (e.g., 0.2) make it more deterministic. Set to <= 0.0 for greedy decoding.
top_k (int, default: 40): Limits the next token selection to the K most probable tokens. Set to <= 0 to use the full vocabulary size.
top_p (float, default: 0.95): Nucleus sampling. Limits selection to a cumulative probability of P. Set to 1.0 to disable.
min_p (float, default: 0.05): Minimum P sampling. Drops tokens with a probability less than min_p relative to the most likely token. Set to 0.0 to disable.
typical_p (float, default: 1.0): Locally typical sampling. Adjusts probabilities based on the entropy of the distribution. Set to 1.0 to disable.

Advanced & Experimental Samplers

XTC (Exclude Top Choice): Removes the most likely tokens to force the model to take creative alternative paths.
xtc_probability (float, default: 0.0): The chance for token removal. 0.0 disables XTC.
xtc_threshold (float, default: 0.1): The minimum probability threshold for a token to be considered for removal.
Dynamic Temperature: Adjusts the temperature dynamically based on the entropy of the current token distribution.
dynatemp_range (float, default: 0.0): The range of the dynamic temperature. 0.0 disables it.
dynatemp_exponent (float, default: 1.0): Controls how entropy maps to temperature.
top_n_sigma (float, default: -1.0): Limits selection to tokens with pre-softmax logits within $n * \sigma$ of the max logit. Set to -1.0 to disable.
Adaptive-P: Dynamically adjusts the target probability using an exponential moving average (EMA).
adaptive_target (float, default: -1.0): The target probability (0.0 to 1.0). Negative values disable it.
adaptive_decay (float, default: 0.9): The EMA decay rate (0.0 to 0.99).

Target Entropy (Mirostat)

Mirostat actively maintains a target entropy (tau) during generation to prevent text from becoming too boring or too chaotic.

mirostat_mode (int, default: 0): 0 = disabled, 1 = Mirostat 1.0, 2 = Mirostat 2.0.
mirostat_tau (float, default: 5.0): The target cross-entropy (surprisal) value.
mirostat_eta (float, default: 0.1): The learning rate used to update the algorithm's internal state.

Repetition Penalties

Standard Penalties:
repeat_penalty (float, default: 1.0): General penalty for repeated tokens. 1.0 = disabled.
frequency_penalty (float, default: 0.0): Penalty based on the absolute frequency of a token in the prompt.
present_penalty (float, default: 0.0): Flat penalty applied if a token is present anywhere in the context.
penalty_last_n (int, default: 64): The number of recent tokens to consider for standard penalties. 0 = disabled, -1 = full context size.
DRY (Don't Repeat Yourself): An advanced exponential penalty specifically designed to break exact repeating sequences.
dry_multiplier (float, default: 0.0): The multiplier for the penalty. 0.0 disables DRY.
dry_base (float, default: 1.75): The base value for the exponential penalty.
dry_allowed_length (int, default: 2): Sequences extending beyond this length receive the penalty.
dry_penalty_last_n (int, default: 0): Tokens to scan for repetitions. 0 = disabled, -1 = context size.
dry_seq_breakers (list[str], default: ["\n", ":", "\"", "*"]): Tokens that reset the DRY sequence matching.

Constraints & Callbacks

logit_bias (Dict[int, float], optional): Manually boost or penalize specific token IDs.
grammar (LlamaGrammar, optional): Force the model to generate text matching a specific BNF-like grammar (e.g., valid JSON).
logits_processor (LogitsProcessorList, optional): Custom Python callbacks to modify the logits tensor in-place before sampling.
stopping_criteria (StoppingCriteriaList, optional): Custom Python callbacks to halt generation based on the current sequence or scores.

Reasoning Budget (First Reasoning Block)

llama-cpp-python provides a generic reasoning-budget sampler for models that expose their thinking content with visible start/end tags. It controls only the first visible reasoning block in the generated output. After that block naturally ends or is forcibly closed, the sampler switches to passthrough mode and later reasoning tags are ignored.

This feature is intentionally model-agnostic. It does not infer model families, inspect chat templates, or guess thinking tags. If a model uses tags other than <think>...</think>, pass the correct reasoning_start and reasoning_end explicitly.

Parameter	Default	Description
`reasoning_budget`	`-1`	Token budget for the first visible reasoning block. `-1` disables the sampler, `0` forces an immediate end after the block starts, and `N > 0` allows at most `N` generated tokens inside the block.
`reasoning_start`	`"<think>"`	Token/text sequence that marks the beginning of the first reasoning block.
`reasoning_end`	`"</think>"`	Token/text sequence that naturally ends the reasoning block. When the budget is exhausted, the sampler forces this sequence.
`reasoning_budget_message`	`None`	Optional message inserted before `reasoning_end` when the budget is exhausted.
`reasoning_start_in_prompt`	`False`	Set to `True` only when the prompt/chat template has already inserted `reasoning_start`, so the sampler should start counting from the first generated token.
`reasoning_start_max_tokens`	`32`	Safety window for non-reasoning outputs. If `reasoning_start` is not generated within this many output tokens, the sampler becomes a no-op. Set to `None` to wait indefinitely.

Basic usage with the default <think>...</think> tags:

response=llm.create_chat_completion(
messages=[{"role": "user", "content": "Solve this carefully."}],
max_tokens=1024,
reasoning_budget=256,
reasoning_budget_message="\n[reasoning budget exhausted]\n",
# You can also inject a natural-language transition before reasoning_end:# reasoning_budget_message="\n...Wait, I have been thinking long enough. Let me start answering the user's question.\n",
)

When the budget is exhausted, the sampler forces: reasoning_budget_message + reasoning_end

For Mistral-style thinking tags, pass the tags explicitly:

response=llm.create_chat_completion(
messages=[{"role": "user", "content": "Solve this carefully."}],
max_tokens=1024,
reasoning_budget=256,
reasoning_start="[THINK]",
reasoning_end="[/THINK]",
)

For Gemma4 channel-style thinking, adjust the start and end markers to match the visible channel tags:

response=llm.create_chat_completion(
messages=[{"role": "user", "content": "Solve this carefully."}],
max_tokens=1024,
reasoning_budget=256,
reasoning_start="<|channel>",
reasoning_end="<channel|>",
)

Use reasoning_start_in_prompt=True when the prompt or chat template has already inserted the reasoning start tag. In that case, the sampler will not see the start tag during generation, so it must start directly in COUNTING state from the first generated token. This is suitable for thinking models or handlers that prefill the assistant prefix with a thinking tag, for example:

<|im_start|>assistant\n<think>\n

Example:

response=llm.create_chat_completion(
messages=[{"role": "user", "content": "Solve this carefully."}],
max_tokens=1024,
reasoning_budget=256,
reasoning_start="<think>",
reasoning_end="</think>",
reasoning_start_in_prompt=True,
)

reasoning_start_in_prompt is not a generic "thinking enabled" switch. It should only be set when the final prompt already contains reasoning_start before generation begins. For templates that merely enable thinking but still expect the model to generate the start tag itself, keep reasoning_start_in_prompt=False.

When verbose=True, high-level reasoning-budget transitions are printed to stderr, such as initialization, start-tag detection, budget exhaustion, forced ending, and final passthrough.

🛠️ Usage Example

You can pass these parameters directly when calling the model to generate text.

fromllama_cppimportLlama# Load the modelmodel=Llama(model_path="path/to/your/model.gguf")
# Generate text with advanced samplingresponse=model.create_completion(
prompt="The secret to a happy life is",
max_tokens=100,
# Adjust core randomnesstemperature=0.85,
top_p=0.90,
min_p=0.05,
# Prevent the model from repeating specific phrasesdry_multiplier=0.8,
dry_base=1.75,
dry_allowed_length=3,
# Standard repetition penaltyrepeat_penalty=1.1,
penalty_last_n=256,
)
print(response["choices"][0]["text"])

Multi-modal Models

llama-cpp-python supports such as llava1.5 which allow the language model to read information from both text and images.

Below are the supported multi-modal models and their respective chat handlers (Python API) and chat formats (Server API).

Model	`LlamaChatHandler`	`chat_format`
llava-v1.5-7b	`Llava15ChatHandler`	`llava-1-5`
llava-v1.6-34b	`Llava16ChatHandler`	`llava-1-6`
moondream2	`MoondreamChatHandler`	`moondream2`
nanollava	`NanollavaChatHandler`	`nanollava`
llama-3-vision-alpha	`Llama3VisionAlphaChatHandler`	`llama-3-vision-alpha`
minicpm-v-2.6	`MiniCPMv26ChatHandler`	`minicpm-v-2.6`, `minicpm-v-4.0`
minicpm-v-4.5	`MiniCPMv45ChatHandler`	`minicpm-v-4.5`
minicpm-v-4.6	`MiniCPMv46ChatHandler`	`minicpm-v-4.6`
gemma3	`Gemma3ChatHandler`	`gemma3`
gemma4	`Gemma4ChatHandler`	`gemma4`
glm4.1v	`GLM41VChatHandler`	`glm4.1v`
glm4.6v	`GLM46VChatHandler`	`glm4.6v`
granite-docling	`GraniteDoclingChatHandler`	`granite-docling`
lfm2-vl	`LFM2VLChatHandler`	`lfm2-vl`
lfm2.5-vl	`LFM25VLChatHandler`	`lfm2.5-vl`
deepseek-ocr	`MTMDChatHandler`	`None`
mineru2.5-pro	`Qwen25VLChatHandler`	`qwen2.5-vl`
paddleocr-vl-1.5	`PaddleOCRChatHandler`	`paddleocr`
qwen2.5-vl	`Qwen25VLChatHandler`	`qwen2.5-vl`
qwen3-asr	`Qwen3ASRChatHandler`	`qwen3-asr`
qwen3-vl	`Qwen3VLChatHandler`	`qwen3-vl`
qwen3.5	`Qwen35ChatHandler`	`qwen3.5`
qwen3.6	`Qwen35ChatHandler`	`qwen3.6`
step3-vl	`Step3VLChatHandler`	`step3-vl`

Then you'll need to load the multimodal projection model (mmproj) together with the main language model.

Starting from 0.3.41-preview, new multimodal implementations are recommended to use the updated interfaces in llama_multimodal. For backward compatibility, the legacy llama_chat_format path is still retained, but may be deprecated in future versions.

The parameter clip_model_path has been renamed to mmproj_path to better reflect its purpose and align with llama.cpp's multimodal projection model naming convention. New code should use mmproj_path exclusively.

Generic MTMD Chat Handler

For multimodal GGUF models that already include a valid tokenizer.chat_template, you can use the generic MTMD handler through mmproj_path.

This is especially useful for newer multimodal models that have not yet received a dedicated Python chat handler. The generic handler renders the model-provided Jinja chat template, then normalizes rendered media placeholders or media URLs into the canonical llama.cpp MTMD media marker, usually <__media__>, before calling mtmd_tokenize.

Note:GenericMTMDChatHandler is intended as a flexible fallback for template-driven multimodal models. Because different model families may use different media ordering rules, reasoning switches, stop tokens, or special template variables, some models may still require a dedicated chat handler. Please test carefully and report issues if you encounter incorrect prompts, missing media markers, or mismatched media counts.

fromllama_cppimportLlama# Model and multimodal projection pathsMODEL_PATH=r"path/to/model.gguf"MMPROJ_PATH=r"path/to/mmproj.gguf"llm=Llama(
model_path=MODEL_PATH,
mmproj_path=MMPROJ_PATH,
n_gpu_layers=-1,
n_ctx=10240,
verbose=True,
verbosity=2,
chat_handler_kwargs={
"verbose": True,
},
)
response=llm.create_chat_completion(
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "path/to/image.jpg",
},
},
{
"type": "text",
"text": "Describe this image in detail.",
},
],
}
]
)
print(response["choices"][0]["message"]["content"])

Chat Template Resolution Order

GenericMTMDChatHandler resolves the chat template in the following order:

Use the explicit chat_format passed through chat_handler_kwargs, if provided.
Use the named model chat template if chat_template_name is provided.
Fall back to the default tokenizer.chat_template stored in the GGUF model metadata.
Fall back to the built-in MTMD chat template if no model template is available.

Example using a named chat template:

llm=Llama(
model_path=r"path/to/model.gguf",
mmproj_path=r"path/to/mmproj.gguf",
# chat_template_name="default",n_gpu_layers=-1,
n_ctx=4096,
chat_handler_kwargs={
"verbose": False,
},
)

Passing Extra Template Arguments

Some model chat templates expose optional Jinja variables such as enable_thinking, add_vision_id, or model-specific media token switches. Further details can be obtained by analyzing the chat templates provided in chat_template.jinja or tokenizer_config.json for each model.

You can pass those values through chat_handler_kwargs["extra_template_arguments"]:

fromllama_cppimportLlama# Model and multimodal projection pathsMODEL_PATH=r"path/to/model.gguf"MMPROJ_PATH=r"path/to/mmproj.gguf"llm=Llama(
model_path=MODEL_PATH,
mmproj_path=MMPROJ_PATH,
n_gpu_layers=-1,
n_ctx=10240,
verbose=False,
verbosity=1,
chat_handler_kwargs={
"extra_template_arguments": {
"enable_thinking": True,
},
"verbose": False,
},
)
...

The values inside extra_template_arguments are passed directly into the Jinja template render call.

For models that already have a dedicated handler, you can still instantiate that handler directly:

fromllama_cppimportLlamafromllama_cpp.llama_multimodalimportPaddleOCRChatHandlerMODEL_PATH=r"path/to/model.gguf"MMPROJ_PATH=r"path/to/mmproj.gguf"llm=Llama(
model_path=MODEL_PATH,
chat_handler=PaddleOCRChatHandler(
mmproj_path=MMPROJ_PATH,
),
n_gpu_layers=-1, # Use all available GPU layersn_ctx=0, # Context window sizen_batch=2048,
)
...

Use GenericMTMDChatHandler when the model-provided tokenizer.chat_template already works correctly. Prefer a dedicated handler when the model requires custom prompt construction, special reasoning behavior, custom stop tokens, OCR/ASR-specific handling, or non-standard media ordering.

Note: Multi-modal models also support tool calling and JSON mode.

Loading a Local Image With Qwen3VL(Thinking/Instruct)

This script demonstrates how to load a local image, encode it as a base64 Data URI, and pass it to a local Qwen3-VL model (with the 'force_reasoning' parameter enabled for thinking model, disabled for instruct model) for processing using the llama-cpp-python library.

Example Code:

Details

# Import necessary librariesfromllama_cppimportLlama# from llama_cpp.llama_chat_format import Qwen3VLChatHandlerfromllama_cpp.llama_multimodalimportQwen3VLChatHandlerimportbase64importos# --- Model Configuration ---# Define the path to the main model fileMODEL_PATH=r"./Qwen3-VL-8B-Thinking-F16.gguf"# Define the path to the multi-modal projector fileMMPROJ_PATH=r"./mmproj-Qwen3-VL-8b-Thinking-F16.gguf"# --- Initialize the Llama Model ---llm=Llama(
model_path=MODEL_PATH,
# Set up the chat handler for Qwen3-VL, specifying the projector pathchat_handler=Qwen3VLChatHandler(
clip_model_path=MMPROJ_PATH,
force_reasoning=True, # Note: Some models use `enable_thinking` as a switch variable. See the comments in the corresponding model's chathandler for details.image_min_tokens=1024, # Note: Qwen3-VL models require at minimum 1024 image tokens to function correctly on bbox grounding tasks
),
n_gpu_layers=-1, # Offload all layers to the GPUn_ctx=10240, # Set the context window sizeswa_full=True,
)
# Comprehensive MIME type mapping (updated as of 2025)# Based on Pillow 10.x+ "Fully Supported" (Read & Write) formats# Reference: IANA official media types + common real-world usage# See: https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html_IMAGE_MIME_TYPES= {
# Most common formats'.png': 'image/png',
'.jpg': 'image/jpeg',
'.jpeg': 'image/jpeg',
'.gif': 'image/gif',
'.webp': 'image/webp',
# Next-generation formats'.avif': 'image/avif',
'.jp2': 'image/jp2',
'.j2k': 'image/jp2',
'.jpx': 'image/jp2',
# Legacy / Windows formats'.bmp': 'image/bmp',
'.ico': 'image/x-icon',
'.pcx': 'image/x-pcx',
'.tga': 'image/x-tga',
'.icns': 'image/icns',
# Professional / Scientific imaging'.tif': 'image/tiff',
'.tiff': 'image/tiff',
'.eps': 'application/postscript',
'.dds': 'image/vnd-ms.dds',
'.dib': 'image/dib',
'.sgi': 'image/sgi',
# Portable Map formats (PPM/PGM/PBM)'.pbm': 'image/x-portable-bitmap',
'.pgm': 'image/x-portable-graymap',
'.ppm': 'image/x-portable-pixmap',
# Miscellaneous / Older formats'.xbm': 'image/x-xbitmap',
'.mpo': 'image/mpo',
'.msp': 'image/msp',
'.im': 'image/x-pillow-im',
'.qoi': 'image/qoi',
}
defimage_to_base64_data_uri(
file_path: str,
*,
fallback_mime: str="application/octet-stream"
) ->str:
""" Convert a local image file to a base64-encoded data URI with the correct MIME type. Supports 20+ image formats (PNG, JPEG, WebP, AVIF, BMP, ICO, TIFF, etc.). Args: file_path: Path to the image file on disk. fallback_mime: MIME type used when the file extension is unknown. Returns: A valid data URI string (e.g., data:image/webp;base64,...). Raises: FileNotFoundError: If the file does not exist. OSError: If reading the file fails. """ifnotos.path.isfile(file_path):
raiseFileNotFoundError(f"Image file not found: {file_path}")
extension=os.path.splitext(file_path)[1].lower()
mime_type=_IMAGE_MIME_TYPES.get(extension, fallback_mime)
ifmime_type==fallback_mime:
print(f"Warning: Unknown extension '{extension}' for '{file_path}'. "f"Using fallback MIME type: {fallback_mime}")
try:
withopen(file_path, "rb") asimg_file:
encoded_data=base64.b64encode(img_file.read()).decode("utf-8")
exceptOSErrorase:
raiseOSError(f"Failed to read image file '{file_path}': {e}") fromereturnf"data:{mime_type};base64,{encoded_data}"# --- Main Logic for Image Processing ---# 1. Create a list containing all image pathsimage_paths= [
r'./scene.jpeg',
r'./cat.png',
r'./network.webp',
# Add more image paths here if needed
]
# 2. Create an empty list to store the message objects (images and text)images_messages= []
# 3. Loop through the image path list, convert each image to a Data URI,# and add it to the message list as an image_url object.forpathinimage_paths:
data_uri=image_to_base64_data_uri(path)
images_messages.append({"type": "image_url", "image_url": {"url": data_uri}})
# 4. Add the final text prompt at the end of the listimages_messages.append({"type": "text", "text": "Describes the images."})
# 5. Use this list to build the chat_completion requestres=llm.create_chat_completion(
messages=[
{"role": "system", "content": "You are a highly accurate vision-language assistant. Provide detailed, precise, and well-structured image descriptions."},
# The user's content is the list containing both images and text
{"role": "user", "content": images_messages}
]
)
# Print the assistant's responseprint(res["choices"][0]["message"]["content"])

Speech Recognition With Qwen3-ASR (Speech-to-Text)

The Qwen3ASRChatHandler is specifically designed for the Qwen3 Automatic Speech Recognition (ASR) models. Unlike standard multimodal models, this handler aggregates system prompts for instructions and automatically extracts audio data from the user's message, ignoring any user text.

⚠️ Important Note on Quantization: > For Qwen3-ASR models, it is highly recommended to use the BF16 version of the multimodal projector (mmproj). Other quantizations are known to cause severe audio degradation.

Example Code:

Details

fromllama_cppimportLlama# from llama_cpp.llama_chat_format import Qwen3ASRChatHandlerfromllama_cpp.llama_multimodalimportQwen3ASRChatHandlerimportbase64importos# 1. Define paths to the model and the BF16 multimodal projectorMODEL_PATH=r"./Qwen3-ASR-1.7B-BF16.gguf"MMPROJ_PATH=r"./mmproj-Qwen3-ASR-1.7b-BF16.gguf"# 2. Initialize the Llama model with the dedicated ASR handlerllm=Llama(
model_path=MODEL_PATH,
chat_handler=Qwen3ASRChatHandler(
clip_model_path=MMPROJ_PATH,
verbose=False,
),
n_gpu_layers=-1,
n_ctx=10240,
verbose=False,
verbosity=0
)
# 3. Helper function to encode audio files into OpenAI-compatible payloads_MEDIA_MIME_TYPES= {
'.wav': ('audio', 'wav'),
'.mp3': ('audio', 'mp3'),
}
defbuild_media_payload(file_path: str) ->dict:
"""Reads a local audio file and converts it into the LLM input structure."""ifnotos.path.isfile(file_path):
raiseFileNotFoundError(f"Media file not found: {file_path}")
extension=os.path.splitext(file_path)[1].lower()
media_category, mime_or_format=_MEDIA_MIME_TYPES.get(extension, ('unknown', 'application/octet-stream'))
ifmedia_category=='unknown':
print(f"Warning: Unknown extension '{extension}'.")
# Read and Base64 encode the filewithopen(file_path, "rb") asf:
encoded_data=base64.b64encode(f.read()).decode("utf-8")
ifmedia_category=='audio':
return {
"type": "input_audio",
"input_audio": {
"data": encoded_data,
"format": mime_or_format
}
}
else:
return {"type": "text", "text": f"[Attached unsupported file: {file_path}]"}
# ========================# Main Inference Section# ========================media_paths= ["./audio/test.wav"]
user_content= [build_media_payload(path) forpathinmedia_paths]
# 4. Generate the transcriptionresponse=llm.create_chat_completion(
messages=[
{
"role": "system",
"content": (
"You are an advanced multilingual Speech-to-Text model. ""Accurately transcribe the audio into text in its original spoken language. ""You should ignore background noise, filler words, and stutters where possible, ""and format the final output with correct grammar and capitalization."
)
},
{
"role": "user",
"content": user_content
}
],
temperature=1.0,
top_p=0.95,
top_k=64,
max_tokens=10240,
)
print(f"Transcribe: {response['choices'][0]['message']['content']}")

How it works:

input_audio Schema: The script reads the local .wav or .mp3 file, encodes it in Base64, and wraps it in an OpenAI-compatible "type": "input_audio" dictionary.
System Prompt: Because the Qwen3-ASR template strips out user text, all instructions (like translation requests or formatting rules) must be placed in the "system" role.

Comprehensive Omni MultiModal Example: Gemma-4 (Vision + Audio + Text)

Below is a complete, production-ready example demonstrating how to dynamically route and process both image and audio files. It includes a universal media processor that automatically converts local files into the correct payload structure (Data URIs for images, and input_audio for audio files).

⚠️ IMPORTANT: GEMMA-4 MODEL CAPABILITIES & LIMITATIONS
Gemma4 E2B / E4B: Supports Full Multimodal (Vision + Audio + Text). enable_thinkingMUST be True(default).
Gemma4 31B / 26BA4B: Supports Vision + Text ONLY (Audio is NOT supported). enable_thinking can be toggled (True or False).

fromllama_cppimportLlama# from llama_cpp.llama_chat_format import Gemma4ChatHandlerfromllama_cpp.llama_multimodalimportGemma4ChatHandlerimportbase64importos# Model and multimodal projection pathsMODEL_PATH=r"/path/to/Gemma-4-E4B-It-BF16.gguf"# BF16 mmproj is required for audio. Other quantizations are known to have degraded performance.MMPROJ_PATH=r"/path/to/mmproj-Gemma-4-E4B-It-BF16.gguf"# Initialize the Llama model with multimodal support# Note: Since we are using E4B here, enable_thinking MUST be True, and audio is supported.llm=Llama(
model_path=MODEL_PATH,
chat_handler=Gemma4ChatHandler(
clip_model_path=MMPROJ_PATH,
enable_thinking=True, # MUST be True for E2B/E4B modelsverbose=True, # Enable Debug Info
),
n_gpu_layers=-1,
n_ctx=10240,
verbose=True, # Enable Debug Info
)
# 1. Extend the MIME dictionary to support audio formats_MEDIA_MIME_TYPES= {
# ------ Image formats ------'.png': ('image', 'image/png'),
'.jpg': ('image', 'image/jpeg'),
'.jpeg': ('image', 'image/jpeg'),
'.gif': ('image', 'image/gif'),
'.webp': ('image', 'image/webp'),
'.bmp': ('image', 'image/bmp'),
# ------ Audio formats ------'.wav': ('audio', 'wav'), # OpenAI standard usually uses raw format names for audio'.mp3': ('audio', 'mp3'),
# '.flac': ('audio', 'flac'),
}
defbuild_media_payload(file_path: str) ->dict:
""" Read a local media file (image or audio) and convert it into a valid input payload for the LLM. """ifnotos.path.isfile(file_path):
raiseFileNotFoundError(f"Media file not found: {file_path}")
extension=os.path.splitext(file_path)[1].lower()
media_category, mime_or_format=_MEDIA_MIME_TYPES.get(extension, ('unknown', 'application/octet-stream'))
ifmedia_category=='unknown':
print(f"Warning: Unknown extension '{extension}'. It might not be processed correctly.")
# Read and Base64 encode the filewithopen(file_path, "rb") asf:
encoded_data=base64.b64encode(f.read()).decode("utf-8")
# 2. Return the appropriate dictionary structure based on the media typeifmedia_category=='image':
# Image format: Data URI (OpenAI compatible)data_uri=f"data:{mime_or_format};base64,{encoded_data}"return {
"type": "image_url",
"image_url": {"url": data_uri}
}
elifmedia_category=='audio':
# Audio format: input_audio (OpenAI compatible)return {
"type": "input_audio",
"input_audio": {
"data": encoded_data,
"format": mime_or_format
}
}
else:
# Fallback for unsupported formatsreturn {"type": "text", "text": f"[Attached unsupported file: {file_path}]"}
defrun_inference(media_paths: list, text_prompt: str):
""" Helper function to dynamically build the payload and run inference. """# 3. Build the user_content listuser_content= []
# Automatically parse each file and append to the payloadforpathinmedia_paths:
payload=build_media_payload(path)
user_content.append(payload)
# Append the final text instructionuser_content.append({
"type": "text",
"text": text_prompt
})
print(f"\n--- Running Inference with {len(media_paths)} media file(s) ---")
# 4. Send to the model for inferenceresponse=llm.create_chat_completion(
messages=[
{"role": "system", "content": """ You are a highly capable multimodal assistant that can process both text, vision and audio. """}, # Note: Supported ONLY by Gemma4 E2B / E4B.
{"role": "user", "content": user_content}
],
temperature=1.0,
top_p=0.95,
top_k=64,
max_tokens=8192,
)
print("\n[Model Response]:")
print(response["choices"][0]["message"]["content"])
print("-"*60)
# ==============================================================================# Main Inference Examples# Uncomment the example block you wish to execute.# ==============================================================================# --- Example A: Image + Audio (Full Multimodal) ---# Note: Supported ONLY by Gemma4 E2B / E4B.run_inference(
media_paths=[r"/path/to/test.png", r"/path/to/test.wav"],
text_prompt="Introduce the content by combining the images and converting the audio to text."
)
# --- Example B: Image Only (Vision + Text) ---# Note: Supported by all Gemma4 variants (E2B, E4B, 31B, 26BA4B).# run_inference(# media_paths=[r"/path/to/test.png"],# text_prompt="Describe the contents of this image in detail."# )# --- Example C: Audio Only (Audio + Text) ---# Note: Supported ONLY by Gemma4 E2B / E4B.# run_inference(# media_paths=[r"/path/to/test.wav"],# text_prompt="Transcribe this audio and summarize the main points."# )

Embeddings & Reranking (GGUF)

llama-cpp-python provides a high-performance, memory-efficient specialized class LlamaEmbedding for generating text embeddings and calculating reranking scores.

Key Features:

Streaming Batch Processing: Process massive datasets (e.g., Hundreds of documents) without running out of memory (OOM).
Native Reranking: Built-in support for Cross-Encoder models (outputting relevance scores instead of vectors).
Optimized Performance: Utilizes Unified KV Cache for parallel encoding of multiple documents.
Chat Template Support: Support for rerank templates has been introduced (via llama_model_chat_template(model, b"rerank")), which can automatically populate the query and document into a specific format.

Support Embeddings & Rerank Model:

Model	Type	Link	Status
`bge-m3`	Embedding	bge-m3-GGUF	Useful ✅
`jina-embeddings-v2-base-zh`	Embedding	jina-embeddings-v2-base-zh-GGUF	Useful ✅
`jina-embeddings-v3`	Embedding	jina-embeddings-v3-GGUF	Useful ✅
`bge-reranker-v2-m3`	Rerank	bge-reranker-v2-m3-GGUF	Useful ✅
`qwen3-reranker`	Rerank	Qwen3-Reranker-GGUF	Useful ✅

TODO(JamePeng): Needs more extensive testing with various embedding and rerank models. :)

1. Text Embeddings (Vector Search)

To generate embeddings, use the LlamaEmbedding class. It automatically configures the model for vector generation.

fromllama_cpp.llama_embeddingimportLlamaEmbedding, LLAMA_POOLING_TYPE_NONE# Initialize the model (automatically sets embeddings=True)llm=LlamaEmbedding(
model_path="path/to/bge-m3.gguf",
n_gpu_layers=-1,
pooling_type=LLAMA_POOLING_TYPE_NONE,
n_seq_max=128, # Maximum independent sequences in one decode batch
)
# 1. Simple usage (OpenAI-compatible format)response=llm.create_embedding("Hello, world!")
print(response['data'][0]['embedding'])
# 2. Batch processing (High Performance)# You can pass a large list of strings; the streaming batcher handles memory automatically.documents= ["Hello, world!", "Goodbye, world!", "Llama is cute."] *100embeddings=llm.embed(documents) # Returns a list of lists (vectors)print(f"Generated {len(embeddings)} vectors.")

Parallel batch capacity:n_seq_max controls how many independent sequence IDs may coexist in one decode batch; it is not the total number of documents accepted by embed(). For batch embedding, set it high enough for the number of short documents that can fit within n_batch. If an error says seq_id=1 exceeds n_seq_max=1, initialize the model with at least n_seq_max=2. For example, use n_seq_max=8 for up to eight parallel sequences. Larger values can use more context resources.

Advanced Output Formats: You can request raw arrays or cosine similarity matrices directly:

fromllama_cpp.llama_embeddingimportLlamaEmbedding, LLAMA_POOLING_TYPE_NONE# Initialize the model (automatically sets embeddings=True)llm=LlamaEmbedding(model_path="path/to/bge-m3.gguf", n_gpu_layers=-1, pooling_type=LLAMA_POOLING_TYPE_NONE)
# Returns raw List[float] instead of a dictionary wrappervector=llm.create_embedding("Text", output_format="array")
# Returns a similarity matrix (A @ A.T) in the response# Note: Requires numpy installedresponse=llm.create_embedding(
["apple", "fruit", "car"],
output_format="json+"
)
print(response["cosineSimilarity"])

2. Reranking (Cross-Encoder Scoring)

Reranking models (like bge-reranker) take a Query and a list of Documents as input and output a relevance score (scalar) for each document.

Important: You must explicitly set pooling_type to LLAMA_POOLING_TYPE_RANK (4) when initializing the model.

importllama_cppfromllama_cpp.llama_embeddingimportLlamaEmbedding# Initialize a Reranking modelranker=LlamaEmbedding(
model_path="path/to/qwen3-reranker-0.6b-q8_0.gguf",
pooling_type=llama_cpp.LLAMA_POOLING_TYPE_RANK, # Crucial for Rerankers!n_gpu_layers=-1,
n_ctx=0
)
query="What causes Rain?"docs= [
"Clouds are made of water droplets...", # Relevant"To bake a cake you need flour...", # Irrelevant"Rain is liquid water in the form of droplets..."# Highly Relevant
]
# Calculate relevance scores# Logic: Constructs inputs like "[BOS] query [SEP] doc [EOS]" automaticallyscores=ranker.rank(query, docs)
# Result: List of floats (higher means more relevant)print(scores) # e.g., [0.0011407170677557588, 5.614783731289208e-05, 0.7173627614974976] -> The 3rd doc is the best match

3. Normalization

The embed method supports various mathematical normalization strategies via the normalize parameter.

Normalization modes	$Integer$	Description	Formula
NORM_MODE_NONE	$-1$	none
NORM_MODE_MAX_INT16	$0$	max absolute int16	$\Large{{32760 * x_i} \over\max \lvert x_i\rvert}$
NORM_MODE_TAXICAB	$1$	taxicab	$\Large{x_i \over\sum \lvert x_i\rvert}$
NORM_MODE_EUCLIDEAN	$2$	euclidean (default)	$\Large{x_i \over\sqrt{\sum x_i^2}}$
NORM_MODE_PNORM	$>2$	p-norm	$\Large{x_i \over\sqrt[p]{\sum \lvert x_i\rvert^p}}$

This is useful for optimizing storage or preparing vectors for cosine similarity search (which requires L2 normalization).

fromllama_cpp.llama_embeddingimport (
LLAMA_POOLING_TYPE_NONE,
NORM_MODE_MAX_INT16,
NORM_MODE_TAXICAB,
NORM_MODE_EUCLIDEAN
)
# Initialize the model (automatically sets embeddings=True)llm=LlamaEmbedding(model_path="path/to/bge-m3.gguf", n_gpu_layers=-1, pooling_type=LLAMA_POOLING_TYPE_NONE)
# Taxicab (L1)vec_l1=llm.embed("text", normalize=NORM_MODE_TAXICAB)
# Default is Euclidean (L2) - Standard for vector databasesvec_l2=llm.embed("text", normalize=NORM_MODE_EUCLIDEAN)
# Max Absolute Int16 - Useful for quantization/compressionvec_int16=llm.embed("text", normalize=NORM_MODE_MAX_INT16)
# Raw Output (No Normalization) - Get the raw floating point values from the modelembeddings_raw=llm.embed(["search query", "document text"], normalize=NORM_MODE_NONE)

Using the standard `Llama` class

The standard Llama class also supports the maintained streaming embedding implementation. Initialize it with embeddings=True, then call embed() for raw results or create_embedding() for an OpenAI-compatible response. LlamaEmbedding remains a convenient specialized interface because it enables embedding-oriented defaults and provides the rank() helper.

llm=llama_cpp.Llama(
model_path="path/to/model.gguf",
embeddings=True,
n_batch=512,
n_seq_max=8,
kv_unified=True,
)
# OpenAI-compatible response; normalize=True selects L2 normalization.response=llm.create_embedding(["query", "document"], normalize=True)
# Raw vectors. Integer normalization modes are also supported.vectors=llm.embed(["query", "document"], normalize=2)

Speculative Decoding

llama-cpp-python provides a stateful speculative-decoding path aligned with the begin -> process -> draft -> accept lifecycle used by llama.cpp. Configure it with speculative=SpecConfig(...); the older draft_model= API is deprecated and is kept only for compatibility with stateless draft callbacks.

The current implementation is text-only, supports one sequence (seq_id=0), and provides three usable modes:

Mode	`SpeculativeType`	Draft source
MTP	`DRAFT_MTP`	Target model NextN/MTP heads or an external MTP GGUF
N-gram K	`NGRAM_MAP_K`	Previous matching positions in verified token history
N-gram K4V	`NGRAM_MAP_K4V`	Up to four cached continuations per n-gram key, matching `llama.cpp`

Eagle3, DFlash, DSpark, draft-simple, and the other n-gram variants appear in SpeculativeType for llama.cpp API compatibility but do not yet have Python engines.

More Information see wiki: Llama Speculative Decoding

MTP speculative decoding

The built-in and external MTP paths have currently been tested only with the Qwen3.5, Qwen3.6, and Qwen3.8 model families. Other model families may work when their GGUF tensors are compatible, but they have not yet been validated.

Built-in MTP

When the target GGUF contains compatible NextN/MTP tensors, omit draft_model_path. Llama automatically enables loading the target MTP layers.

fromllama_cppimportLlamafromllama_cpp.llama_speculativeimportSpecConfig, SpeculativeTypellm=Llama(
model_path="path/to/model-with-mtp.gguf",
n_batch=512,
n_gpu_layers="all",
speculative=SpecConfig(
spec_type=SpeculativeType.DRAFT_MTP,
draft_n_max=2,
draft_p_min=0.0,
),
)

External MTP model

Set draft_model_path when the MTP tensors are stored in a separate compatible GGUF. Target and draft vocabularies and output embedding dimensions must match.

llm=Llama(
model_path="path/to/target.gguf",
n_batch=512,
n_gpu_layers="all",
speculative=SpecConfig(
spec_type=SpeculativeType.DRAFT_MTP,
draft_model_path="path/to/mtp.gguf",
draft_n_max=2,
draft_n_gpu_layers="all",
draft_backend_sampling=True,
),
)

For Qwen3.8 27B, testing so far suggests draft_n_max=2 as the best starting point. This is not a universal optimum: GPU, backend, quantization, prompt, sampling settings, and whether MTP is built in or external can change the result. Run the included benchmark and choose the fastest stable value for the actual deployment environment.

The verification batch contains [id_last, draft...], so the maximum draft length must not exceed n_batch - 1. Longer drafts only help when their additional acceptance outweighs verification and rollback cost.

N-gram speculative decoding

N-gram decoding is model-free and works best for repeated JSON, tables, code, templates, and boilerplate. It does not require a second GGUF model.

fromllama_cppimportLlamafromllama_cpp.llama_speculativeimportSpecConfig, SpeculativeTypellm=Llama(
model_path="path/to/model.gguf",
n_ctx=4096,
n_batch=512,
n_gpu_layers="all",
speculative=SpecConfig(
spec_type=SpeculativeType.NGRAM_MAP_K,
ngram_size_n=8,
ngram_size_m=16,
ngram_min_hits=1,
),
)
response=llm.create_chat_completion(
messages=[
{
"role": "user",
"content": "Write a Python script using sqlite3 with repeated CRUD classes.",
}
]
)

Use SpeculativeType.NGRAM_MAP_K4V to cache continuations directly:

speculative=SpecConfig(
spec_type=SpeculativeType.NGRAM_MAP_K4V,
ngram_size_n=8,
ngram_size_m=16,
ngram_min_hits=1,
ngram_max_entries_per_key=4,
)

SpecConfig follows the llama.cpp n-gram defaults (N=12, M=48). The best values are workload-dependent. N=8, M=16 is a conservative starting point; longer drafts such as M=32 or M=48 can be faster for highly repetitive output. Always benchmark against ordinary decoding.

For hybrid or recurrent targets, n-gram rejection needs target checkpoints:

llm=Llama(
model_path="path/to/hybrid-model.gguf",
speculative=speculative,
ctx_checkpoints=16,
checkpoint_on_device=True,
)

Runtime statistics

With verbose=True, Llama.generate prints calls, acceptance, phase timings, checkpoint activity, rollbacks, TTFT, and sustained generation speed. The same values are available programmatically after a generation:

stats=llm.last_speculative_statsprint(stats["draft_token_acceptance_rate"])
print(stats["mean_accepted_length"])
print(stats["generation_tokens_per_second"])
print(stats["checkpoint_restore_seconds"])

Use the included examples for repeatable comparisons:

# Ordinary vs built-in/external MTP
python -m examples.high_level_api.high_level_api_mtp_speculative -h
# N-gram N x M scans and cross-method benchmarks
python -m examples.benchmark.benchmark_speculative -h

Notes and limitations

Speculative decoding does not skip target-model verification. Low acceptance can make it slower than ordinary decoding.
The current stateful engines are text-only and single-sequence. Do not enable them for MTMD/multimodal embedding batches or parallel sequence decoding.
A speculative reset clears target and draft state together; public prompt cache restoration does not currently persist the speculative engine state.
draft_model= and LlamaDraftModel are legacy compatibility APIs. New code should use speculative=SpecConfig(...).
Close Llama explicitly in long-running applications to release an external draft model and its context deterministically.

Docker image

See here: https://github.com/JamePeng/llama-cpp-python/tree/main/docker#cuda_simple

OpenAI Compatible Web Server (Deprecated)

llama-cpp-python offers a web server which aims to act as a drop-in replacement for the OpenAI API. This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).

To install the server package and get started:

pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf

Similar to Hardware Acceleration section above, you can also install with GPU (cuBLAS) support like this:

CMAKE_ARGS="-DGGML_CUDA=on" FORCE_CMAKE=1 pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35

Navigate to http://localhost:8000/docs to see the OpenAPI documentation.

To bind to 0.0.0.0 to enable remote connections, use python3 -m llama_cpp.server --host 0.0.0.0. Similarly, to change the port (default is 8000), use --port.

You probably also want to set the prompt format. For chatml, use

python3 -m llama_cpp.server --model models/7B/llama-model.gguf --chat_format chatml

That will format the prompt according to how model expects it. You can find the prompt format in the model card. For possible options, see llama_cpp/llama_chat_format.py and look for lines starting with "@register_chat_format".

If you have huggingface-hub installed, you can also use the --hf_model_repo_id flag to load a model from the Hugging Face Hub.

python3 -m llama_cpp.server --hf_model_repo_id Qwen/Qwen2-0.5B-Instruct-GGUF --model '*q8_0.gguf'

Web Server Features

Low-level API

API Reference

The low-level API is a direct ctypes binding to the C API provided by llama.cpp. The entire low-level API can be found in llama_cpp/llama_cpp.py and directly mirrors the C API in llama.h.

Below is a short example demonstrating how to use the low-level API to tokenize a prompt:

importllama_cppimportctypesllama_cpp.llama_backend_init(False) # Must be called once at the start of each programparams=llama_cpp.llama_context_default_params()
# use bytes for char * paramsmodel=llama_cpp.llama_load_model_from_file(b"./models/7b/llama-model.gguf", params)
ctx=llama_cpp.llama_new_context_with_model(model, params)
max_tokens=params.n_ctx# use ctypes arrays for array paramstokens= (llama_cpp.llama_token*int(max_tokens))()
n_tokens=llama_cpp.llama_tokenize(ctx, b"Q: Name the planets in the solar system? A: ", tokens, max_tokens, llama_cpp.c_bool(True))
llama_cpp.llama_free(ctx)

Check out the examples folder for more examples of using the low-level API.

Documentation

Documentation is available via https://llama-cpp-python.readthedocs.io/. If you find any issues with the documentation, please open an issue or submit a PR.

Development

This package is under active development and I welcome any contributions.

To get started, clone the repository and install the package in editable / development mode:

git clone https://github.com/JamePeng/llama-cpp-python --recursive
cd llama-cpp-python
# Upgrade pip (required for editable mode)
pip install --upgrade pip
# Install with pip
pip install -e .# if you want to use the fastapi / openapi server
pip install -e '.[server]'# to install all optional dependencies
pip install -e '.[all]'# to clear the local build cache
make clean

Now try running the tests

pytest

There's a Makefile available with useful targets. A typical workflow would look like this:

make build
make test

You can also test out specific commits of llama.cpp by checking out the desired commit in the vendor/llama.cpp submodule and then running make clean and pip install -e . again. Any changes in the llama.h API will require changes to the llama_cpp/llama_cpp.py file to match the new API (additional changes may be required elsewhere).

FAQ

Are there pre-built binaries / binary wheels available?

The recommended installation method is to install from source as described above. The reason for this is that llama.cpp is built with compiler optimizations that are specific to your system. Using pre-built binaries would require disabling these optimizations or supporting a large number of pre-built binaries for each platform.

That being said there are some pre-built binaries available through the Releases as well as some community provided wheels.

In the future, I would like to provide pre-built binaries and wheels for common platforms and I'm happy to accept any useful contributions in this area.

How does this compare to other Python bindings of `llama.cpp`?

I originally wrote this package for my own use with two goals in mind:

Provide a simple process to install llama.cpp and access the full C API in llama.hand mtmd.h from Python
Provide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to use llama.cpp
Provide a high-throughput, relatively low-latency Python library by continuously optimizing (reducing unnecessary CPU processing or algorithm tuning) and accepting feedback (issues or pull requests), making loading and running GGUF files via Python simpler and more controllable.
Provides clearer code comments and error code analysis feedback in llama.cpp, based on common usage feedback and code execution flow, to help more users who are learning LLM through this project understand the project's operation and subsequent feedback optimization.

OSError: libcudart.so.XX/cudart64_XX.dll: cannot open shared object file: No such file or directory

This error is primarily caused by the following reasons:

Missing Installation or Configuration: The CUDA Toolkit is either not installed, or the environment variables were not correctly configured after installation, preventing the system from locating the required dynamic link libraries. You can try running nvidia-smi or nvcc in your terminal to check if they output results correctly.
Version Mismatch: The CUDA Toolkit environment is installed and configured, but it does not match the CUDA version of the pre-compiled llama-cpp-python wheel you are using. For example, your local environment might be running CUDA 12.1, but you installed a version compiled for CUDA 12.6.
Recommendation (Build from Source): It is recommended to fully configure your local CUDA Toolkit environment (ensuring the PATH for dynamic libraries is set and the nvcc compiler is recognized). Then, clone the code and compile it locally. Remember to enable the -DGGML_CUDA=on CMake option during compilation. This ensures the installation achieves maximum compatibility with your local system.

FileNotFoundError: Could not find module (like ggml.dll, ggml-cpu.dll, ggml-cuda.dll)

Step 1: Locate the lib folder of the llama-cpp-python library within your current Python runtime environment: Python3XX\Lib\site-packages\llama_cpp\lib\

Step 2: Verify that the missing DLL mentioned in the error is correctly present in this directory. Developers often have multiple Python environments locally, or projects like ComfyUI may use embedded virtual environments. Please ensure that you are installing the library and running the code in the exact same environment.

This error is primarily caused by the following reasons:

Environment Mismatch: The Python environment used for installation is different from the one being used for execution.
Instruction Set Incompatibility: Regarding ggml.dll and ggml-cpu.dll, the instruction sets (such as AVX) supported by the pre-compiled version may be incompatible with your local processor. (This typically manifests as OSError: [WinError -1073741795] Windows Error 0xc000001d after execution).
CUDA Version Mismatch: Regarding ggml-cuda.dll, the CUDA version of the pre-compiled library does not match your local CUDA Toolkit version (e.g., a mismatch between CUDA 12.X and CUDA 13.X). It is recommended to fully configure your local CUDA Toolkit environment (ensuring the PATH for dynamic libraries is set and the nvcc compiler is recognized). Then, clone the code and compile it locally.

Why are libraries compiled by other authors only around 100MB, while your pre-compiled versions are 300MB or larger?

My GitHub Actions workflow is configured to compile against multiple supported CUDA compute architectures for each CUDA version I maintain.

For example:

CUDA 13.1 and CUDA 12.8: currently target architectures from SM75 (Turing) up to SM120a / SM121a (Blackwell generation, depending on CUDA support).
CUDA 12.4 and CUDA 12.6: currently target architectures from SM70 (Volta) up to SM90 (Hopper).

Libraries from other authors are often smaller because they may only compile for a single architecture, such as RTX 30 series (SM86) or RTX 40 series (SM89). To maximize compatibility, these wheels include CUDA kernels for a wider range of GPUs. You only need to choose the wheel that matches your installed CUDA version.

Updated 2026-05-16 / 2026-05-17: Starting with 0.3.39-preview, Windows wheels support the GGML_BACKEND_DL + GGML_CPU_ALL_VARIANTS runtime layout. CPU backend libraries such as ggml-cpu-*.dll are packaged under site-packages/llama_cpp/lib and loaded dynamically at runtime. This allows GGML to select a compatible CPU backend automatically, reducing the need for separate Basic / AVX2 wheel variants.
Note: for full x64 CPU variant coverage on Windows, LLVM/Clang builds are preferred. MSVC may skip some variants such as zen4, cooperlake, or sapphirerapids due to compiler intrinsic support limitations.

Quick tips for develop/user (continuously updated):

1. I've determined that llama_cpp.server is currently in a semi-deprecated state (meaning it won't be maintained unless absolutely necessary, and I might even consider deleting or separating it to reduce the library size). I highly recommend using the llama-server program maintained by the upstream llama.cpp project, which offers a lower-level implementation, more frequent maintenance and optimization, and more reliable API calls.
1. Regarding AMD and Intel graphics cards, AMD can use ROCm as the primary backend, while Intel's Sycl will encounter some compilation difficulties. I consistently recommend using the Vulkan backend for these two types of graphics cards for greater efficiency and stability, because the upstream llama.cpp Vulkan backend is actively maintained by many developers, generally allowing you to enjoy new feature optimizations and bug fixes earlier and faster.
1. If you are using hybrid multimodal model for building ComfyUI nodes or running single-turn API wrappers where you do not need multi-turn state rollbacks, simply initialize your Llama instance with ctx_checkpoints=0:
```
llm=Llama(
model_path="./Qwen3.5-VL-9B.gguf",
chat_handler=MTMDChatHandler(clip_model_path="./mmproj.gguf"),
n_ctx=4096,
ctx_checkpoints=0# <-- SET THIS TO 0 TO ENABLE ZERO-LATENCY FAST PATH
)
```

Any suggestions, contributions, and modifications to this package will be directed toward building a user-friendly, efficient, and secure Python library.

License

This project is licensed under the terms of the MIT license.

Name		Name	Last commit message	Last commit date
.github		.github
docker		docker
docs		docs
examples		examples
llama_cpp		llama_cpp
scripts		scripts
tests		tests
tools/abi		tools/abi
vendor		vendor
.dockerignore		.dockerignore
.gitignore		.gitignore
.gitmodules		.gitmodules
CHANGELOG.md		CHANGELOG.md
CMakeLists.txt		CMakeLists.txt
LICENSE.md		LICENSE.md
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml

Latest commit

History

Folders and files

Repository files navigation

Efficient Python Bindings for llama.cpp library

Discussions

Installation

Installation Configuration

Supported Backends

Install OpenVINO Runtime

Supported Devices

Supported OS

Intel GPU

Install Notes

Windows notes

Wheel packaging checklist

Upgrading and Reinstalling

High-level API

Pulling models from Hugging Face Hub

Chat Completion

JSON and JSON Schema Mode

JSON Mode

JSON Schema Mode

Function Calling

Continuing Assistant Responses (Prefill)

Dynamic LoRA Routing & Control Vectors (Multi-Tenant Serving)

Dynamic LoRA Example

Control Vector Injection (Representation Engineering)

Sampling Configuration & Usage (LlamaSamplingParams)

Core Sampling Parameters

Advanced & Experimental Samplers

Target Entropy (Mirostat)

Repetition Penalties

Constraints & Callbacks

Reasoning Budget (First Reasoning Block)

🛠️ Usage Example

Multi-modal Models

Generic MTMD Chat Handler

Chat Template Resolution Order

Passing Extra Template Arguments

Loading a Local Image With Qwen3VL(Thinking/Instruct)

Speech Recognition With Qwen3-ASR (Speech-to-Text)

How it works:

Comprehensive Omni MultiModal Example: Gemma-4 (Vision + Audio + Text)

Embeddings & Reranking (GGUF)

Key Features:

Support Embeddings & Rerank Model:

TODO(JamePeng): Needs more extensive testing with various embedding and rerank models. :)

1. Text Embeddings (Vector Search)

2. Reranking (Cross-Encoder Scoring)

3. Normalization

Using the standard Llama class

Speculative Decoding

MTP speculative decoding

Built-in MTP

External MTP model

N-gram speculative decoding

Runtime statistics

Notes and limitations

Docker image

OpenAI Compatible Web Server (Deprecated)

Web Server Features

Low-level API

Documentation

Development

FAQ

Are there pre-built binaries / binary wheels available?

How does this compare to other Python bindings of llama.cpp?

OSError: libcudart.so.XX/cudart64_XX.dll: cannot open shared object file: No such file or directory

FileNotFoundError: Could not find module (like ggml.dll, ggml-cpu.dll, ggml-cuda.dll)

Why are libraries compiled by other authors only around 100MB, while your pre-compiled versions are 300MB or larger?

Quick tips for develop/user (continuously updated):

Any suggestions, contributions, and modifications to this package will be directed toward building a user-friendly, efficient, and secure Python library.

License

About

Resources

Stars

Watchers

Forks

Releases

Efficient Python Bindings for `llama.cpp` library

Using the standard `Llama` class

How does this compare to other Python bindings of `llama.cpp`?