vit.cpp

Inference Vision Transformer (ViT) in plain C/C++ using ggml without any extra dependencies

Description

This project presents a standalone implementation of the well known Vision Transformer (ViT) model family, used in a broad spectrum of applications and SOTA models like Large Multimodal Models(LMM). The primary goal is to develop a C/C++ inference engine tailored for ViT models, utilizing ggml to enhance performance, particularly on edge devices. Designed to be both lightweight and self-contained, this implementation can be run across diverse platforms.

Table of Contents

1. Description
2. Features
3. Vision Transformer Architecture
4. Quick Example
5. Convert PyTorch to GGUF
6. Build
   • Simple Build
   • Per Device Optimizations
   • OpenMP
7. Run
8. Benchmark against PyTorch
   • ViT Inference
   • Benchmark on Your Machine
9. Quantization
10. To-Do List

Features

• Dependency-free and lightweight inference thanks to ggml.
• 4-bit, 5-bit and 8-bit quantization support.
• Support for timm ViTs with different variants out of the box.

An important aspect of using vit.cpp is that it has short startup times compared to common DL frameworks, which makes it suitable for serverless deployments where the cold start is an issue.

Vision Transformer architecture

The implemented architecture is based on the original Vision Transformer from:

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

ViT architecture. Taken from the original paper.

Quick example

Details

See output

 $ ./bin/vit -t 4 -m ../ggml-model-f16.gguf -i ../assets/magpie.jpeg -k 5
main: seed = 1701176263
main: n_threads = 4 / 8
vit_model_load: loading model from '../ggml-model-f16.gguf' - please wait
vit_model_load: hidden_size = 192
vit_model_load: num_hidden_layers = 12
vit_model_load: num_attention_heads = 3
vit_model_load: patch_size = 16
vit_model_load: img_size = 224
vit_model_load: num_classes = 1000
vit_model_load: ftype = 1
vit_model_load: qntvr = 0
operator(): ggml ctx size = 11.13 MB
vit_model_load: ................... done
vit_model_load: model size = 11.04 MB / num tensors = 152
main: loaded image '../assets/magpie.jpeg' (500 x 470)
vit_image_preprocess: scale = 2.232143
processed, out dims : (224 x 224)
> magpie : 0.87
> goose : 0.02
> toucan : 0.01
> drake : 0.01
> king penguin, Aptenodytes patagonica : 0.01
main: model load time = 17.92 ms
main: processing time = 146.96 ms
main: total time = 164.88 ms

Convert PyTorch to GGUF

# clone the repo recursively
git clone --recurse-submodules https://github.com/staghado/vit.cpp.git
cd vit.cpp
# install torch and timm
pip install torch timm
# list available models if needed; note that not all models are supported
python convert-pth-to-ggml.py --list
# convert the weights to gguf : vit tiny with patch size of 16 and an image # size of 384 pre-trained on ImageNet21k and fine-tuned on ImageNet1k
python convert-pth-to-ggml.py --model_name vit_tiny_patch16_384.augreg_in21k_ft_in1k --ftype 1

Note: You can also download the converted weights from Hugging Face directly.

wget https://huggingface.co/staghado/vit.cpp/blob/main/tiny-ggml-model-f16.gguf

Build

Simple build

# build ggml and vit mkdir build && cd build
cmake .. && make -j4
# run inference
./bin/vit -t 4 -m ../ggml-model-f16.gguf -i ../assets/tench.jpg

The optimal number of threads to use depends on many factors and more is not always better. Usually using a number of threads equal to the number of available physical cores gives the best performance in terms of speed.

Per device optimizations

Generate per-device instructions that work best for the given machine rather than using general CPU instructions. This can be done by specifying -march=native in the compiler flags.

• Multi-threading and vectorization
• Loop transformations(unrolling)

For AMD host processors

You can use a specialized compiler released by AMD to make full use of your specific processor's architecture. Read more here : AMD Optimizing C/C++ and Fortran Compilers (AOCC)

You can follow the given instructions to install the AOCC compiler.

Note : For my AMD Ryzen 7 3700U, the improvements were not very significant but for more recent processors there could be a gain in using a specialized compiler.

Using OpenMP

Additionally compile with OpenMP by specifying the -fopenmp flag to the compiler in the CMakeLists file, allowing multithreaded runs. Make sure to also enable multiple threads when running, e.g.:

OMP_NUM_THREADS=4 ./bin/vit -t 4 -m ../ggml-model-f16.bin -i ../assets/tench.jpg

Run

usage: ./bin/vit [options]
options:
-h, --help show this help message and exit
-s SEED, --seed SEED RNG seed (default: -1)
-t N, --threads N number of threads to use during computation (default: 4)
-m FNAME, --model FNAME model path (default: ../ggml-model-f16.bin)
-i FNAME, --inp FNAME input file (default: ../assets/tench.jpg)
-k N, --topk N top k classes to print (default: 5)
-e FLOAT, --epsilon epsilon (default: 0.000001)

Benchmark against PyTorch

First experiments on Apple M1 show inference speedups(up to 6x faster for base model) compared to native PyTorch inference.

ViT inference

You can efficiently run ViT inference on the CPU. Memory requirements and inference speed on AMD Ryzen 7 3700U(4 cores, 8 threads) for both native PyTorch and vit.cpp. Using 4 threads gives better results for my machine. The reported results of inference speed correspond to 10 runs averages for both PyTorch and vit.cpp.

Model	Max Mem(PyTorch)	Max Mem	Speed(PyTorch)	Speed
tiny	~780 MB	~20 MB	431 ms	120 ms
small	~965 MB	~52 MB	780 ms	463 ms
base	~1.61 GB	~179 MB	2393 ms	1441 ms
large	~3.86 GB	~597 MB	8151 ms	4892 ms

Note: The models used are of the form vit_{size}_patch16_224.augreg_in21k_ft_in1k.

Benchmark on your machine

In order to test the inference speed on your machine, you can run the following scripts:

chmod +x scripts/benchmark.*
# install memory_profiler & threadpoolctl
pip install memory_profiler threadpoolctl
# run the benchmark of PyTorch
python scripts/benchmark.py
# run the benchmark of vit.cpp for non-qunatized model
./scripts/benchmark.sh
# to run the benchamrk for qunatized models; 4 threads and quantize flag
./scripts/benchmark.sh 4 1

Both scripts use 4 threads by default. In Python, the threadpoolctl library is used to limit the number of threads used by PyTorch.

Quantization

vit.cpp supports many quantization strategies from ggml such as q4_0, q4_1, q5_0, q5_1 and q8_0 types. You can quantize a model in F32 (the patch embedding is in F16) to one of these types by using the ./bin/quantize binary.

usage: ./bin/quantize /path/to/ggml-model-f32.gguf /path/to/ggml-model-quantized.gguf type type = 2 - q4_0 type = 3 - q4_1 type = 6 - q5_0 type = 7 - q5_1 type = 8 - q8_0 

For example, you can run the following to convert the model to q5_1:

./bin/quantize ../tiny-ggml-model-f16.gguf ../tiny-ggml-model-f16-quant.gguf 7

Then you can use tiny-ggml-model-f16-quant.gguf just like the model in F16.

Results

Here are the benchmarks for the different models and quantizations on my machine: For accurate estimation of run times, these benchmarks were run 100 times each.

Model	Quantization	Speed (ms)	Mem (MB)
tiny	q4_0	105 ms	12 MB
tiny	q4_1	97 ms	12 MB
tiny	q5_0	116 ms	13 MB
tiny	q5_1	112 ms	13 MB
tiny	q8_0	90 ms	15 MB
small	q4_0	240 ms	23 MB
small	q4_1	224 ms	24 MB
small	q5_0	288 ms	25 MB
small	q5_1	277 ms	27 MB
small	q8_0	228 ms	33 MB
base	q4_0	704 ms	61 MB
base	q4_1	626 ms	66 MB
base	q5_0	851 ms	71 MB
base	q5_1	806 ms	76 MB
base	q8_0	659 ms	102 MB
large	q4_0	2189 ms	181 MB
large	q4_1	1919 ms	199 MB
large	q5_0	2676 ms	217 MB
large	q5_1	2547 ms	235 MB
large	q8_0	1994 ms	325 MB

To-Do List

• Evaluate performance on ImageNet1k:

  Run evaluation on ImageNet1k test set and analyze the performance of different quantization schemes.

This project was highly inspired by the following projects:

• whisper.cpp
• llama.cpp

Star History