llama.cpp

Inference of Facebook's LLaMA model in pure C/C++

Hot topics

• Running on Windows: ggml-org#22

Description

The main goal is to run the model using 4-bit quantization on a MacBook.

• Plain C/C++ implementation without dependencies
• Apple silicon first-class citizen - optimized via Arm Neon and Accelerate framework
• AVX2 support for x86 architectures
• Mixed F16 / F32 precision
• 4-bit quantization support
• Runs on the CPU

This was hacked in an evening - I have no idea if it works correctly. Please do not make conclusions about the models based on the results from this implementation. For all I know, it can be completely wrong. This project is for educational purposes and is not going to be maintained properly. New features will probably be added mostly through community contributions, if any.

---

Here is a typical run using LLaMA-7B:

make -j && ./main -m ./models/7B/ggml-model-q4_0.bin -p"Building a website can be done in 10 simple steps:" -t8 -n512Illama.cppbuildinfo:
IUNAME_S: DarwinIUNAME_P: armIUNAME_M: arm64ICFLAGS: -I. -O3 -DNDEBUG -std=c11 -fPIC -pthread -DGGML_USE_ACCELERATEICXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthreadILDFLAGS: -frameworkAccelerateICC: Appleclangversion14.0.0 (clang-1400.0.29.202)
ICXX: Appleclangversion14.0.0 (clang-1400.0.29.202)
make: Nothingtobedonefor `default'.
main: seed = 1678486056llama_model_load: loadingmodelfrom'./models/7B/ggml-model-q4_0.bin' - pleasewait ...
llama_model_load: n_vocab = 32000llama_model_load: n_ctx = 512llama_model_load: n_embd = 4096llama_model_load: n_mult = 256llama_model_load: n_head = 32llama_model_load: n_layer = 32llama_model_load: n_rot = 128llama_model_load: f16 = 2llama_model_load: n_ff = 11008llama_model_load: ggmlctxsize = 4529.34MBllama_model_load: memory_size = 512.00MB, n_mem = 16384llama_model_load: .................................... donellama_model_load: modelsize = 4017.27MB / numtensors = 291main: prompt: 'Building a website can be done in 10 simple steps:'main: numberoftokensinprompt = 151 -> ''
8893 -> 'Build'292 -> 'ing'263 -> ' a'4700 -> ' website'508 -> ' can'367 -> ' be'2309 -> ' done'297 -> ' in'29871 -> ' '29896 -> '1'29900 -> '0'2560 -> ' simple'6576 -> ' steps'29901 -> ':'samplingparameters: temp = 0.800000, top_k = 40, top_p = 0.950000Buildingawebsitecanbedonein10simplesteps:
1) Selectadomainnameandwebhostingplan2) Completeasitemap3) Listyourproducts4) Writeproductdescriptions5) Createauseraccount6) Buildthetemplate7) Startbuildingthewebsite8) Advertisethewebsite9) Provideemailsupport10) SubmitthewebsitetosearchenginesAwebsiteisacollectionofwebpagesthatareformattedwithHTML. HTMListhecodethatdefineswhatthewebsitelookslikeandhowitbehaves.
TheHTMLcodeisformattedintoatemplateoraformat. Oncethisisdone, itisdisplayedontheuser's browser.
Thewebpagesarestoredinawebserver. Thewebserverisalsocalledahost. Whenthewebsiteisaccessed, itisretrievedfromtheserveranddisplayedontheuser's computer.
Awebsiteisknownasawebsite when itishosted. Thismeansthatitisdisplayedonahost. Thehostisusuallyawebserver.
Awebsitecanbedisplayedondifferentbrowsers. Thebrowsersarebasicallythesoftwarethatrendersthewebsiteontheuser's screen.
Awebsitecanalsobeviewedondifferentdevicessuchasdesktops, tabletsandsmartphones.
Hence, tohaveawebsitedisplayedonabrowser, thewebsitemustbehosted.
Adomainnameisanaddressofawebsite. Itisthenameofthewebsite.
Thewebsiteisknownasawebsite when itishosted. Thismeansthatitisdisplayedonahost. Thehostisusuallyawebserver.
Awebsitecanbedisplayedondifferentbrowsers. Thebrowsersarebasicallythesoftwarethatrendersthewebsiteontheuser’sscreen.
Awebsitecanalsobeviewedondifferentdevicessuchasdesktops, tabletsandsmartphones. Hence, tohaveawebsitedisplayedonabrowser, thewebsitemustbehosted.
Adomainnameisanaddressofawebsite. Itisthenameofthewebsite.
Awebsiteisanaddressofawebsite. ItisacollectionofwebpagesthatareformattedwithHTML. HTMListhecodethatdefineswhatthewebsitelookslikeandhowitbehaves.
TheHTMLcodeisformattedintoatemplateoraformat. Oncethisisdone, itisdisplayedontheuser’sbrowser.
Awebsiteisknownasawebsite when itishostedmain: mempertoken = 14434244bytesmain: loadtime = 1332.48msmain: sampletime = 1081.40msmain: predicttime = 31378.77ms / 61.41mspertokenmain: totaltime = 34036.74ms

And here is another demo of running both LLaMA-7B and whisper.cpp on a single M1 Pro MacBook:

whisper-llama-lq.mp4

Usage

Here are the step for the LLaMA-7B model:

# build this repo
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
# obtain the original LLaMA model weights and place them in ./models
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
# install Python dependencies
python3 -m pip install torch numpy sentencepiece
# convert the 7B model to ggml FP16 format
python3 convert-pth-to-ggml.py models/7B/ 1
# quantize the model to 4-bits
./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2
# run the inference
./main -m ./models/7B/ggml-model-q4_0.bin -t 8 -n 128

For the bigger models, there are a few extra quantization steps. For example, for LLaMA-13B, converting to FP16 format will create 2 ggml files, instead of one:

ggml-model-f16.bin
ggml-model-f16.bin.1

You need to quantize each of them separately like this:

./quantize ./models/13B/ggml-model-f16.bin ./models/13B/ggml-model-q4_0.bin 2
./quantize ./models/13B/ggml-model-f16.bin.1 ./models/13B/ggml-model-q4_0.bin.1 2

Everything else is the same. Simply run:

./main -m ./models/13B/ggml-model-q4_0.bin -t 8 -n 128

The number of files generated for each model is as follows:

7B -> 1 file
13B -> 2 files
30B -> 4 files
65B -> 8 files

When running the larger models, make sure you have enough disk space to store all the intermediate files.

Limitations

• Not sure if my tokenizer is correct. There are a few places where we might have a mistake:
  • https://github.com/ggerganov/llama.cpp/blob/26c084662903ddaca19bef982831bfb0856e8257/convert-pth-to-ggml.py#L79-L87
  • https://github.com/ggerganov/llama.cpp/blob/26c084662903ddaca19bef982831bfb0856e8257/utils.h#L65-L69 In general, it seems to work, but I think it fails for unicode character support. Hopefully, someone can help with that
• I don't know yet how much the quantization affects the quality of the generated text
• Probably the token sampling can be improved
• The Accelerate framework is actually currently unused since I found that for tensor shapes typical for the Decoder, there is no benefit compared to the ARM_NEON intrinsics implementation. Of course, it's possible that I simlpy don't know how to utilize it properly. But in any case, you can even disable it with LLAMA_NO_ACCELERATE=1 make and the performance will be the same, since no BLAS calls are invoked by the current implementation