llama.cpp

Inference of Facebook's LLaMA model in pure C/C++

TEMPORARY NOTICE: Currently the quantized models run only on Apple Silicon. On other architectures, you can use the F16 models, but they will be much slower. Support will be added later

Description

The main goal is to run the model using 4-bit quantization on a MacBook.

• Plain C/C++ implementation without dependencies
• Apple silicon first-class citizen - optimized via Arm Neon and Accelerate framework
• Mixed F16 / F32 precision
• 4-bit quantization support
• Runs on the CPU

This was hacked in an evening - I have no idea if it works correctly.

So far, I've tested just the 7B model. Here is a typical run:

make -j && ./main -m ../LLaMA-4bit/7B/ggml-model-q4_0.bin -p"Building a website can be done in 10 simple steps:" -t8 -n512Illama.cppbuildinfo: IUNAME_S: DarwinIUNAME_P: armIUNAME_M: arm64ICFLAGS: -I. -O3 -DNDEBUG -std=c11 -fPIC -pthread -DGGML_USE_ACCELERATEICXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthreadILDFLAGS: -frameworkAccelerateICC: Appleclangversion14.0.0 (clang-1400.0.29.202)
ICXX: Appleclangversion14.0.0 (clang-1400.0.29.202)
make: Nothingtobedonefor `default'.
main: seed = 1678486056llama_model_load: loadingmodelfrom'../LLaMA-4bit/7B/ggml-model-q4_0.bin' - pleasewait ...
llama_model_load: n_vocab = 32000llama_model_load: n_ctx = 512llama_model_load: n_embd = 4096llama_model_load: n_mult = 256llama_model_load: n_head = 32llama_model_load: n_layer = 32llama_model_load: n_rot = 128llama_model_load: f16 = 2llama_model_load: n_ff = 11008llama_model_load: ggmlctxsize = 4529.34MBllama_model_load: memory_size = 512.00MB, n_mem = 16384llama_model_load: .................................... donellama_model_load: modelsize = 4017.27MB / numtensors = 291main: prompt: 'Building a website can be done in 10 simple steps:'main: numberoftokensinprompt = 151 -> ''
8893 -> 'Build'292 -> 'ing'263 -> ' a'4700 -> ' website'508 -> ' can'367 -> ' be'2309 -> ' done'297 -> ' in'29871 -> ' '29896 -> '1'29900 -> '0'2560 -> ' simple'6576 -> ' steps'29901 -> ':'samplingparameters: temp = 0.800000, top_k = 40, top_p = 0.950000Buildingawebsitecanbedonein10simplesteps:
1) Selectadomainnameandwebhostingplan2) Completeasitemap3) Listyourproducts4) Writeproductdescriptions5) Createauseraccount6) Buildthetemplate7) Startbuildingthewebsite8) Advertisethewebsite9) Provideemailsupport10) SubmitthewebsitetosearchenginesAwebsiteisacollectionofwebpagesthatareformattedwithHTML. HTMListhecodethatdefineswhatthewebsitelookslikeandhowitbehaves.
TheHTMLcodeisformattedintoatemplateoraformat. Oncethisisdone, itisdisplayedontheuser's browser.
Thewebpagesarestoredinawebserver. Thewebserverisalsocalledahost. Whenthewebsiteisaccessed, itisretrievedfromtheserveranddisplayedontheuser's computer.
Awebsiteisknownasawebsite when itishosted. Thismeansthatitisdisplayedonahost. Thehostisusuallyawebserver.
Awebsitecanbedisplayedondifferentbrowsers. Thebrowsersarebasicallythesoftwarethatrendersthewebsiteontheuser's screen.
Awebsitecanalsobeviewedondifferentdevicessuchasdesktops, tabletsandsmartphones.
Hence, tohaveawebsitedisplayedonabrowser, thewebsitemustbehosted.
Adomainnameisanaddressofawebsite. Itisthenameofthewebsite.
Thewebsiteisknownasawebsite when itishosted. Thismeansthatitisdisplayedonahost. Thehostisusuallyawebserver.
Awebsitecanbedisplayedondifferentbrowsers. Thebrowsersarebasicallythesoftwarethatrendersthewebsiteontheuser’sscreen.
Awebsitecanalsobeviewedondifferentdevicessuchasdesktops, tabletsandsmartphones. Hence, tohaveawebsitedisplayedonabrowser, thewebsitemustbehosted.
Adomainnameisanaddressofawebsite. Itisthenameofthewebsite.
Awebsiteisanaddressofawebsite. ItisacollectionofwebpagesthatareformattedwithHTML. HTMListhecodethatdefineswhatthewebsitelookslikeandhowitbehaves.
TheHTMLcodeisformattedintoatemplateoraformat. Oncethisisdone, itisdisplayedontheuser’sbrowser.
Awebsiteisknownasawebsite when itishostedmain: mempertoken = 14434244bytesmain: loadtime = 1332.48msmain: sampletime = 1081.40msmain: predicttime = 31378.77ms / 61.41mspertokenmain: totaltime = 34036.74ms

And here is another demo of running both LLaMA-7B and whisper.cpp on a single M1 Pro MacBook:

whisper-llama-lq.mp4

Usage

# build this repo
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
# obtain the original LLaMA model weights and place them in ./models
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
# convert the 7B model to ggml FP16 format
python3 convert-pth-to-ggml.py models/7B/ 1
# quantize the model to 4-bits
./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2
# run the inference
./main -m ./models/7B/ggml-model-q4_0.bin -t 8 -n 128

Limitations

• Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook
• Not sure if my tokenizer is correct. There are a few places where we might have a mistake:
  • https://github.com/ggerganov/llama.cpp/blob/26c084662903ddaca19bef982831bfb0856e8257/convert-pth-to-ggml.py#L79-L87
  • https://github.com/ggerganov/llama.cpp/blob/26c084662903ddaca19bef982831bfb0856e8257/utils.h#L65-L69 In general, it seems to work, but I think it fails for unicode character support. Hopefully, someone can help with that
• I don't know yet how much the quantization affects the quality of the generated text
• Probably the token sampling can be improved
• x86 quantization support not yet ready. Basically, you want to run this on Apple Silicon. For now, on Linux and Windows you can use the F16 ggml-model-f16.bin model, but it will be much slower.