Speculative sampling

[andriyanthon]

llama.cpp added a feature for speculative inference:
ggml-org/llama.cpp#2926
but when running llama_cpp.server, it says it does not recognize the new parameters.

There are two new parameters:

1. -md (model_draft) - the path to the draft model.
2. -draft (n_draft) - how many tokens to draft each time

Can this new feature please be supported?

[abetlen]

@andriyanthon good idea, I'll take a look into this. I think a similar API to hugginface's Assisted Generation would work well.

[Chainfire]

+1. Would probably double performance in my setup.

[galatolofederico]

+1. It would be very useful

[gssci]

Any updates on this?

[LynxPDA]

+1

Also added the -ngld parameter which tells how many layers to unload in VRAM for Draft model.

On hardware:

• Ryzen 7950X
• DDR5 5800
• RTX 3060 12Gb

For me, the acceleration amounted to phind-codellama-34b-v2.Q4_K_M.gguf

• Only CPU - 1.88x (3.3-->6.2)
• GPU+CPU - 1.78x (5.4-->9.6)

An example of the full CLI in llama.cpp and the results for me are below:

./speculative -m ./models/phind-codellama-34b-v2.Q4_K_M.gguf -md ./models/codellama-7b-instruct.Q3_K_M.gguf -p "// Quick-sort implementation in C (4 spaces indentation + detailed comments) and sample usage:\n\n#include" -e -t 16 -n 256 -c 2048 -s 8 --draft 15 -b 512 **-ngld 35 -ngl 15**

And the results:

encoded 25 tokens in 1.490 seconds, speed: 16.780 t/s
decoded 280 tokens in 29.131 seconds, speed: 9.612 t/s

n_draft = 27
n_predict = 280
n_drafted = 334
n_accept = 245
accept = 73.353%

draft:

llama_print_timings: load time = 538.74 ms
llama_print_timings: sample time = 522.77 ms / 1 runs ( 522.77 ms per token, 1.91 tokens per second)
llama_print_timings: prompt eval time = 171.93 ms / 25 tokens ( 6.88 ms per token, 145.41 tokens per second)
llama_print_timings: eval time = 7165.94 ms / 360 runs ( 19.91 ms per token, 50.24 tokens per second)
llama_print_timings: total time = 30621.07 ms

target:

llama_print_timings: load time = 1015.12 ms
llama_print_timings: sample time = 91.54 ms / 280 runs ( 0.33 ms per token 3058.81 tokens per second)
llama_print_timings: prompt eval time = 20972.04 ms / 386 tokens ( 54.33 ms per token, 18.41 tokens per second)
llama_print_timings: eval time = 1603.66 ms / 7 runs ( 229.09 ms per token 4.37 tokens per second)
llama_print_timings: total time = 31163.91 ms

[oobabooga]

I have made some speculative decoding tests with the following models on my RTX 3090:

• Target: wizardlm-70b-v1.0.Q4_K_S.gguf
• Draft: tinyllama-1.1b-chat-v0.3.Q4_K_M.gguf

With speculative, I get 3.41 tokens/second, while without it I get 2.08 tokens/second. That's a +64% increase.

This is the command that I used:

./speculative \
 -m ../models/wizardlm-70b-v1.0.Q4_K_S.gguf \
 -md ../models/tinyllama-1.1b-chat-v0.3.Q4_K_M.gguf \
 -e \
 -t 6 \
 -tb 12 \
 -n 256 \
 -c 4096 \
 --draft 15 \
 -ngld 128 \
 -ngl 42 \
 -p "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n\nUSER: Give me an example of Python script.\nASSISTANT:"

Having this feature available in llama-cpp-python would be amazing.

[rangehow]

This features looks so cool : ) Looking forward to this！

[abetlen]

#1120 is almost ready, need to do some more testing and perf benchmarks but it works now with prompt lookup decoding.