llm-api

Web API and websocket for Large Language Models in C++

Buıld

1. clone the repo and cd into it:

git clone https://github.com/monatis/llm-api.git &&cd llm-api

2. Install asio for the web API.

apt install libasio-dev

Note: You can also run scripts/install-dev.sh to install asio (and websocat additionally, in order to test websocket on the terminal).

3. Build with cmake and make:

mkdir build &&cd build
cmake -DLLM_NATIVE=ON ..
make -j4

Find the executible in ./bin/llm-api.

Run

1. Download gpt4all-j model if you haven't already:

wget https://gpt4all.io/models/ggml-gpt4all-j.bin -O ./bin/ggml-gpt4all-j.bin

2. Run the executible:

./bin/llm-api

Note: You can pass the model path with -m argument if it's located elsewhere. See below for more options.

Options

./bin/llm-api -h
usage: ./bin/llm-api [options] options:
-h, --help show this help message and exit
-v, --verbose log generation in stdout (default: disabled)
-s SEED, --seed SEED RNG seed (default: -1)
-t N, --threads N number of threads to use during computation (default: 4)
--port PORT port to listen on (default: 8080)
-p PROMPT, --prompt PROMPT prompt to start generation with (default: random)
-n N, --n_predict N number of tokens to predict (default: 200)
--top_k N top-k sampling (default: 40)
--top_p N top-p sampling (default: 0.9)
--temp N temperature (default: 0.9)
-b N, --batch_size N batch size for prompt processing (default: 8)
-m FNAME, --model FNAME model path (default: ggml-gpt4all-j.bin)

Roadmap

• Improve multi-user experience
• Integrate StableLM model.
• Add embedding endpoint.
• Provide a chain mechanism.
• Integrate a chat UI.
• Add Docker support.
• Extend readme and docs

Credits

• ggerganov for his awesome work in ggml.
• Nomic AI for their continuus contribution in truely open-source LLMs.