feat: add support for cublas/openblas in the llama.cpp backend

[mudler]

Depends on: go-skynet/go-llama.cpp#51

See upstream PR: ggml-org/llama.cpp#1412

Allows to build LocalAI with the llama.cpp backend with cublas/openblas:

Cublas

To build, run:

make BUILD_TYPE=cublas CUDA_LIBPATH=.... build

OpenBLAS

make BUILD_TYPE=openblas build

To set the number of GPU layers, in the config file:

gpu_layers: 4

This also drops the "generic" build type, as I'm sunsetting it in favor of specific cmake parameters

Related to: #69

[mudler]

Let's merge this to master as it add-only and doesn't hurt as a starting point. I successfully built it on colab, but no way to test this locally. I'll update the docs and let see out of bug reports.

[bubthegreat]

Might be worth dropping this command in a readme that should allow folks to test that they have a valid detectable GPU:

docker run --gpus all --rm nvidia/cuda:10.2-base nvidia-smi

Example output showing a valid GPU:

PS C:\Users\bubth\Development\LocalAI\nvidia> docker run --gpus all --rm nvidia/cuda:10.2-base nvidia-smi
Unable to find image 'nvidia/cuda:10.2-base' locally
10.2-base: Pulling from nvidia/cuda
25fa05cd42bd: Already exists
24a22c1b7260: Already exists
8dea37be3176: Already exists
b4dc78aeafca: Already exists
a57130ec8de1: Already exists
Digest: sha256:86aba51da8781cc370350a6e30166ab2714229d505fd87f8d28ff6d3677a0ba4
Status: Downloaded newer image for nvidia/cuda:10.2-base
Tue May 16 18:56:46 2023 +---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.50 Driver Version: 531.79 CUDA Version: 12.1 ||-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC || Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |||| MIG M. ||=========================================+======================+======================|| 0 NVIDIA GeForce RTX 3080 Ti On | 00000000:01:00.0 On | N/A || 35% 46C P8 36W / 350W| 6131MiB / 12288MiB | 6% Default |||| N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: || GPU GI CI PID Type Process name GPU Memory || ID ID Usage ||=======================================================================================|| No running processes found |
+---------------------------------------------------------------------------------------+
PS C:\Users\bubth\Development\LocalAI\nvidia>

[Thireus]

Good stuff!

Although it seems that BUILD_TYPE=cublas doesn't automatically pass LLAMA_CUBLAS=1 to llama.cpp make.

The following solves the issue:

make BUILD_TYPE=cublas LLAMA_CUBLAS=1 build

Which is necessary, otherwise llama.cpp compiles without -DGGML_USE_CUBLAS as seen below.

make -C llama.cpp llama.o
make[2]: Entering directory '/home/thireus/LocalAI/go-llama/llama.cpp'
I llama.cpp build info: I UNAME_S: Linux
I UNAME_P: x86_64
I UNAME_M: x86_64
I CFLAGS: -I. -O3 -std=c11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native
I LDFLAGS: I CC: cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
I CXX: g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -c llama.cpp -o llama.o

With LLAMA_CUBLAS=1:

make[1]: Entering directory '/home/thireus/LocalAI/go-llama/llama.cpp'
I llama.cpp build info: I UNAME_S: Linux
I UNAME_P: x86_64
I UNAME_M: x86_64
I CFLAGS: -I. -O3 -std=c11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include
I LDFLAGS: -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64
I CC: cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
I CXX: g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -c llama.cpp -o llama.o

[mudler]

Good stuff!

Although it seems that BUILD_TYPE=cublas doesn't automatically pass LLAMA_CUBLAS=1 to llama.cpp make.

The following solves the issue:

make BUILD_TYPE=cublas LLAMA_CUBLAS=1 build

Which is necessary, otherwise llama.cpp compiles without -DGGML_USE_CUBLAS as seen below.

make -C llama.cpp llama.o
make[2]: Entering directory '/home/thireus/LocalAI/go-llama/llama.cpp'
I llama.cpp build info: I UNAME_S: Linux
I UNAME_P: x86_64
I UNAME_M: x86_64
I CFLAGS: -I. -O3 -std=c11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native
I LDFLAGS: I CC: cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
I CXX: g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -c llama.cpp -o llama.o

With LLAMA_CUBLAS=1:

make[1]: Entering directory '/home/thireus/LocalAI/go-llama/llama.cpp'
I llama.cpp build info: I UNAME_S: Linux
I UNAME_P: x86_64
I UNAME_M: x86_64
I CFLAGS: -I. -O3 -std=c11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include
I LDFLAGS: -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64
I CC: cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
I CXX: g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -c llama.cpp -o llama.o

good catch @Thireus! thanks! - do you also have a GPU at hand so you can test this out? also, do you feel taking a stab at fixing it? otherwise I'll have a look soon

[ghost]

Hey there!
I was able to build localai from an nvidia/cuda image, modifying the Dockerfile to install golang like this:

ARG GO_VERSION=1.20.4
ARG BUILD_TYPE=cublas
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
ENV REBUILD=true
WORKDIR /build
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y checkinstall libgomp1 libopenblas-dev libopenblas-base libopencv-dev libopencv-core-dev git make
RUN apt-get install -y curl unzip
RUN curl -L https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz -o /usr/local/go${GO_VERSION}.linux-amd64.tar.gz
RUN tar -C /usr/local -xzf /usr/local/go${GO_VERSION}.linux-amd64.tar.gz
ENV PATH="$PATH::/usr/local/go/bin"
RUN curl -L https://github.com/Kitware/CMake/releases/download/v3.26.4/cmake-3.26.4-linux-x86_64.tar.gz -o /usr/local/cmake-3.26.4-linux-x86_64.tar.gz
RUN tar -C /usr/local -xzf /usr/local/cmake-3.26.4-linux-x86_64.tar.gz
ENV PATH="$PATH::/usr/local/cmake-3.26.4-linux-x86_64/bin"
RUN apt-get install -y ca-certificates
ENV PATH /usr/lib/go-${GO_VERSION}/bin:$PATH
COPY . .
RUN ln -s /usr/include/opencv4/opencv2/ /usr/include/opencv
RUN make build
EXPOSE 8080
ENTRYPOINT [ "/build/entrypoint.sh" ]

Ive run into a couple of issues:
• gpu_layers is ignored and i can't get it to offload any work to the GPU
• If i set REBUILD=false, then the GPU is not used and it assumes that the container is non-cublas/openblas
This is my config file:

- name: gpt-3.5-turboparameters:
model: Manticore-13B.ggmlv3.q4_0.bintemperature: 0.3context_size: 2048threads: 6backend: llamastopwords:
- "USER:"
- "### Instruction:"roles:
user: "USER:"system: "ASSISTANT:"assistant: "ASSISTANT:"gpu_layers: 40

Using the provided yaml like in model-gallery yield the error

ERR error loading config file: cannot load config file: cannot unmarshal config file: yaml: unmarshal errors:
line 1: cannot unmarshal !!map into []*api.Config

Cheers!