problems with CUDA 12.1: missing named symbol

[paniq]

I compiled bitsandbytes from source for tloen/alpaca-lora and CUDA_VERSION=121, but execution failed with this error:

CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary <REDACTED>/alpaca-lora/venv/lib/python3.10/site-packages/bitsandbytes-0.37.1-py3.10.egg/bitsandbytes/libbitsandbytes_cuda121.so...
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. 
The class this function is called from is 'LlamaTokenizer'.
Error named symbol not found at line 508 in file <REDACTED>/alpaca-lora/bitsandbytes/csrc/ops.cu

What else can I do to figure out what went wrong? is CUDA 12.1 supported yet?

The system is a Ubuntu 22.04 with a RTX 3060 (driver 530). Other pytorch setups work without issue.

[gkamer8]

having the same issue. good luck and please follow up if you fix it. thank you

also- I'm using CUDA 11.6, so doesn't seem like a version specific problem. And I'm also compiling from source for tloen/alpaca-lora

what's the size of your libbitsandbytes_cuda121.so? mine was around 1.7 mb, which didn't look right when comparing them to the older object files provided by the pip package.

i tried using the libbitsandbytes_cuda.so that i built (in the bitsandbytes dir) in place of the cuda121 object file in my site-packages, and it appears to be working. that object was 7.1 mb for me, which looked a lot better.

i haven't spent any time figuring out why this is the case, but it appears to work for me.

[weitunglin]

@J4ded's solution worked for me.

just copy the libbitsandbytes_cuda_120.so in site-packages to libbitsandbytes_cuda_121.so, then i appears to work for me as well.

tested on RTX 3060 with cuda 12.1.

if the cuda120.so file works too without building the other object i used, that’s great.

glad to hear it’s working. i spent way too long fixing this.

[rrfaria]

I had same problem, And I solved building version 12.1.
if you want to build this, you can do:

git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
export CUDA_HOME=/usr/local/cuda-12.1 && make cuda12x CUDA_VERSION=121
export CUDA_HOME=/usr/local/cuda-12.1 && make cuda12x_nomatmul CUDA_VERSION=121
CUDA_VERSION=121 && python setup.py install

this will generate 2 files one with approximately 7mb and other with 14mb

libbitsandbytes_cuda121.so
libbitsandbytes_cuda121_nocublaslt.so

Copy those files to

python3.9/site-packages/bitsandbytes/

Restart the env and try again

I guess what @weitunglin said can also help

I had same problem, And I solved building version 12.1. if you want to build this, you can do:

git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
export CUDA_HOME=/usr/local/cuda-12.1 && make cuda12x CUDA_VERSION=121
export CUDA_HOME=/usr/local/cuda-12.1 && make cuda12x_nomatmul CUDA_VERSION=121
CUDA_VERSION=121 && python setup.py install

that explains where my libbitsandbytes_cuda.so came from and why it works. i forgot to set CUDA_VERSION when trying making cuda12x, so it came out without the 121 at the end. kudos for figuring it out.

[rrfaria]

Only issue I found using this way is:
using multi-GPU with different memory size 12GB and 8GB

UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at .[./c10/cuda/CUDAFunctions.cpp:109)
  return torch._C._cuda_getDeviceCount() > 0

UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. 

I can check gpus using:

!nvidia-smi

GPU are there
if I try use gpu with same sizes

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "0,2"

it works but now

the error is in transformers

RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1

I guess If I change 2 for slot 1 maybe fix.

but the problem with 12Gb remains

[xNul]

In my case, I just used an older version of PyTorch. It took some time but I got it working again on Windows without WSL. I wrote up some instructions which may be helpful.

[TimDettmers]

This has been resolved with the latest release.

[swumagic]

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

[jinjinran2001]

I had same problem, And I solved building version 12.1. if you want to build this, you can do:

git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
export CUDA_HOME=/usr/local/cuda-12.1 && make cuda12x CUDA_VERSION=121
export CUDA_HOME=/usr/local/cuda-12.1 && make cuda12x_nomatmul CUDA_VERSION=121
CUDA_VERSION=121 && python setup.py install

this will generate 2 files one with approximately 7mb and other with 14mb

libbitsandbytes_cuda121.so
libbitsandbytes_cuda121_nocublaslt.so

Copy those files to

python3.9/site-packages/bitsandbytes/

Restart the env and try again

I guess what @weitunglin said can also help

Hi, I am using terminal inside VSC, so when I type this: CUDA_VERSION=121 && python setup.py install it says it is not recognized, especially for CUDA_VERSION=121. Are you running on Linux?

[ani0075saha]

@rrfaria

Two questions:

(1) I am getting these messages.

make: *** No rule to make target 'cuda12x'.  Stop.
make: *** No rule to make target 'cuda12x_nomatmul'.  Stop.

Is this a problem? Do I need to modify something in the make commands?

(2) I cannot find the files

libbitsandbytes_cuda121.so
libbitsandbytes_cuda121_nocublaslt.so

I do not see them in the bitsandbytes folder. Where are these generated?