bitsandbytes just never works.

[venturaEffect]

I've tried with different repos to finetune llms. Alsways when I see bitsandbytes are needed I already know this will be a pain in the ass. And this time was no difference.

Used the repo of Llama-Factory to fine tune a model. And needed bitsandbytes.

It tells there is a bug and following the instructions I write python -m bitsandbytes to see what is wrong. But, it just sends me the Bug Report from before.

`===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll
False
C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\cuda_setup\main.py:156: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('C:/Users/zaesa/anaconda3/envs/llama_factory/bin')}
warn(msg)
C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\cuda_setup\main.py:156: UserWarning: C:\Users\zaesa\anaconda3\envs\llama_factory did not contain ['cudart64_110.dll', 'cudart64_120.dll', 'cudart64_12.dll'] as expected! Searching further paths...
warn(msg)
CUDA SETUP: CUDA runtime path found: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\cudart64_110.dll
CUDA SETUP: Highest compute capability among GPUs detected: 8.9
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll...
Could not find module 'C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll' (or one of its dependencies). Try using the full path with constructor syntax.
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=118 make cuda11x
python setup.py install
Traceback (most recent call last):
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, Error)
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\runpy.py", line 146, in get_module_details
return get_module_details(pkg_main_name, error)
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\runpy.py", line 110, in get_module_details
import(pkg_name)
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes_init.py", line 6, in
from . import cuda_setup, utils, research
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\research_init.py", line 1, in
from . import nn
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\research\nn_init.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\optim_init.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "C:\Users\zaesa\anaconda3\envs\llama_factory\lib\site-packages\bitsandbytes\cextension.py", line 20, in
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues`

Would really appreciate how to solve this. I can't imagine how many people is going through this for fine tune and use LLMs. Because doesn't matter the path I take, when there is bitsandbytes involved it just breaks things.

Appreciate any suggestion.

[iamanthos]

I am there with you. Same is happening with me.

[cbjrobertson]

Can I bump this 1000 times?

[josemerinom]

colab
===================================BUG REPORT===================================
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

warn(msg)

/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: /usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}
The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('//172.28.0.1'), PosixPath('8013')}
The following directories listed in your path were found to be non-existent: {PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-t4-s-k1i9keaqrl3p --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')}
The following directories listed in your path were found to be non-existent: {PosixPath('/datalab/web/pyright/typeshed-fallback/stdlib,/usr/local/lib/python3.10/dist-packages')}
The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//ipykernel.pylab.backend_inline')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')}
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...
libcusparse.so.11: cannot open shared object file: No such file or directory

[Arindam75]

Same problem here.
I removed all the NVIDIA Drivers and CUDA related stuff. Re-installed cuda_12.0.0_527.41_windows.exe
Exactly same problem.

  RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):

  CUDA Setup failed despite GPU being available. Please run the following command to get more information:

  python -m bitsandbytes

  Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
  to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
  and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

[josemerinom]

In colab, I use:

1. !pip install -U bitsandbytes

2. !pip install bitsandbytes==...( version: v35.. v40 v41 )

3. !git clone https://github.com/timdettmers/bitsandbytes.git
   %cd bitsandbytes
   !CUDA_VERSION=118 make cuda11x
   !python setup.py install

4. !wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/install_cuda.sh
   !bash install_cuda.sh 118 ~/local 1

5. https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Linux

6. ...
   !apt-get install cuda...

7. I tried version 12.2, which is installed by default

Colab has cuda version 12.2, I reinstalled version 11.8, I had both versions, I used the suggested installation in the repository and it didn't work for me

[venturaEffect]

Yeah, I tried everything, installed from scratch CUDA different versions, compatibility with Pytorch and also with bitsandbytes. Also on different enviroments for different repos. Nothing, it just never works.

[josemerinom]

I have changed the patch location of the libraries, I have copied the libraries to the location that requests bit, the only file that I cannot find is libcudart.so in google colab, (libcudart.so.11 and libcudart.so.12 if I have them)

[Titus-von-Koeller]

Dear all, Sorry that you all are having these hard to solve issues. Unfortunately, a few people keep having these issues, partly to external reasons but I agree it super frustrating. The majority of downloads (millions) seem to be working, luckily enough, but let's try to figure this out together and try to improve the situation going forward. I'll look more into this next week, but I would be very interested if this proposed fix changes your situation at all: #915 (comment) Another approach to debug dynamic linking issues would be to run `ldd` on the respective bnb binary and see if it's linked with the correct dependencies. Unfortunately these things are tricky because through pip we can only control the Python part of the equation and package binaries, but there are a bunch of dependencies that come from the system libraries and CUDA install libraries, that have to be detected and linked together correctly. The situation is a bit complex unfortunately and only partially under bnb control. The trace that you're all sharing is our attempt of helping in detecting and guiding to resolve these issues. Obviously, there's still potential for improvement that's why I just opened an improvement proposal RFC on the topic. The situation becomes even trickier with Windows, which is only has limited (unofficial) support in bnb. Sorry again for your struggles. We re working hard to get on top of maintenance issues, as this library has only recently found sponsorship and has a big backlog. Thanks also for any info that you're providing to help along with improvements. Best, Titus

…

On Sat, Dec 16, 2023, 16:37 Jose Esteban Merino ***@***.***> wrote: I have changed the patch location of the libraries, I have copied the libraries to the location that requests bit, the only file that I cannot find is libcudart.so in google colab, (libcudart.so.11 and libcudart.so.12 if I have them) — Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFBEOZOHIK56USDH2PRU2LYJW53DAVCNFSM6AAAAABAU4BEKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJYHA2DKNRVGQ> . You are receiving this because you were assigned.Message ID: ***@***.***>

[josemerinom]

I understand the situation, last week I was training in colab with bitsandbytes (I did tests and trained 6 to 8 models every day), but one morning, I started to get the error.
Sorry if I'm only talking about collab, but it's where I work.
In colab, every time it is used, bitsandbytes must be installed, I installed the latest version.
What changed from a few days ago to today?
I don't know if there was any update to bitsandbytes, or in my case, something changed in colab.
In the error message it indicates places where you search for the libraries, perhaps one of those sites is down and producing the error. I don't know, I speak without knowing about programming.

Thank you for your work and we hope to find a solution

[Titus-von-Koeller]

Hey! Thanks for getting back to me. Which type of instance are you using, is it with a T4 GPU? Can you share the notebook with me, so I can take a look + debug? Is it now always not working or only some of the time?

…

On Sat, Dec 16, 2023, 17:36 Jose Esteban Merino ***@***.***> wrote: I understand the situation, last week I was training in colab with bitsandbytes (I did tests and trained 6 to 8 models every day), but one morning, I started to get the error. Sorry if I'm only talking about collab, but it's where I work. In colab, every time it is used, bitsandbytes must be installed, I installed the latest version. What changed from a few days ago to today? I don't know if there was any update to bitsandbytes, or in my case, something changed in colab. In the error message it indicates places where you search for the libraries, perhaps one of those sites is down and producing the error. I don't know, I speak without knowing about programming. Thank you for your work and we hope to find a solution — Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFBEO7TTTQINUDK7IFQWCTYJXEZBAVCNFSM6AAAAABAU4BEKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJYHA3DENJWGU> . You are receiving this because you were assigned.Message ID: ***@***.***>

[josemerinom]

here I uploaded my notebook + debugging messages
It is the notebook I used to train
the only modification is:
!pip install bitsandbytes > !pip install bitsandbytes==0.41.1

https://colab.research.google.com/github/josemerinom/test/blob/master/test.ipynb

pd:

[bcallonnec]

In colab i had the same issue,

command !nvcc --version outputs -> cuda version 12.2
But command torch.version.cuda outputs -> 11.7

So i followed the recommandations of this post to install cuda 11-7

[apurtell]

#934

So i followed the recommandations of this post to install cuda 11-7

If you have Colab Pro you can get a shell, so I followed those recommendations, but nvcc --version still outputs "release 12.2" for me afterward. 🤷