bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

[XixuHu]

Hi, I came across this problem when I try to use bitsandbytes to load a big model from huggingface, and I cannot fix it. My CUDA version is 12.0 and my torch version is 1.13.1+cu116. I would like to know if there is any way to solve this problem? Thanks!

[chris-aeviator]

facing the same issue when trying to resolve #134 with a downgrade to 0.35

[JanKalo]

Any news about this issue?

[Jacobsolawetz]

Hit it as well

(env) ubuntu@ip-172-31-7-128:~/uss-vision-deploy-demo$ pip list
Package                  Version    
------------------------ -----------
accelerate               0.16.0     
aiohttp                  3.8.4      
aiosignal                1.3.1      
async-timeout            4.0.2      
attrs                    22.2.0     
bitsandbytes             0.37.0     
boto3                    1.26.83    
botocore                 1.29.83    
certifi                  2022.12.7  
charset-normalizer       3.0.1      
click                    8.1.3      
datasets                 2.10.1     
deeplake                 3.2.12     
dill                     0.3.6      
entrypoints              0.4        
filelock                 3.9.0      
frozenlist               1.3.3      
fsspec                   2023.3.0   
hub                      3.0.1      
huggingface-hub          0.12.1     
humbug                   0.2.8      
idna                     3.4        
jmespath                 1.0.1      
loralib                  0.1.1      
multidict                6.0.4      
multiprocess             0.70.14    
numcodecs                0.11.0     
numpy                    1.24.2     
nvidia-cublas-cu11       11.10.3.66 
nvidia-cuda-nvrtc-cu11   11.7.99    
nvidia-cuda-runtime-cu11 11.7.99    
nvidia-cudnn-cu11        8.5.0.96   
packaging                23.0       
pandas                   1.5.3      
pathos                   0.3.0      
peft                     0.2.0.dev0 
Pillow                   9.4.0      
pip                      20.0.2     
pkg-resources            0.0.0      
pox                      0.3.2      
ppft                     1.7.6.6    
psutil                   5.9.4      
pyarrow                  11.0.0     
PyJWT                    2.6.0      
python-dateutil          2.8.2      
pytz                     2022.7.1   
PyYAML                   6.0        
regex                    2022.10.31 
requests                 2.28.2     
responses                0.18.0     
s3transfer               0.6.0      
setuptools               44.0.0     
six                      1.16.0     
tokenizers               0.13.2     
torch                    1.13.1     
tqdm                     4.64.1     
transformers             4.27.0.dev0
typing-extensions        4.5.0      
urllib3                  1.26.14    
wheel                    0.38.4     
xxhash                   3.2.0      
yarl                     1.8.2      

[silvacarl2]

I have the same problem, any ideas?

[zhangxianwei2015]

same issue

[jcharlet]

• I followed solution offered on Warning: "The installed version of bitsandbytes was compiled without GPU support." #112,
  replace lib/python3.9/site-packages/bitsandbytes/lib/bitsandbytes_cpu.so with lib/python3.9/site-packages/bitsandbytes/lib/bitsandbytes_cuda114.so (with my version of cuda)
• to solve following error 'CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected', I followed that suggestion of installing conda toolkit in new env
  conda install cudatoolkit

[silvacarl2]

pip install -i https://test.pypi.org/simple/ bitsandbytes

that worked for me.

[leezjs]

• replace lib/python3.9/site-packages/bitsandbytes/lib/bitsandbytes_cpu.so with lib/python3.9/site-packages/bitsandbytes/lib/bitsandbytes_cuda114.so (with my version of cuda)

This works for me, thanks : )

[boersmamarcel]

Manual copy of the .so file worked. I have version cuda version 11.7 so the following command in the conda environment directory ensured that it worked correctly again.

cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

[moorehousew]

Manual copy of the .so file worked. I have version cuda version 11.7 so the following command in the conda environment directory ensured that it worked correctly again.

cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

This works without a conda env as well, fwiw.

[XuJianzhi]

• bitsandbytes_cpu.so

not work

[XuJianzhi]

thanks, it works !!

[lywinged]

Manual copy of the .so file worked. I have version cuda version 11.7 so the following command in the conda environment directory ensured that it worked correctly again.

cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

It works!!! 😄

[sfxworks]

Yikes this also appears official anaconda image
edit: ..and the pytorch image
oh man

Adding that cp as a final run command solves this though yay

[sooftware]

• bitsandbytes_cpu.so

It works Thank you!

[bolongliu]

conda install cudatoolkit -y

that worked for me.

[Sritharan-racap]

conda install cudatoolkit -y

that worked for me.

thx it worked for me

[anda-ren]

Manual copy of the .so file worked. I have version cuda version 11.7 so the following command in the conda environment directory ensured that it worked correctly again.

cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

@boersmamarcel Thank you for the answer, it saved my day. Do you have any solution other than manually replacing the so file. I am doing things like model finetune automation

[Daryl149]

I needed an additional step: since I am on CUDA 11.6, I needed to delete the 11.7 .so file from packages. Otherwise the cp command above would not fix it.

In my venv (no conda) it worked as follows:

pip install -i https://test.pypi.org/simple/ bitsandbytes #make sure to delete this package first if you already installed another version.
rm venv_1/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
cp venv_1/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda116.so venv_1/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so 

after this, my python code ran fine:

from transformers import GPTNeoXForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5")
model = GPTNeoXForCausalLM.from_pretrained("OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5",device_map="auto", load_in_8bit=True)

[WakingHours-GitHub]

it not work for me, i don't know why. i try for my all of best, but it still don't work.
this error still in hear!

[Daryl149]

it not work for me, i don't know why. i try for my all of best, but it still don't work. this error still in hear!

are you able to run nvidia-smi successfully in the terminal? If so, take note of the CUDA version (11.x). Follow my steps above, but make sure to rm all libbitsandbytes_cuda11x.so files in the bitsandbytes folder greater than the one you have.

[bbecausereasonss]

pip install -i https://test.pypi.org/simple/ bitsandbytes

Jesus... This finally worked. I've been looking for a solution all week!

[Daryl149]

it do not work for me

when I cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so error： /arrow/cpp/src/arrow/filesystem/s3fs.cc:2598: arrow::fs::FinalizeS3 was not called even though S3 was initialized. This could lead to a segmentation fault at exit

That looks like an S3 and arrow specific error, can't help you there :/

[dacorvo]

I came across the same issue and after running the bug report checks, it turns out my CUDA runtime installation was not properly detected by bitsandbytes, leading to a CPU fallback:

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
...
<local_path>/hf_venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
  warn(msg)
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary <local_path>/hf_venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...

To fix this, one can simply add the path to the CUDA runtime library (in my configuration it is under a virtual env at <local_path>/hf_venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib but it depends on your local setup) to the LD_LIBRARY_PATH:

$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:<local_path>/hf_venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib
$ python -m bitsandbytes
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin <local_path>/hf_venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
CUDA SETUP: CUDA runtime path found: <local_path>/hf_venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
...
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['7.5']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...

SUCCESS!
Installation was successful!

Once CUDA runtime was properly detected, I was able to load the model and run an inference as expected (with a nice speedup too).

[SingL3]

My solution:

# find / -name libcuda.so.1
/usr/local/nvidia/lib64/libcuda.so.1
/usr/local/cuda-11.2/compat/libcuda.so.1
# cd /usr/local/nvidia/lib64
# ln -s libcuda.so.1 libcuda.so

Hope this helps you.

[yoqoo]

thanks, it works for me.

[crasylph]

in my situation, the BUG REPORT says:
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
so I linked /usr/local/cuda/lib64/libcudart.so.11.0 to /usr/local/cuda/lib64/libcudart.so and the problem is solved
ln -s /usr/local/cuda/lib64/libcudart.so.11.0 /usr/local/cuda/lib64/libcudart.so

[lucas0]

This is the right solution

To fix this, one can simply add the path to the CUDA runtime library (in my configuration it is under a virtual env at <local_path>/hf_venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib but it depends on your local setup) to the LD_LIBRARY_PATH:

[havietisov]

This is the right solution

To fix this, one can simply add the path to the CUDA runtime library (in my configuration it is under a virtual env at <local_path>/hf_venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib but it depends on your local setup) to the LD_LIBRARY_PATH:

well, this shouldn't even be the case

[wzrzt]

Manual copy of the .so file worked. I have version cuda version 11.7 so the following command in the conda environment directory ensured that it worked correctly again.

cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

It works for me, too. Thank you very much. I'm running it on wsl2 on windows 11

[sumukshashidhar]

My solution:

# find / -name libcuda.so.1
/usr/local/nvidia/lib64/libcuda.so.1
/usr/local/cuda-11.2/compat/libcuda.so.1
# cd /usr/local/nvidia/lib64
# ln -s libcuda.so.1 libcuda.so

Hope this helps you.

+1ing this. A better, system level fix imo

[satyanshu404]

conda install cudatoolkit -y

that worked for me.

Finally This worked for me, thanks man you saved my day

[MaxMls]

Manual copy of the .so file worked. I have version cuda version 11.7 so the following command in the conda environment directory ensured that it worked correctly again.

cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

Thank you. It works on WSL as well. I am using CUDA for WSL https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_network

[r3curs10n]

For people who are using the manual copy "cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so" approach, a cleaner solution is as follows.

This is how bitsandbytes tries to locate libcudart.so: https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/cuda_setup/env_vars.py#L47

So you basically need to export an environment variable that contains values that look like a path (i.e. contain a /) and bitsandbytes will pick your libcudart.so from there.

For example:

$ locate libcudart.so

<local_path>/.env/lib/python3.8/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0

$ export CUDART_PATH="<local_path>/.env/lib/python3.8/site-packages/nvidia/cuda_runtime/lib"
$ python -m bitsandbytes <--- this should succeed

@TimDettmers I'm wondering if we should modify the script to look for a libcudart.so file in the $VIRTUAL_ENV directory. This would solve the issue out of the box for people who use it via a virtual environment on popular cloud providers (GCP, LambdaLabs, etc)

[puppetm4st3r]

for me worked from python:

import os
cuda_runtime_path = "/home/dario/nlp_test/venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib" # path of my virtual env
os.environ['LD_LIBRARY_PATH'] = f"{os.environ.get('LD_LIBRARY_PATH')}:{cuda_runtime_path}"

[swumagic]

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[boolmriver]

"I also encountered this issue. Later, I found out that the cloud server I rented updated its system, and I needed to manually load CUDA each time."

[FarziBuilder]

I tried implementing this solution
cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

But the error still remains the same. Any ideas?

I have tried both installing bitsandbytes from source and thru pip install.

I am on wsl & CUDA 12.3 I am using python venv and ran the lib cmd by finding the bitsandbytes folder in the lib folder of env3

[kerkathy]

Manual copy of the .so file worked. I have version cuda version 11.7 so the following command in the conda environment directory ensured that it worked correctly again.
cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

Works for me also (I'm having CUDA 11.4)! For ppl seeing this just make sure you go the env path indicated in the error msg (for me, it's /anaconda3/envs/<env_name>/lib/python3.9/site-packages/bitsandbytes) and the above trick would work!

@FarziBuilder perhaps you could find something similar to libbitsandbytes_cuda123.so as you're having CUDA 12.3?