Clarification on g++ / libcxx requirements: `GLIBCXX_3.4.32' not found

[pramodk]

Hello All,

This is somewhat related to #907 but I wanted to explicitly clarify the g++ / libcxx requirements - hence this issue.

I am on Ubuntu 22.04 with g++ v11 and python 3.10, and have bitsandbytes version 0.46.0 installed:

(venv) root@login-0:~# pip show bitsandbytes
Name: bitsandbytes
Version: 0.46.0
Summary: k-bit optimizers and matrix multiplication routines.
Home-page: https://github.com/bitsandbytes-foundation/bitsandbytes
...
Location: /root/dgx-stage/venv/lib/python3.10/site-packages
Requires: numpy, torch

However, attempting to load the module results in the following error:

Could not load bitsandbytes native library: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /root/dgx-stage/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so)

Traceback (most recent call last):
  File "/root/dgx-stage/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 85, in <module>
    lib = get_native_library()
  File "/root/dgx-stage/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 72, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
  File "/usr/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found

My system's libstdc++ doesn't have GLIBCXX_3.4.32:

# strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_3.4.22
GLIBCXX_3.4.23
GLIBCXX_3.4.24
GLIBCXX_3.4.25
GLIBCXX_3.4.26
GLIBCXX_3.4.27
GLIBCXX_3.4.28
GLIBCXX_3.4.29
GLIBCXX_3.4.30
GLIBCXX_DEBUG_MESSAGE_LENGTH

Based on this StackOverflow thread, it seems GLIBCXX_3.4.32 is from g++-13.

As a workaround, I built bitsandbytes from source, which avoids the libcxx compatibility issue by compiling with system toolchain:

git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/
cmake -DCOMPUTE_BACKEND=cpu -S .
make
pip install -e .

I am opening this issue to confirm whether this is expected behavior and whether the wheel could be made backward-compatible (e.g., manylinux-compliant). I haven't checked the wheel build pipeline but suspect this may be a side effect of building with a newer GCC.

Thanks in advance!

Reproduction

As described above, install on Ubuntu 22.04 where GPU stack is not available and try to load the module.

Expected behavior

wheel should be compatible with platform like Ubuntu 22.04

[pramodk]

To add some additional information:

On a node where we have GPUs and CUDA driver installed:

(venv) root@worker-0:~/nemo-llama31_25# nvidia-smi
Sat May 31 06:25:41 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   34C    P0             70W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
...

(venv) root@worker-0:~/nemo-llama31_25# python -m bitsandbytes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
CUDA specs: CUDASpecs(highest_compute_capability=(9, 0), cuda_version_string='126', cuda_version_tuple=(12, 6))
PyTorch settings found: CUDA_VERSION=126, Highest Compute Capability: (9, 0).
To manually override the PyTorch CUDA version please see: https://github.com/TimDettmers/bitsandbytes/blob/main/docs/source/nonpytorchcuda.mdx
The directory listed in your path is found to be non-existent: SC|screen.xterm-256color|VT 100/ANSI X3.64 virtual terminal
CUDA SETUP: WARNING! CUDA runtime files not found in any environmental path.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking that the library is importable and CUDA is callable...
SUCCESS!
Installation was successful!

And the same environment on a node without GPU:

(venv) root@login-1:~# nvidia-smi
(venv) root@login-1:~# python -m bitsandbytes
Could not load bitsandbytes native library: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /root/dgx-stage-2/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so)
Traceback (most recent call last):
  File "/root/dgx-stage-2/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 85, in <module>
    lib = get_native_library()
  File "/root/dgx-stage-2/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 72, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
  File "/usr/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /root/dgx-stage-2/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
CUDA specs: None
Torch says CUDA is not available. Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
CUDA SETUP: WARNING! CUDA runtime files not found in any environmental path.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking that the library is importable and CUDA is callable...
Couldn't load the bitsandbytes library, likely due to missing binaries.
Please ensure bitsandbytes is properly installed.

For source installations, compile the binaries with `cmake -DCOMPUTE_BACKEND=cuda -S .`.
See the documentation for more details if needed.

Trying a simple check anyway, but this will likely fail...
Traceback (most recent call last):
  File "/root/dgx-stage-2/venv/lib/python3.10/site-packages/bitsandbytes/diagnostics/main.py", line 66, in main
    sanity_check()
  File "/root/dgx-stage-2/venv/lib/python3.10/site-packages/bitsandbytes/diagnostics/main.py", line 33, in sanity_check
    p = torch.nn.Parameter(torch.rand(10, 10).cuda())
  File "/root/dgx-stage-2/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 372, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx
Above we output some debug information.
Please provide this info when creating an issue via https://github.com/TimDettmers/bitsandbytes/issues/new/choose
WARNING: Please be sure to sanitize sensitive info from the output before posting it.

[pramodk]

I looked into details and here is the summary if anyone ends up in a similar issue:

• In our packages, we have added dependency pinned to v0.45.3.

• From Build: use ubuntu-22.04 instead of 24.04 for CPU build (glibc compat) #1538 we know that v0.45.3 was using Ubuntu 24.04 which brings gcc-13:

  root@48991504ed18:/# gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04)
  root@48991504ed18:/# strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX
  ..
  GLIBCXX_3.4.32
  GLIBCXX_3.4.33

• Using v0.45.4 is missing libbitsandbytes_cpu.so (libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory #1584), and hence one needs to use at least v0.45.5 to ensure compatibility with Ubuntu 22.04 and to use the CPU backend.

I will raise this internally to bump the dependency.

I was skimming through Python wheel building workflow, and the only question that came to my mind was: wouldn't it be better to build wheels under pypa's manylinux container images? This is to ensure that when runners (or their packages) are updated, we don't have to worry about compatibility issues. (mentioning from previous experiences)

Thanks again! Feel free to close the issue if you think there is no action needed here.

[matthewdouglas]

Hi,

Our minimum target is currently manylinux 2.24, so it's not expected. However it's correctly pointed out that there were a few issues with the libbitsandbytes_cpu.so build in the v0.45.x series,, but those are resolved in v0.45.5 and v0.46.0.

We have thought about using the manylinux container images, but there's a couple reasons why it was avoided::

1. The Debian 9 based manylinux 2.24 image has been EOL for a few years now. When we were setting up the build pipeline we were not ready to jump to the next image, manylinux 2.28, because that would exclude Amazon Linux 2 users. (With that said I am aware that torch distributions now are manylinux 2.28, but I'm still not sure we're ready to make that same jump).
2. For the CUDA builds it has been more convenient for us to rely on NVIDIA's containers than to build on top of the manylinux ones.

In order to improve this situation, I'm going to make the auditwheel step fail the whole build if we don't meet our target.

As an aside, I am wanting to consider dropping the libbitsandbytes_cpu.so build entirely, or at least making it optional, as we start to implement more CPU support. It only implements 8bit blockwise quantization, while all other bitsandbytes features on CPU so far have been implemented with plain PyTorch code or in external libraries like IPEX.

[pramodk]

Thank you @matthewdouglas for detailed response!

I didn't look much into cuda/gpu builds but the cpu code parts (common.cpp, cpu_ops.cpp, pythonInterface.cpp) doesn't seem to use any latest C++ standards feature (?) and hence wanted to clarify.

I have updated NeMo deepndency to avoid the issue with 0.45.3.

Thanks again!