Unable to setup environment

[FirokOtaku]

Error messages:

Collecting wandb (from ChatUniVi==1.0.1)
  Using cached https://mirrors.aliyun.com/pypi/packages/5c/81/1780aa129564b11709a6d7f0739257174f0a3a1b432ba804b3c6f00e0f88/wandb-0.16.0-py3-none-any.whl (2.1 MB)
Collecting shortuuid (from ChatUniVi==1.0.1)
  Using cached https://mirrors.aliyun.com/pypi/packages/c3/46/644a4df3061e96ef24998c0623d3b12287090ab9a0e0d6ad8408f7b87283/shortuuid-1.0.11-py3-none-any.whl (10 kB)
Collecting httpx==0.24.0 (from ChatUniVi==1.0.1)
  Using cached https://mirrors.aliyun.com/pypi/packages/4e/c1/692013f1e6115a061a14f6c7d05947515a1eb7b85ef6e9bf0ffbf0e92738/httpx-0.24.0-py3-none-any.whl (75 kB)
Collecting deepspeed==0.9.5 (from ChatUniVi==1.0.1)
  Using cached https://mirrors.aliyun.com/pypi/packages/99/0f/a4ebd3b3f6a8fd9bca77ca5f570724f3902ca90b491f8146e45c9733e64f/deepspeed-0.9.5.tar.gz (809 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      test.c
      LINK : fatal error LNK1181: 无法打开输入文件“aio.lib”
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\firok\AppData\Local\Temp\pip-install-vtdgutu9\deepspeed_b542e0f942ab421ab08ed6e316a8e493\setup.py", line 163, in <module>
          abort(f"Unable to pre-compile {op_name}")
        File "C:\Users\firok\AppData\Local\Temp\pip-install-vtdgutu9\deepspeed_b542e0f942ab421ab08ed6e316a8e493\setup.py", line 51, in abort
          assert False, msg
      AssertionError: Unable to pre-compile async_io
      DS_BUILD_OPS=1
       [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
       [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
       [WARNING]  One can disable async_io with DS_BUILD_AIO=0
       [ERROR]  Unable to pre-compile async_io
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

How to reproduce:

conda create -n chatunivi python=3.10 -y
conda activate chatunivi
pip install --upgrade pip
# Using PyTorch 11.8
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
# Error occurs when running command below
pip install -e .

System info:

• Windows 10 22H2

[jpthu17]

Thank you for your valuable feedback. I have replicated the error and am actively seeking a resolution.

[FirokOtaku]

Okay found solution to this problem.
Windows is not well supported by DeepSpeed. We need to compile it manually.

Pull DeepSpeed repo:

git clone https://github.com/microsoft/DeepSpeed.git
cd DeepSpeed
# rollback files to DeepSpeed 0.9.5
# because in the previous step, the command line log shows that the version being installed is this one
# using higher version may be okay?
git reset --hard 8b7423d2

When compiling DeepSpeed, you may encounter a type conversion error between size_t and _Ty.
Just add (unsigned) into line 536, 537, 545, 546, 1570 for csrc/transformer/inference/csrc/pt_binding.cpp:

// line 536, 537
{hidden_dim * (unsigned) InferenceContext::Instance().GetMaxTokenLength(),
    k * (unsigned) InferenceContext::Instance().GetMaxTokenLength(),

// line 545, 546
{hidden_dim * (unsigned) InferenceContext::Instance().GetMaxTokenLength(),
    k * (unsigned) InferenceContext::Instance().GetMaxTokenLength(),

// line 1570
    at::from_blob(intermediate_ptr, {input.size(0), input.size(1), (unsigned) mlp_1_out_neurons}, options);

Compile and install DeepSpeed

# use same conda env to Chat-UniVi is okay
conda activate chatunivi
# run build script for Windows
build_win.bat
# then should a deepspeed-0.9.5-....whl file be in dist folder
# install it with pip
pip install dist/deepspeed-0.9.5-....whl

Now pip install -e . command of Chat-UniVi should run successfully.

---

Here is the .whl file built on my machine.
It's built by WinSDK 10.0.22000.0 with CUDA 11.7.
Don't know if it can help you guys.

deepspeed-0.9.5+8b7423d2-cp310-cp310-win_amd64.whl.zip

---

Then another problem rises: Windows is not well supported by flash-attn too. 🤣🤣

I'm trying to solve this now.

[jpthu17]

Thank you very much!

Are you planning to train the model on Windows? If you only intend to perform inference, there's no need to install deepspeed and flash-attn.

[FirokOtaku]

Not yet. I'm just trying to setup environment and evaluate inference performance.

Thanks for that information! I will continue with testing. 😆😆

[FirokOtaku]

Error occurs when running python main_demo_13B.py:

logs

T:\Projects\Chat-UniVi\main_demo_7B.py:17: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  assert model_path is not ""
False
===================================BUG REPORT===================================

C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes\cuda_setup\main.py:166: UserWarning: Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
warn(msg)
The following directories listed in your path were found to be non-existent: {WindowsPath('C'), WindowsPath('/Users/firok/.conda/envs/chatunivi/lib')}

C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes\cuda_setup\main.py:166: UserWarning: C:\Users\firok.conda\envs\chatunivi did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...

warn(msg)

CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...

The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}

DEBUG: Possible options found for libcudart.so: set()

CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5.

CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md

CUDA SETUP: Loading binary C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so...

argument of type 'WindowsPath' is not iterable

CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.

CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable

CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null

CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a

CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc

CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.

CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh

CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.

CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local

Traceback (most recent call last):

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\transformers\utils\import_utils.py", line 1099, in get_module

return importlib.import_module("." + module_name, self.name)

File "C:\Users\firok.conda\envs\chatunivi\lib\importlib_init.py", line 126, in import_module

return _bootstrap.gcd_import(name[level:], package, level)

File "", line 1050, in gcd_import

File "", line 1027, in find_and_load

File "", line 1006, in find_and_load_unlocked

File "", line 688, in load_unlocked

File "", line 883, in exec_module

File "", line 241, in call_with_frames_removed

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\transformers\models\llama\modeling_llama.py", line 32, in 

from ...modeling_utils import PreTrainedModel

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\transformers\modeling_utils.py", line 38, in 

from .deepspeed import deepspeed_config, is_deepspeed_zero3_enabled

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\transformers\deepspeed.py", line 37, in 

from accelerate.utils.deepspeed import HfDeepSpeedConfig as DeepSpeedConfig

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\accelerate_init.py", line 3, in 

from .accelerator import Accelerator

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\accelerate\accelerator.py", line 35, in 

from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\accelerate\checkpointing.py", line 24, in 

from .utils import (

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\accelerate\utils_init.py", line 131, in 

from .bnb import has_4bit_bnb_layers, load_and_quantize_model

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\accelerate\utils\bnb.py", line 42, in 

import bitsandbytes as bnb

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes_init.py", line 6, in 

from . import cuda_setup, utils, research

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes\research_init.py", line 1, in 

from . import nn

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes\research\nn_init.py", line 1, in 

from .modules import LinearFP8Mixed, LinearFP8Global

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in 

from bitsandbytes.optim import GlobalOptimManager

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes\optim_init.py", line 6, in 

from bitsandbytes.cextension import COMPILED_WITH_CUDA

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\bitsandbytes\cextension.py", line 20, in 

raise RuntimeError('''

RuntimeError:

CUDA Setup failed despite GPU being available. Please run the following command to get more information:
    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "T:\Projects\Chat-UniVi\main_demo_7B.py", line 4, in 

from ChatUniVi.conversation import conv_templates, Conversation

File "T:\Projects\Chat-UniVi\ChatUniVi_init_.py", line 1, in 

from .model import ChatUniViLlamaForCausalLM

File "T:\Projects\Chat-UniVi\ChatUniVi\model_init_.py", line 1, in 

from .language_model.llama import ChatUniViLlamaForCausalLM, ChatUniViConfig

File "T:\Projects\Chat-UniVi\ChatUniVi\model\language_model\llama.py", line 5, in 

from transformers import AutoConfig, AutoModelForCausalLM, 

File "", line 1075, in _handle_fromlist

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\transformers\utils\import_utils.py", line 1090, in getattr

value = getattr(module, name)

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\transformers\utils\import_utils.py", line 1089, in getattr

module = self._get_module(self._class_to_module[name])

File "C:\Users\firok.conda\envs\chatunivi\lib\site-packages\transformers\utils\import_utils.py", line 1101, in _get_module

raise RuntimeError(

RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
    CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

According to this document, manually setting environment varibles (BNB_CUDA_VERSION and LD_LIBRARY_PATH) can solve that.

---

Then comes a lot of module not found errors.

Running pip install gradio transformers timm accelerate decord SentencePiece protobuf with admin permission (dont know why. installation fails if no admin permission) helps.

---

I will continue to test tomorrow. 🚲

[FirokOtaku]

Adding offload_folder="offload" param to AutoModelForCausalLM.from_pretrained in model/builder.py should fix OOM error of loading model.

line 75:

model = AutoModelForCausalLM.from_pretrained(model_path, offload_folder='offload', low_cpu_mem_usage=True, **kwargs)

line 82:

model = AutoModelForCausalLM.from_pretrained(model_base, offload_folder='offload', torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map="auto")

line 92

model = AutoModelForCausalLM.from_pretrained(model_path, offload_folder='offload', low_cpu_mem_usage=True, **kwargs)

---

set CUDA_VISIBLE_DEVICES=0 && set BNB_CUDA_VERSION=117 && set LD_LIBRARY_PATH="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin" && uvicorn main_demo_7B:app --host 0.0.0.0 --port 8888

bitsandbytes lib raise the CUDA Setup failed error again. 😭

[jpthu17]

bitsandbytes is used to speed up model inference and appears to be optional. However, I'm currently occupied with another deadline, and I plan to test the demo on Windows over this weekend.

[Yue-S]

Adding offload_folder="offload" param to AutoModelForCausalLM.from_pretrained in model/builder.py should fix OOM error of loading model.

line 75:

model = AutoModelForCausalLM.from_pretrained(model_path, offload_folder='offload', low_cpu_mem_usage=True, **kwargs)

line 82:

model = AutoModelForCausalLM.from_pretrained(model_base, offload_folder='offload', torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map="auto")

line 92

model = AutoModelForCausalLM.from_pretrained(model_path, offload_folder='offload', low_cpu_mem_usage=True, **kwargs)

set CUDA_VISIBLE_DEVICES=0 && set BNB_CUDA_VERSION=117 && set LD_LIBRARY_PATH="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin" && uvicorn main_demo_7B:app --host 0.0.0.0 --port 8888

bitsandbytes lib raise the CUDA Setup failed error again. 😭

Hello, I've encountered the same problem. It seems that bitsandbytes didn't support Windows very well.
You can try to uninstall it and then try pip install bitsandbytes-windows.