Installation is a mess, instructions are a mess

[magnacartatron]

ChatGPT helped remove all the swear words.

I've encountered difficulties trying to set this up on Ubuntu, MacOS, and Windows. I've noticed some inconsistencies in the instructions, and it seems some tools and libraries might be either too new or outdated. It would be greatly appreciated if these issues could be addressed to make the project more user-friendly for everyone. Thank you.

[imoneoi]

We've tested the following installation instructions and pip package. What error message did you encounter, can you post it here?

conda create -y --name openchat
conda activate openchat

conda install -y python
conda install -y cudatoolkit-dev -c conda-forge
pip3 install torch torchvision torchaudio

pip3 install packaging ninja
pip3 install --no-build-isolation "flash-attn<2"

pip3 install ochat

[ugref]

pip 23.3.1 from C:\Python311\Lib\site-packages\pip (python 3.11)

With your commands : First 6 ok,
then :

PS C:\Users\xxxxxxxx\projects>pip3 install --no-build-isolation "flash-attn<2"
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting flash-attn<2
  Downloading flash_attn-1.0.9.tar.gz (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 7.2 MB/s eta 0:00:00
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [34 lines of output]
      No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3'
      fatal: not a git repository (or any of the parent directories): .git

      Warning: Torch did not find available GPUs on this system.
       If your intention is to cross-compile, this is not an error.
      By default, Apex will cross-compile for Pascal (compute capabilities 6.0, 6.1, 6.2),
      Volta (compute capability 7.0), Turing (compute capability 7.5),
      and, if the CUDA version is >= 11.0, Ampere (compute capability 8.0).
      If you wish to cross-compile for a single specific architecture,
      export TORCH_CUDA_ARCH_LIST="compute capability" before running setup.py.



      torch.__version__  = 2.1.0+cpu


      running dist_info
      creating C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info
      writing C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\PKG-INFO
      writing dependency_links to C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\dependency_links.txt
      writing requirements to C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\requires.txt
      writing top-level names to C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\top_level.txt
      writing manifest file 'C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\SOURCES.txt'
      reading manifest file 'C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.cu' under directory 'flash_attn'
      warning: no files found matching '*.h' under directory 'flash_attn'
      warning: no files found matching '*.cuh' under directory 'flash_attn'
      warning: no files found matching '*.cpp' under directory 'flash_attn'
      adding license file 'LICENSE'
      adding license file 'AUTHORS'
      writing manifest file 'C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn.egg-info\SOURCES.txt'
      creating 'C:\Users\xxxxxxxx\AppData\Local\Temp\pip-modern-metadata-2xvr8d9z\flash_attn-1.0.9.dist-info'
      error: invalid command 'bdist_wheel'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

and conda -v :

PS C:\Users\xxxxxxxx\projects> conda -v

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "C:\Users\xxxxxxxx\miniconda3\Lib\site-packages\conda\exception_handler.py", line 17, in __call__
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "C:\Users\xxxxxxxx\miniconda3\Lib\site-packages\conda\cli\main.py", line 78, in main_subshell
        exit_code = do_call(args, parser)
                    ^^^^^^^^^^^^^^^^^^^^^
      File "C:\Users\xxxxxxxx\miniconda3\Lib\site-packages\conda\cli\conda_argparse.py", line 160, in do_call
        relative_mod, func_name = args.func.rsplit(".", 1)
                                  ^^^^^^^^^
    AttributeError: 'Namespace' object has no attribute 'func'

`$ C:\Users\xxxxxxxx\miniconda3\Scripts\conda-script.py -v`

  environment variables:
                 CIO_TEST=<not set>
                CONDA_EXE=C:\Users\xxxxxxxx\miniconda3\condabin\..\Scripts\conda.exe
               CONDA_EXES="C:\Users\xxxxxxxx\miniconda3\condabin\..\Scripts\conda.exe"
               CONDA_ROOT=C:\Users\xxxxxxxx\miniconda3
                CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3
           CURL_CA_BUNDLE=<not set>
                 HOMEPATH=\Users\xxxxxxxx
               LD_PRELOAD=<not set>
                     PATH=C:\Program Files\NVIDIA GPU Computing
                          Toolkit\CUDA\v12.3\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\C
                          UDA\v12.3\libnvvp;C:\Python311\Scripts\;C:\Python311\;C:\Python312\Scr
                          ipts\;C:\Python312\;C:\Windows\system32;C:\Windows;C:\Windows\System32
                          \Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\
                          OpenSSH\;C:\ProgramData\chocolatey\bin;C:\Program
                          Files\nodejs\;C:\Program Files\Git\cmd;C:\Program Files\NVIDIA
                          Corporation\Nsight Compute 2023.3.0\;C:\Program Files (x86)\NVIDIA
                          Corporation\PhysX\Common;C:\Users\xxxxx
                          xxxxx\miniconda3;C:\Users\xxxxx
                          xxxxx\miniconda3\Library\mingw-w64\bin;C:\Users\xxxxx
                          xxxxx\miniconda3\Library\usr\bin;C:\Users\xxxxx
                          xxxxx\miniconda3\Library\bin;C:\Users\xxxxx
                          xxxxx\miniconda3\Scripts;C:\Users\xxxxx
                          xxxxx\.cargo\bin;C:\Users\xxxxx
                          xxxxx\AppData\Local\Microsoft\WindowsApps;C:\Users\xxxxx
                          xxxxx\AppData\Roaming\npm
             PSMODULEPATH=C:\Users\xxxxx
                          xxxxx\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPow
                          erShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : None
       user config file : C:\Users\xxxxxxxx\.condarc
 populated config files :
          conda version : 23.9.0
    conda-build version : not installed
         python version : 3.11.5.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.3=0
                          __win=0=0
       base environment : C:\Users\xxxxxxxx\miniconda3  (writable)
      conda av data dir : C:\Users\xxxxxxxx\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\xxxxxxxx\miniconda3\pkgs
                          C:\Users\xxxxxxxx\.conda\pkgs
                          C:\Users\xxxxxxxx\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\xxxxxxxx\miniconda3\envs
                          C:\Users\xxxxxxxx\.conda\envs
                          C:\Users\xxxxxxxx\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.9.0 requests/2.31.0 CPython/3.11.5 Windows/10 Windows/10.0.22621
          administrator : True
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.
If you suspect this error is being caused by a malfunctioning plugin,
consider using the --no-plugins option to turn off plugins.

Example: conda --no-plugins install <package>

Alternatively, you can set the CONDA_NO_PLUGINS environment variable on
the command line to run the command without plugins enabled.

Example: CONDA_NO_PLUGINS=true conda install <package>

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers? [y/N]: N

No report sent. To permanently opt-out, use

    $ conda config --set report_errors false

help pls

[DoodleBears]

I successfully install it on WSL 2 (Windows 11) with following step:

Environment

• I cannot make it works on my windows 11 (failed on installing (build) vLLM), but
• Windows 11, WSL 2, Anaconda works fine

Here is the Environment in WSL

(openchat) doodlebear@BEAR-PC:/mnt/c/Users/admin$ conda -v

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/doodlebear/miniconda3/lib/python3.11/site-packages/conda/exception_handler.py", line 17, in __call__
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/home/doodlebear/miniconda3/lib/python3.11/site-packages/conda/cli/main.py", line 78, in main_subshell
        exit_code = do_call(args, parser)
                    ^^^^^^^^^^^^^^^^^^^^^
      File "/home/doodlebear/miniconda3/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 160, in do_call
        relative_mod, func_name = args.func.rsplit(".", 1)
                                  ^^^^^^^^^
    AttributeError: 'Namespace' object has no attribute 'func'

`$ /home/doodlebear/miniconda3/bin/conda -v`

  environment variables:
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=openchat
                CONDA_EXE=/home/doodlebear/miniconda3/bin/conda
             CONDA_PREFIX=/home/doodlebear/miniconda3/envs/openchat
           CONDA_PREFIX_1=/home/doodlebear/miniconda3
    CONDA_PROMPT_MODIFIER=(openchat)
         CONDA_PYTHON_EXE=/home/doodlebear/miniconda3/bin/python
               CONDA_ROOT=/home/doodlebear/miniconda3
              CONDA_SHLVL=2
           CURL_CA_BUNDLE=<not set>
               LD_PRELOAD=<not set>
                     PATH=/home/doodlebear/miniconda3/envs/openchat/bin:/home/doodlebear/minicon
                          da3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/
                          bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Users/admin/Ap
                          pData/Local/fnm_multishells/2820_1699353565754:/mnt/c/ProgramData/anac
                          onda3:/mnt/c/ProgramData/anaconda3/Library/mingw-
                          w64/bin:/mnt/c/ProgramData/anaconda3/Library/usr/bin:/mnt/c/ProgramDat
                          a/anaconda3/Library/bin:/mnt/c/ProgramData/anaconda3/Scripts:/mnt/c/Pr
                          ogramData/anaconda3/bin:/mnt/c/ProgramData/anaconda3/condabin:/mnt/c/P
                          rogram Files/PowerShell/7:/mnt/c/Program Files/NVIDIA GPU Computing
                          Toolkit/CUDA/v11.8/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolk
                          it/CUDA/v11.8/libnvvp:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WI
                          NDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mn
                          t/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files (x86)/NVIDIA
                          Corporation/PhysX/Common:/mnt/c/Program Files/NVIDIA
                          Corporation/NVIDIA NvDLISR:/mnt/c/Program
                          Files/Bandizip:/mnt/c/Program
                          Files/Git/cmd:/mnt/d/code/Environment/cudnn/cudnn-windows-x86_64-
                          8.9.0.131_cuda12-
                          archive/bin:/mnt/c/ProgramData/anaconda3/Scripts:/mnt/d/code/Environme
                          nt/wget:/mnt/c/Program
                          Files/Docker/Docker/resources/bin:/mnt/c/Program
                          Files/PowerShell/7:/mnt/c/Program Files (x86)/Windows Kits/10/Windows
                          Performance Toolkit:/mnt/c/Program
                          Files/NVIDIA/CUDNN/v8.9.6/bin:/mnt/c/Program Files/NVIDIA
                          Corporation/Nsight Compute 2022.3.0:/mnt/c/Users/admin/AppData/Local/M
                          icrosoft/WindowsApps:/mnt/c/Users/admin/AppData/Local/Programs/Microso
                          ft VS Code/bin:/mnt/c/Users/admin/AppData/Local/Microsoft/WinGet/Packa
                          ges/Schniz.fnm_Microsoft.Winget.Source_8wekyb3d8bbwe:/mnt/c/Users/admi
                          n/AppData/Local/Programs/oh-my-
                          posh/bin:/mnt/c/texlive/2023/bin/windows:/snap/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : openchat
    active env location : /home/doodlebear/miniconda3/envs/openchat
            shell level : 2
       user config file : /home/doodlebear/.condarc
 populated config files :
          conda version : 23.9.0
    conda-build version : not installed
         python version : 3.11.5.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.3=0
                          __glibc=2.35=0
                          __linux=5.15.90.1=0
                          __unix=0=0
       base environment : /home/doodlebear/miniconda3  (writable)
      conda av data dir : /home/doodlebear/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/doodlebear/miniconda3/pkgs
                          /home/doodlebear/.conda/pkgs
       envs directories : /home/doodlebear/miniconda3/envs
                          /home/doodlebear/.conda/envs
               platform : linux-64
             user-agent : conda/23.9.0 requests/2.31.0 CPython/3.11.5 Linux/5.15.90.1-microsoft-standard-WSL2 ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Install OpenChat on Windows 11 (WSL)

Using the instructions a bit different from above

Same for create conda env

conda create -y --name openchat
conda activate openchat

⭐ Alomst same, specify the python version

// conda install -y python
conda install -y python=3.11

Same for install cudatoolkit in conda

• but want to mention that when I am running this instruction, it takes a while and only show the \ and / while processing and ended with nothing in my PowerShell (terminal), so I have to close the terminal and reopen it.

conda install -y cudatoolkit-dev -c conda-forge

⭐ Alomst same, but specify the torch version.

• I noticed that using conda to install torch will some how install cpu version instead of cuda using conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
• I saw @ugref using torch.__version__ = 2.1.0+cpu, and I did not facing the same issue when using 2.0.1+cu118 installing flash-attn when using Windows 11 (not in WSL)

// pip3 install torch torchvision torchaudio
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

• torch==2.1.0 may works well, did not try, check out other pytorch version here
  • pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
  • pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Same

pip3 install packaging ninja

⭐ Before install flash-attn, I need to Install g++ Compiler first. (Basically, GPT-4 teaches me everything)

sudo apt update
sudo apt install build-essential

Same

pip3 install --no-build-isolation "flash-attn<2"
pip3 install ochat

I try almost 6 hours on making it works on Windows but fail, So I try WSL 2
I already using Docker, so there is nothing too much when install openchat on WSL
Simply

1. Install Conda (maybe come across some env issue, but GPT-4 got it)
2. Little issue when install some python package, just ask GPT-4
3. Stuck on running the server for a while, if you are not using cluster with multiple GPUs, and like me using a Desktop RTX 3090.
   Instead of using

python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --engine-use-ray --worker-use-ray

Use this just works for me

python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --tensor-parallel-size 1

Use this to see more parameters settings

python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 -h

Test with request

[DoodleBears]

@imoneoi
I am wondering if a Run Server section right after Installation section will be better?

For example

Run Server

For single GPT (e.g. RTX 3090, 4090)

python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --tensor-parallel-size 1

For cluster with multiple GPUs

python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 --engine-use-ray --worker-use-ray

For more parameters settings

python -m ochat.serving.openai_api_server --model openchat/openchat_3.5 -h

Original README has run server instruction above, instead of right after the installation

[DoodleBears]

Just made a PR #70

[vpodpecan]

After a little bit of experimenting and getting errors related to cuda versions, here are up-to-date steps (for 3.6.1) that worked for me (conda 4.10.3):

conda create -y --name openchat3.6
conda activate openchat3.6
conda install -y python==3.11
conda install nvidia/label/cuda-12.1.1::cuda-toolkit
pip3 install torch

pip3 install packaging ninja
pip3 install --no-build-isolation "flash-attn<2"

pip3 install ochat==3.6.1

Finally:

python -m ochat.serving.openai_api_server --model openchat/openchat-3.6-8b-20240522