pip install doesn't work on Nvidia RTX 2070 Super on Ubuntu 20.04.3 LTS

[bilawalriaz]

NVIDIA-SMI 525.147.05
Driver Version: 525.147.05
CUDA Version: 12.0 (also tested with 12.2)

Fresh install of Ubuntu
Using virtualenv to create a "packages" folder then source packages/bin/activate
Then "pip install -r requirements.txt"

Using this command: python fam/llm/sample.py --huggingface_repo_id="metavoiceio/metavoice-1B-v0.1" --spk_cond_path="assets/ava.flac"

I get this:

/home/user/metavoice-src/packages/lib/python3.10/site-packages/df/io.py:9: UserWarning: torchaudio.backend.common.AudioMetaDatahas been moved totorchaudio.AudioMetaData. Please update the import path. from torchaudio.backend.common import AudioMetaData Traceback (most recent call last): File "/home/user/metavoice-src/fam/llm/sample.py", line 22, in <module> from fam.llm.model import GPT, GPTConfig File "/home/user/metavoice-src/fam/llm/model.py", line 12, in <module> from fam.llm.layers import Block, LayerNorm, RMSNorm File "/home/user/metavoice-src/fam/llm/layers/__init__.py", line 1, in <module> from fam.llm.layers.attn import SelfAttention File "/home/user/metavoice-src/fam/llm/layers/attn.py", line 3, in <module> from flash_attn import ( # type: ignore File "/home/user/metavoice-src/packages/lib/python3.10/site-packages/flash_attn/__init__.py", line 3, in <module> from flash_attn.flash_attn_interface import ( File "/home/user/metavoice-src/packages/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 10, in <module> import flash_attn_2_cuda as flash_attn_cuda ImportError: /home/user/metavoice-src/packages/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15sum_IntList_out4callERKNS_6TensorEN3c1016OptionalArrayRefIlEEbSt8optionalINS5_10ScalarTypeEERS2_

This occurs even when trying a new virtualenv. Is there a specific Python/CUDA version i should be on?

[vatsalaggarwal]

hey, this is likely related to flash attention 2 not being supported on older NVIDIA GPUs. There is implementation in the code for using hand-coded kv-caching + vanilla torch attention already, which should work on 2090, but these haven't "hooked up" yet

[samikama]

try reinstalling flash-attention pip install -U --force-reinstall flash-attn

[bilawalriaz]

try reinstalling flash-attention pip install -U --force-reinstall flash-attn

I tried that but it didn't work

hey, this is likely related to flash attention 2 not being supported on older NVIDIA GPUs. There is implementation in the code for using hand-coded kv-caching + vanilla torch attention already, which should work on 2090, but these haven't "hooked up" yet

That makes sense - i feel like i've seen a list of supported GPUs somewhere that mentioned RTX 3 series and above. That vanilla torch stuff should be compatible with Macs right? If so, i'll probably just end up installing it there instead.

[vatsalaggarwal]

hey @bilawalriaz can you try now? Vanilla torch should be compatible with Mac, but we've got some other difficulties there, so would recommend trying again on the 2090 with #46... Note: the models require 20GB of VRAM, so you might have to do some offloading/onloading onto CPU RAM to make this work with the 8GB 2090...

[SinclairSchneider]

It might help to go to the previous version of flash-attn. Like pip install flash-attn==2.5.2

[sidroopdaska]

@bilawalriaz, checkout PR #62
You no longer need to install flash-attn for Turing arch