Support pre-training 8*V100 (32G) gpus with xformers

[guanlaoda]

FlashAttention does not support V100 GPU, and llava cannot train on V100, but most of the cards now are still V100. Now we introduce the FastChat solution to llava, and xFormers replaces FlashAttention. Like FastChat（https://github.com/lm-sys/FastChat）, you can now use LLaVa to train on V100.

[guanlaoda]

llava-7B-v1.1 model

[haotian-liu]

Thank you for your contribution! It looks nice to me, and thank you for providing the training logs as well. One minor: can you remove xformers from pyproject.toml and add a short instruction about how to install xformers? Similar to the one in FastChat:

If you are using V100 which is not supported by FlashAttention, you can use the memory-efficient attention implemented in xFormers. Install xformers and replace fastchat/train/train_mem.py above with fastchat/train/train_xformers.py.

[guanlaoda]

OK, I'll modify it as soon as possible and submit it.

[guanlaoda]

Completed the modification and submitted

[guanlaoda]

[guanlaoda]

no response，Is there anything else that needs to be modified?

[tingxueronghua]

no response，Is there anything else that needs to be modified?

Thanks for your effort and I want to inquire about the GPU assumption. After applying xformers like FastChat, could LLaVA 13B be fine-tuned on 8 V100?

[guanlaoda]

I have tested it and GPU OOM will occur. I'll test it tomorrow and see if I can adjust the parameters.

[guanlaoda]

Test results on 8*V100 (32G) gpus with xformers: pre-training works on 7b, but fine-tuned does not work on 7b, causing OOM.
Do I need to try to resolve this issue?

[tingxueronghua]

Test results on 8*V100 (32G) gpus with xformers: pre-training works on 7b, but fine-tuned does not work on 7b, causing OOM. Do I need to try to resolve this issue?

Thanks! I think I will just use Lora. Thanks for your effort and patience!

[haotian-liu]

Sorry for the late response, and thank you for the modification. I will review and merge this week.

[hojisu]

is there any way to pretrain on 4*v100 (32gb)?

[aileenliao03]

Test results on 8*V100 (32G) gpus with xformers: pre-training works on 7b, but fine-tuned does not work on 7b, causing OOM. Do I need to try to resolve this issue?

Hi, is there a way to fine tune 7b on 8*V100s now by any chance? Thanks!

[pritamqu]

could you please confirm which version of xformers did you use?

[zfr00]

So there's no way to do full-parameter fine-tuning with v100, right?