[Question] Training with 4 RTX 4090

[RajeshRadha]

Question Training with 4 RTX 4090

Can I train and fine tune LlaVa model w/ model config as 13B Llama model + vision encoder (default) + connector module with workstation that has 4 RTX 4090?

[YerongLi]

Did you succeed in training with 4 x RTX 4090? I tried A10 GPU with 24 GB and received OOM.

[JoeLeelyf]

I'm also working on training with 8 RTX 3090 with 24GB each these days, but also received OOM. I have tried to use deepspeed to solve the problem, getting about 6GB mem usage on each of my GPU when finishing loading the model and data, but received "RuntimeError: is_sm80 || is_sm90 to be true, but got false". It seems that flash-attention doesn't support 3090/4090 yet. I also tried to use run train.py instead of train_mem.py to avoid using flash-attein, but got OOM in torchrun script and "AttributeError: 'DummyScheduler' object has no attribute 'step'" when using deepspeed.

[haotian-liu]

@RajeshRadha @YerongLi You can use zero3_offload.json to further save gpu ram by offloading parameters / optimizer states to CPU.

@JoeLeelyf you may try reinstalling flash-attn by pip install flash-attn --no-build-isolation. It works for my 3090s.

[RajeshRadha]

@haotian-liu Thanks for the suggestion, How much CPU RAM is expected during offloading and do you see any performance degradation during training?

[haotian-liu]

@RajeshRadha

I think you can view CPU RAM as an extension of GPU VRAM, the same amount offloaded from GPU will be on CPU RAM. You can benchmark the exact ratio.

There will be no degradation in terms of accuracy, but definitely slower than having all parameters/states on GPU. Degradation in training speed varies case by case, and you can benchmark on your workstation.

[MiroFurtado]

How much CPU ram do you typically require? I am getting OOM before anything is even transferred to GPU but I have very weird machine with 100 GB GPU RAM & only 100 GB CPU RAM

[JoeLeelyf]

@haotian-liu Thanks for the suggestion. I have successfully reproduced LLaVA on 4*GTX 3090. I found that the real problem lies in zero3_offload.json, which lacks the configuration of "scheduler". After assigning it to "type":"WarmupLR", the program runs successfully.

"scheduler": {
"type": "WarmupLR",
"params": {
"warmup_min_lr": "auto",
"warmup_max_lr": "auto",
"warmup_num_steps": "auto"
}
},

[MiroFurtado]

@haotian-liu Thanks for the suggestion. I have successfully reproduced LLaVA on 4*GTX 3090. I found that the real problem lies in zero3_offload.json, which lacks the configuration of "scheduler". After assigning it to "type":"WarmupLR", the program runs successfully.

"scheduler": { "type": "WarmupLR", "params": { "warmup_min_lr": "auto", "warmup_max_lr": "auto", "warmup_num_steps": "auto" } },

Cheers - that was also necessary to make it work for me.

[haotian-liu]

@JoeLeelyf Thanks for the info, I just updated the scripts in the repo :)

[haotian-liu]

How much CPU ram do you typically require? I am getting OOM before anything is even transferred to GPU but I have very weird machine with 100 GB GPU RAM & only 100 GB CPU RAM

Usually zero3 will use fewer CPU RAM when loading.

[MiroFurtado]

Unfortunately, with 193 GB RAM and zero3_offload, while the pre-training works, I cannot find enough RAM to get the finetune.sh working.

[haotian-liu]

@MiroFurtado Are you trying to finetune a 7B or 13B model? And are you trying to do full finetuning or lora tuning? Maybe you can try LoRA tuning with a smaller model.