Training with 8GB VRAM

[Doev]

Hello,

in some Blog read, that the training of a own embedding is possible with 8GB VRAM. When I try it, I get on OOM-Error.

Can someone confirm, that it basically works with 8GB VRAM?

[Hagasus]

Hello,
this configuration works with me. Use the --medvram parameter, then it should work.😉

[Konvoname]

thank you.

I still searching where to put the parameter. Using it at command line is not working.

It happen sometime try restarting program

[Doev]

I found it in webui-user.sh:

.# Commandline arguments for webui.py, for example: export COMMANDLINE_ARGS="--medvram --opt-split-attention"
export COMMANDLINE_ARGS="--listen"

[Hagasus]

exactly there. 😉

[Doev]

Now I run it on a headless system and there are additional 600MB VRAM free.

Do you know some demo training set, so I can quick test if training now works?

[OrenjiVR]

Hello, this configuration works with me. Use the --medvram parameter, then it should work.😉

Currently (at least as of yesterday) there seems to be a strange issue, where when you want to train hypernetworks with --medvram enabled, the training will not progress. If you train this way for a couple of thousand steps and then put the different milestones (like every 1000 steps) into a grid, to check which one is the best, they will not differ.

[Konvoname]

1070 ti 8 GB VRAM work perfectly fine for me

[Doev]

Finally training seems to run well. For some reason the GPU runs only at 30% but I guess this is the price of the low memory version.

[GiuseppePerugia]

Hi, my training seems not to work on a 1070. I'm using the following parameters: python main.py --lowvram --opt-split-attention --base configs/stable-diffusion/v1-finetune_unfrozen.yaml -t --actual_resume models/ldm/Stable-diffusion/model.ckpt -n DMA-Giulia --gpus 0, --data_root training/GiuliaDIMA --reg_data_root classes/class-face-samples --logdir G:\log --class_word GiuliaDIMA --max_steps 1001 --seed 56

Any chance that I may ask yours?

[Doev]

Hi! First of all, I would like to mention that I still have no understanding of the two concepts of embeddings and hypernetworks.

I'm happy to share my parameters with you, but I don't know how to start the training process from the command line. So far I only use the web UI.

[OrenjiVR]

I run a 3070 with 8GB and I'm bordering on being out of memory. But it is possible, even without --medvram (which seems to cause problems with training hypernetworks).

Should you get OOM warnings, try to lower your learning rate, or lower the preview width/height.
I for example train 704x704 right now, but if I would try to generate the preview at the same size, I would get an OOM error. lowering it down to 448x448 helped me in that regard.
It also helps to restart webui or the whole PC before starting to train.