Any updates on getting Cosine with annealing?

[DarkAlchy]

I asked for this from Kohya a while back, and it is not cosine with restart, only now the new Prodigy wants that to be at its best (per the dev on the github site).

Any chance?

https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html

[Isotr0py]

You can use --lr_scheduler_type and --lr_scheduler_args to set custom scheduler.

• More details in Add '--lr_scheduler_type' and '--lr_scheduler_args' argument #271

[DarkAlchy]

You can use --lr_scheduler_type and --lr_scheduler_args to set custom scheduler.

• More details in Add '--lr_scheduler_type' and '--lr_scheduler_args' argument #271

WOW, that rocks if it is really that easy.

Thank you.

[DarkAlchy]

That isn't in sd-scripts though, or if it is I didn't find it, BUT is in Kohya_ss. I need it in sd-scripts.

[dill-shower]

+1

[Isotr0py]

@DarkAlchy Just pass --lr_scheduler_type and --lr_scheduler_args to command line.

Most of schedulers in torch.optim.lr_scheduler can be imported directly.

For example:

accelerate launch "train_network.py" ... --lr_scheduler_type "CosineAnnealingLR" --lr_scheduler_args "T_max=100"

[DarkAlchy]

Well, I use .toml and I wonder how sd-scripts wants those in toml format? Besides, as I said, in Kohya_ss I found it in a couple of files while in sd-scripts I found no references to it.

[Isotr0py]

It seems that .toml works similarly with command line args:

[general]
lr_scheduler_type = "CosineAnnealingLR"
lr_scheduler_args = "T_max=100"

I don't know how Kohya_ss gui implemented this scheduler since I seldom used it.

[DarkAlchy]

I do not know myself but I can tell you a friend worked with this over 8h with me on his 3090 and on 2.1 we nailed down some good settings that floored us. One thing I will say is that people need to look at the specs for annealing because T_max at 100 is flat out wrong AND don't use any defaults. We tried 100, and the defaults and after many hours of log tracing, and testing we came to this conclusion. If anyone is training on 2.1 go to Pytorch.org and read up on the ALL the settings and you will see.

For me this sucks in one department and that is at 15GB on Colab I am forced to turn on checkpointing (else I can't even get BS1) to use what Annealing works fantastic with called Prodigy. For Adam anything is flat out sucked. I know why but only so many hours in a day. Btw, with Prodigy BS1 and checkpointing it takes 13.7-13.8GB of vram.

Thank you for all this help, but I must admit I found it odd that doing a scan all files I found nothing in sd-scripts and a few hits with Kohya_ss.

[DarkAlchy]

@Isotr0py

It seems that .toml works similarly with command line args:

[general]
lr_scheduler_type = "CosineAnnealingLR"
lr_scheduler_args = "T_max=100"

I don't know how Kohya_ss gui implemented this scheduler since I seldom used it.

I never did manage to get this to work as it says

I inserted a print for that args.blahblah and it showed T_max=100, or other things I tried too yet they always result in that error. Are we sure sd-scripts really does work with these?

[Isotr0py]

@DarkAlchy Oh, my fault.

It seems that args.blahblah just passed str instead of a list through .toml.

We should pass a list through .toml:

lr_scheduler_args = ['T_max=100', ...]

[DarkAlchy]

@Isotr0py That was the first way that I tried and it had the same issue. I tried it as that is how we pass the other args (such as weight etc...). Some reason it didn't grab it.

Here, this did not work and as you can see it is in the same syntax as network_args:

network_args = [ "conv_dim=128", "conv_alpha=128", "algo=lora",]
network_train_unet_only = false
network_train_text_encoder_only = false

[optimizer_arguments]
min_snr_gamma = 1
optimizer_type = "Prodigy"
learning_rate = 1
max_grad_norm = 1.0
optimizer_args = [ "eps=1e-6",]
lr_scheduler = "CosineAnnealingLR"
lr_scheduler_args = [ "T_max=66",]
lr_warmup_steps = 0

network_args works. optimizer_args works. lr_scheduler_args does not work with that 2 expect 1 error.

[Isotr0py]

I tested the below config on Colab and it worked normally:

[model_arguments]
v2 = false
v_parameterization = false
pretrained_model_name_or_path = "runwayml/stable-diffusion-v1-5"

[additional_network_arguments]
no_metadata = false
unet_lr = 0.0001
text_encoder_lr = 5e-5
network_module = "networks.lora"
network_dim = 64
network_alpha = 32
network_train_unet_only = false
network_train_text_encoder_only = false

[network_arguments]
network_args = [ "conv_dim=128", "conv_alpha=128", "algo=lora",]
network_train_unet_only = false
network_train_text_encoder_only = false

[optimizer_arguments]
optimizer_type = "AdamW8bit"
learning_rate = 0.0001
max_grad_norm = 1.0
optimizer_args = [ "weight_decay=0.6",]
lr_scheduler_type = "CosineAnnealingLR"
lr_scheduler_args = [ "T_max=66",]
lr_warmup_steps = 0

[dataset_arguments]
cache_latents = true
debug_dataset = false
enable_bucket = true
train_data_dir="/content/gdrive/MyDrive/Lora-train"

[training_arguments]
resolution = 512
output_dir = "/content/"
output_name = "test"
save_precision = "fp16"
save_every_n_epochs = 1
train_batch_size = 6
max_token_length = 225
mem_eff_attn = false
xformers = true
max_train_epochs = 20
max_data_loader_n_workers = 8
persistent_data_loader_workers = true
gradient_checkpointing = false
gradient_accumulation_steps = 1
mixed_precision = "fp16"
clip_skip = 2
logging_dir = "/content/LoRA/logs"
log_prefix = "test"
noise_offset = 0.1
lowram = true

It's strange that the value error was raised...