suggestions

[Hugs288]

1. add validation loss to the ui (Validation loss  kohya-ss/sd-scripts#1864) (currently doesnt work at all because extra args converts everything every arg to string)

2. combine sdpa/xformers, caching/to disk options to a single dropdown selector

3. on networks tab:
   dora can be its own network type
   wd on output should be enabled by default for dora and can be removed from ui
   bypass mode can be removed from ui (people who know they need it can use it via network args)
   cache te outputs seems useless to me, why not just train unet only?

4. improve default settings:
   gradiant checkpointing on by default
   if sdxl is checked set resolution to 1024 and max bucket res to 2048
   change default lr to 3e-4, te lr to 3e-5
   remove weight_decay=0.1 optional optimizer arg
   network dim/alpha = 16/16 or 16/8
   lr scheduler either rex with 1 cycle or constant with warmup
   10% warmup by default

[derrian-distro]

1.) Makes sense, though I personally dont use it
2.) Already on the to-do list
3.) Does every major platform support the fixed DoRA? If not, then I'd like to keep it as an arg for it. Definitely think I agree on DoRA just being its own network type, though. Cache TE outputs is absolutely necessary for models like sd3.5 and Flux.1 as they both command a gargantuan t5 xxl, and for most people, it'd be a foolish idea to let it be loaded while training.
4.) Grad checkpointing isn't strictly needed for SDXL, and pretty much not worth using at all on 1.5, of which these 2 models represent most of the userbase, so it doesn't really need to be enabled by default. At this point, I'd rather just default to SDXL as the model type because it's the more commonly trained model. I dont think changing the lrs really matters all that much because each person believes their own black magic anyway. I dont know how I feel about removing the weight decay arg. It serves as a tutorial as much as it is also still a good choice to have to prevent people from charring their lora. So, people have actually decided that smaller dims are better? Cool, I hadn't changed that for a long time because I got major backlash the first time I tried. I will agree on changing the default to rex. It is just largely a better scheduler. Warmup is largely subjective, so sure, any arbitrary low number (that isn't 0) should work fine

[Hugs288]

reforge, forge and comfy all support fixed dora so i think its fine

the defaults are mostly subjective obviously but i think its an improvement over the current defaults, alternatively you can also put some preset tomls somewhere, i think currently its too hard for noobs to get into training and the shitty defaults are not helping.

[derrian-distro]

Good to know, I figured the support was added but I hadn't checked if reforge/forge fixed it, it appears that the newest version of lycoris sets the wd_on_output enabled by default now, so I'll just remove the arg when I get around to it.

I generally dont like providing toml examples due to the subjective nature of things, but I can probably make up a few examples with commonly used args while I'm making other changes.

Since I've never once used any of the adaptive optimizers, They will likely all be based on using either Adamw, or Came, of which Came is the optimizer I typically use personally

[Hugs288]

prodigy and dadaptadam suck anyways imo

[gammahurricane]

prodigy and dadaptadam suck anyways imo

You shouldn’t underestimate it — Prodigy does an excellent job training LoRAs even with small datasets

[derrian-distro]

I personally find that prodigy either burns the lora, or doesn't train it enough, so I largely stay away from them, that being said, everybody is gonna have their own opinions on them so its largely a moot point in the first place

[cobalt-a11y]

Suggestion: add a button that opens tensorboard in a web browser.

OneTrainer and Kohya both have a button that does this.

Can we please have one too?