v21.5.7

[bmaltais]

• 2023/05/01 (v21.5.7)
  • tag_images_by_wd14_tagger.py can now get arguments from outside. PR #453 Thanks to mio2333!
  • Added --save_every_n_steps option to each training script. The model is saved every specified steps.
    • --save_last_n_steps option can be used to save only the specified number of models (old models will be deleted).
    • If you specify the --save_state option, the state will also be saved at the same time. You can specify the number of steps to keep the state with the --save_last_n_steps_state option (the same value as --save_last_n_steps is used if omitted).
    • You can use the epoch-based model saving and state saving options together.
    • Not tested in multi-GPU environment. Please report any bugs.
  • --cache_latents_to_disk option automatically enables --cache_latents option when specified. #438
  • Fixed a bug in gen_img_diffusers.py where latents upscaler would fail with a batch size of 2 or more.
  • Fix triton error
  • Fix issue with merge lora path with spaces
  • Added support for logging to wandb. Please refer to PR Failing to start Lora training - Exit status 1 #428. Thank you p1atdev!
    • wandb installation is required. Please install it with pip install wandb. Login to wandb with wandb login command, or set --wandb_api_key option for automatic login.
    • Please let me know if you find any bugs as the test is not complete.
  • You can automatically login to wandb by setting the --wandb_api_key option. Please be careful with the handling of API Key. PR Possible solution for loss=nan in 16xx series cards. #435 Thank you Linaqruf!
  • Improved the behavior of --debug_dataset on non-Windows environments. PR ModuleNotFoundError: No module named 'library.lpw_stable_diffusion' & raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError #429 Thank you tsukimiya!
  • Fixed --face_crop_aug option not working in Fine tuning method.
  • Prepared code to use any upscaler in gen_img_diffusers.py.
  • Fixed to log to TensorBoard when --logging_dir is specified and --log_with is not specified.
  • Add new docker image solution.. Thanks to @Trojaner