Support pytorch_optimizer

[gesen2egee]

https://github.com/kozistr/pytorch_optimizer

A very practical optimizer library that includes many newer optimizer (or learning rate scheduling) options.

For example, to use CAME:

--optimizer_type "pytorch_optimizer.CAME" --optimizer_args "weight_decay=0.01"

To use Lion with cautious mode:

--optimizer_type "pytorch_optimizer.Lion" --optimizer_args "weight_decay=0.01" "cautious=True"

You can check pytorch_optimizer to see what other options are available.

The author kozistr is very active. If you have an optimizer you’d like to try,
you can provide a paper link or code and open a issue—he implements things quickly!

[6DammK9]

FYI: This had covered #1409 already. Meanwhile this gives a lot more optimizers to explore other than bitsandbytes.
If you experienced OOM on FFT (Full FineTuning), this may help (e.g. adafactor / LION / CAME / mini adam etc.)

    parser.add_argument(
        "--optimizer_type",
        type=str,
        default="",
        help="Optimizer to use / オプティマイザの種類: AdamW (default), AdamW8bit, PagedAdamW, PagedAdamW8bit, PagedAdamW32bit, "
        "Lion8bit, PagedLion8bit, Lion, SGDNesterov, SGDNesterov8bit, "
        "DAdaptation(DAdaptAdamPreprint), DAdaptAdaGrad, DAdaptAdam, DAdaptAdan, DAdaptAdanIP, DAdaptLion, DAdaptSGD, "
        "AdaFactor. "
        "Also, you can use any optimizer by specifying the full path to the class, like 'bitsandbytes.optim.AdEMAMix8bit' or 'bitsandbytes.optim.PagedAdEMAMix8bit'.",
    )

[gesen2egee]

Recommend adding missed prodigy-plus-schedule-free and updating prodigy to the latest version.

[kohya-ss]

Thank you for this PR! The latest update is nice, because I think specifying a specific version is preferable. I will merge this today.

[kohya-ss]

Sorry, I didn't notice, but is the latest version of pytorch-optimizer okay?

[gesen2egee]

okay
But this is a rapidly updating library.🤣

[kozistr]

Hi! I stumbled upon this PR by chance. Thanks for opening the PR and introducing pytorch-optimizer to this repo, @gesen2egee @kohya-ss.

IMO, it might be great to fix the version to the latest release (3.5.0) for reproducibility, consistency, and to prevent unexpected issues or bugs.

I usually update the PATCH version (z in the x.y.z scheme) every couple of weeks to a month and update the minor version (y) every couple of months. Maybe, we could update the minor (or patch) version here when needed.

For your reference, In PATCH updates usually include bug fixes or adding a new optimizer, and I update the minor version when there are enough new features and each new feature looks okay!

feel free to leave a comment if you need anything :)

[kohya-ss]

Thank you for your comment! I will update to specify 3.5.0 after merging.

We will make appropriate changes as new versions are released!