Latest version Adafactor cannot be used with `num_sanity_val_steps>0`

[Daraan]

Using newer versions of lightning, transformers breaks Adafactor when checking the validation data loader for sanity first (trainer.num_sanity_val_steps > 0), as it dictionary is not initialized first in optimizer.run_step

• determine which package/version/change causes it

Traceback and related (but not solved) Issue kohya-ss/sd-scripts#342

---

• Note: This does not affect the currect requirements, however updating them might be tricky!

[Daraan]

Maybe this is just related to fast_dev_run=True