add controlnet training #551
Conversation
library/train_util.py
Outdated
| @@ -1787,6 +2103,8 @@ def backward(ctx, do): | |||
|
|
|||
| def replace_unet_modules(unet: diffusers.models.unet_2d_condition.UNet2DConditionModel, mem_eff_attn, xformers): | |||
| # unet is not used currently, but it is here for future use | |||
| unet.enable_xformers_memory_efficient_attention() | |||
| return | |||
|
Thank you for this! I think this is what everyone wanted! However, I have two concerns. I am aware that this pull request requires Diffusers to be upgraded to the latest version. Am I correct? The Diffusers version upgrade will have a large impact, and it will be necessary to fix them as well. Also, there is an original implementation of ControlNet in the repository: Since I think that Diffusers will need to be upgraded sooner or later, I will consider fixing the other modifications after the pull request is merged. As for ControlNet, I think the implementation in this repository is simpler than the one in Diffusers, so I think it would be an idea to use this implementation. However, if we upgrade Diffusers, we could switch the generation script from own ControlNet implementation to the Diffusers implementation. Therefore, I think it would be best if you implement it in a way that is easy to implement. |
|
Can we use original ControlNet repository's preprocesser to generate |
|
If you plan to upgrade the version of diffusers, I'll use ControlNet implemented with diffusers as it is. |
|
https://github.com/ddPn08/sd-scripts/tree/diffusers_0_16 |
|
I will make it open once I write the documentation. |
|
Cool! Thanks! I am considering implementing U-Net proprietary as I mentioned before. That way the repository would not be dependent on a specific version of Diffusers. I am thinking that it might be better to do that first. So the point at which merges become available may be a little later. I will also consider creating a new branch to work on that and merging this PR into it. |
|
I'm almost done preparing for the Diffusers upgrade in the original-u-net branch, so I'll be able to merge this PR. However, some modifications are duplicated in this PR and the branch, and due to code reorganization related to datasets, please note that I will modify the code after the PR is merged. |
|
I got it. I open this PR. |
|
Thank you! I will merge this when I have time! |
Merge history from kohya-ss/sd-scripts
Add ControlNet training code.
As soon as all the tests are finished, I will write the details and open it.