Creating a Custom ControlNet

[graveyard99]

Hi all

I’m hoping to create a custom controlnet based on segmentation data from the Fashionpedia dataset - https://fashionpedia.github.io/home/Fashionpedia_download.html

I think this could be incredibly powerful at helping to render people realistically as it could enable clothing elements from input images, or 3D render multipasses to be effectively isolated and rendered correctly rather than treated as a whole, where shirts merge with hands, or shoes merging with trousers etc.

I’m using A1111 at the moment and have been experimenting with training hypernetworks and creating various scripts to automate processes which has been a lot of fun, but that’s where my advanced knowledge cuts off. I’m intending to move to ComfyUI next.

I’ve read a few papers on how controlnet works, and seen a few posts about training controlnet, but I haven’t seen a decent step-by-step tutorial. As far as I can tell, training a controlnet through the A1111 or ComfyUI webUIs are not options...?

Would anyone be willing to take me through this?

[trendland]

Any news on this? Been inpainting and creating loras left and right for clients, but have not yet found an easy way to bring clothing into gens. Would be curious if you found and easier way ?

[DiffusionDalmation]

I'm curious about this too. I haven't been able to find any guides that are more comprehensive than the documentation in this repo. https://github.com/lllyasviel/ControlNet/blob/main/docs/train.md

I don't have the hardware to run the training, so I haven't tried it myself.Maybe you could give it a try and then ask a more specific question when you run into problems?

[graveyard99]

Hey

so yeah I actually asked Pedro Cuenca himself on LinkedIn about helping out with this. He was very polite but unsurprisingly busy and redirected me here, which I’m sure you have already seen:

https://github.com/huggingface/diffusers/tree/main/examples/controlnet

what I think we need is a YouTube tutorial, or even better, a trainer actually built into A1111! Imagine the power of being able to just load in a bunch of segmentation pairs into a CN trainer just like using Hypernetworks or Embeddings. I think it’s worth pushing for this.

So overall, I still don’t know the process. I’m not a coder or programmer at all so my knowledge reaches its limits pretty prematurely

[graveyard99]

I made a feature request

AUTOMATIC1111/stable-diffusion-webui#11353 (comment)

[huchenlei]

It probably won't be a useful feature for all A1111 user. I would suggest to build it as a separate extension.

[graveyard99]

Do you really think that? Im surprised. The ability to create bespoke systems to control an image seems full of potential, especially if it were built into the interface with the same ease as the other training modules.

Hopefully someone with the skills and know how sees this and gives making an extension a go 💪

[geroldmeisinger]

I wrote an extensive article about controlnet training here: https://civitai.com/articles/2078
it's still not an UI based solution but it's not difficult either (imo)