ControlNet extension?

[pj4533]

My python skills are just not strong enough...anyone have any idea how (or if) ControlNet could be integrated into the CoreML conversion process?

https://github.com/lllyasviel/ControlNet#controlnet-with-canny-edge

[gavtron2000]

ControlNet looks pretty compelling. I've only just got image to image up and running in my alpha app, and I'm not a python programmer, but if a few people are willing to take on the challenge I would be cheering them on!

[pj4533]

Agreed. Way more confident in my swift skills, but that said, I could probably stumble through python enough to help out, just not even sure where to start for something like ControlNET.

[alelordelo]

+1!

[twinsant]

+1

[crazyshitass]

+1

[korzen]

+1

[GuiyeC]

I was able to get ControlNet working with CoreML on my app Guernika so this is definitely possible, so far I have added preprocessing for poses, depth and HED maps but I have successfully converted all eight existing models.

This does require modifying Unet's inputs so all existing models will not be compatible and will need to be reconverted.

[gavtron2000]

Great work @GuiyeC ! I just dowloaded and had a play. You've done an excellent job on the UI too.
I'd encourage you to submit a pull request to add control net support to the repo if you are willing.

[pj4533]

Wow! @GuiyeC I commented on the other thread before i saw this. Looks amazing. Seems like you modified the underlying UNET via python, then used that in the conversion process of other models? +1 on a merge if you have time!

[GuiyeC]

@gavtron2000 @pj4533 thank you for your comments!

I have the python scripts in my Guernika repo, you can see how the Unet and ControlNet are implemented for conversion, probably not the cleanest way of passing residuals to the Unet but that's how I managed to do it.

As for the PR, I don't think I'll ever make a PR to this repo as my repo is quite different already, for example, I implemented img2img first and the implementation is different, if I'm not wrong random latents on this repo are generated in python and I do that in Swift, my encoder is just the encoder module. I also implemented instruction pix2pix support, ControlNet and other tweaks here and there, so it would practically mean taking over this repo. At most I would make my fork public but I would like to have some time to give my app an edge and get Guernika going.

[pj4533]

☹️

[gavtron2000]

Totally understandable @GuiyeC. Really impressive work, and a really excellent app. I wish you every success.

[Zabriskije]

Good job @GuiyeC! Do you have a Discord server? I have some questions/feedback

[GuiyeC]

@Zabriskije you can contact me on Twitter or Telegram, same name, or you can leave them in Hugging Face or in Reddit :)