Add Multi-ControlNet pipeline

[takuma104]

Discussed in #2331. I created a PoC that supports multiple ControlNets, called Multi-ControlNet, based on the StableDiffusionControlNetPipeline. Any feedback is appreciated.
https://github.com/takuma104/diffusers/tree/multi_controlnet

The idea of using multiple ControlNets and adding their outputs together was proposed in #2331, but at the time, the name Multi-ControlNet was not common. For other implementations, I'm referring to @Mikubill 's pioneering work in this field, sd-webui-controlnet.

Currently, I'm considering opening a PR as a community pipeline and have placed the files in example/community.

The difference from pipeline_stable_diffusion_controlnet.py is as follows.
takuma104/diffusers@1b0f135...multi_controlnet

Modification points:

• Created a new ControlNetProcessor class and made it so that one is specified for each ControlNet processing. Image preprocessing was also moved here.
• Made it so that controlnet is not specified in the Pipeline constructor.
• Made it possible to specify multiple ControlNetProcessors in pipeline's __call__() method (there is no limit to the number).

Usage Example:

• Please refer to the main() function in the py file for details.

pipe = StableDiffusionMultiControlNetPipeline.from_pretrained(
	"runwayml/stable-diffusion-v1-5", safety_checker=None, torch_dtype=torch.float16
).to("cuda")
pipe.enable_xformers_memory_efficient_attention()

controlnet_canny = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16).to("cuda")
controlnet_pose = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16).to("cuda")

canny_left = load_image("https://huggingface.co/takuma104/controlnet_dev/resolve/main/vermeer_left.png")
pose_right = load_image("https://huggingface.co/takuma104/controlnet_dev/resolve/main/pose_right.png")

image = pipe(
        prompt="best quality, extremely detailed",
        negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
        processors=[
            ControlNetProcessor(controlnet_canny, canny_left),
            ControlNetProcessor(controlnet_pose, pose_right),
        ],
        generator=torch.Generator(device="cpu").manual_seed(0),
        num_inference_steps=30,
        width=512,
        height=512,
).images[0]
image.save("canny_left_right.png")

Generate Example:

• Model: andite/anything-v4.0
• Prompt: best quality, extremely detailed, cowboy shot
• Negative_prompt: cowboy, monochrome, lowres, bad anatomy, worst quality, low quality
• Seed: 19 (cherry-picked)
• Pose & Canny Control Image Generated by Character bones that look like Openpose for blender _ Ver_4.7 Depth+Canny

Control Image1	Control Image2	Generated
	(none)
	(none)

[mikegarts]

Actually this one looks even better than the StableDiffusionControlNetPipeline you recently merged, imho.

[becausecurious]

Is ControlNetProcessor.conditioning_scale same as ControlNet weight in AUTOMATIC1111 implementation of Multi-ControlNet?

[takuma104]

@mikegarts At the very end of the PR, there was a major API change. ControlNet files became an independent distribution rather than being distributed with Stable Diffusion pipeline files. I wish I had thought of this when I saw it, but unfortunately, I didn't come up with this idea at the time.

@becausecurious conditioning_scale is also in the arguments of StableDiffusionControlNetPipeline's __call()__ method, and I just ported it over. It probably works the same way as A1111 (more precisely, the Mikubill extension).

[takuma104]

I replaced the generated samples with more meaningful controls.
Code: https://huggingface.co/takuma104/controlnet_dev/blob/main/multi_controlnet/README.md

[patrickvonplaten]

Hey @takuma104,

Thanks a lot for the write-up - that's great!
Generally, I think the design is nice, it's just that we cannot rename things anymore because it would break backwards compatibility, so we have to stick with controlnet and image.

But your overall design looks very nice, I think we should allow the user to just pass a list of controlnets, i.e. changing:

controlnet: Union[ControlNetModel, List[ControlNetModel]]

at init. (The only problem is that this pipeline could then not be loaded saved at the moment, but this is not very important and can be changed down the road.)

Then we can just pass a list of images:

image = pipe(
        prompt="best quality, extremely detailed",
        negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
        image=[canny_left, pose_right],
        generator=torch.Generator(device="cpu").manual_seed(0),
        num_inference_steps=30,
        width=512,
        height=512,
).images[0]

instead of passing the whole model to pipe. I'm not a big fan of passing the whole model to the pipeline's call function
as it makes everything hard to debug, read (we're then essentially passing a function to a function which is not very pytorch-y)

Given that our image processors can easily process a list of images, I don't think we have to change any code really.

cc @williamberman could you also maybe help here?

[takuma104]

Hi @patrickvonplaten , Thanks for your feedback! I hadn't thought of your approach. Indeed, with this approach, we can extend StableDiffusionControlNetPipeline to Multi-ControlNet without breaking its compatibility. It's better than adding another community pipeline for no reason.

I think it should be fine for most use cases. If there is a weakness, it might be the following:

• conditioning_scale: also needs to support arrays (The default is 1.0 for multiple ControlNets in common, but I would like to specify them separately as well.).
• If we add more parameters in the future, we will need to support arrays as well, and it may become difficult for users to handle.

Regarding parameters: Since I was originally thinking about it in terms of the community pipeline, I also considered adding some more parameters supported by the Mikubill extension. I haven't written the implementation for those two yet, and I don't fully understand how effective they are. So, just for your information.

• Guess Mode: a mode that allows to skip the input of a prompt.
• Guidance Start & End: specify the timestep range to perform the ControlNetModel inference.

First, I'll try to create a new diff for this approach.

[takuma104]

I tried it a bit, but allowing a list of modules as an argument for DiffusionPipeline.register_modules() seems like it would be quite a wide-ranging change. I don't think it's a very good idea. Instead, should we make it so that it's okay to not register the module with register_module()?

[takuma104]

I created a proposal for making StableDiffusionControlNetPipeline compatible with Multi-ControlNet.
main...takuma104:diffusers:multi-controlnet-ext

Adding multiple controlnets on init() was very frustrating because I had to use a very hacky method. It feels like there might still be some pitfalls.

The fast-tests for the previous version and the additional tests for Multi-ControlNet passed. However, some of the PipelineTesterMixin tests for the additional tests did not pass, so they need to be rewritten. The generated test is not yet sufficient, but the results of the generation are consistent with those created previously.

To improve the extensibility of the conditions, I prepared a class called ControlNetCondition for parameters (reusing the remains of the ControlNetProcessor). The advantage is that you can specify the image and scale as a pair, as shown in the following example.

import torch
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, ControlNetCondition

controlnet_canny = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", 
                                                   torch_dtype=torch.float16).to("cuda")
controlnet_pose = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-openpose", 
                                                   torch_dtype=torch.float16).to("cuda")

pipe = StableDiffusionControlNetPipeline.from_pretrained(
	"andite/anything-v4.0", safety_checker=None, torch_dtype=torch.float16,
	controlnet=[
		controlnet_pose, 
		controlnet_canny
	],
).to("cuda")

image = pipe(prompt='...',
             controlnet_conditions=[
                 ControlNetCondition(pose_image),
                 ControlNetCondition(canny_image, scale=1.2),
             ],
        ).images[0]
image.save("output.png")

[patrickvonplaten]

Hey @takuma104,

Thanks a lot for playing around with the proposed design. It's indeed not as straight-forward - I've made a draft here that I think would be a nice design and solve our use-case quite clearly:
#2621

Would you maybe like to copy-paste the PR and try to get a PR ready with this design? Think this should work quite nicely :-)

[patrickvonplaten]

Also happy to introduce a scale parameter to the ControlNet in a follow-up PR :-)

[patrickvonplaten]

Also cc @williamberman can you help @takuma104 here ?

[takuma104]

@patrickvonplaten Wow, this is amazing! I didn't think of this method. I feel like a weight has been lifted off my shoulders. I'll manually merge and then open the draft PR.