community controlnet inpainting pipelines #2561

williamberman · 2023-03-06T08:49:14Z

These are community controlnet pipelines inspired by https://github.com/haofanwang/ControlNet-for-Diffusers/

They are not exhaustively tested but I ran through a few examples manually and they seem to work ok. Please note that they will not be officially supported. Feel free to open issues and I can get to them when I have time but they won't be prioritized.

I do not know how sound the img2img pipeline is. I haven't thought too hard about if noising the encoded latents as such + using the additional controlnet conditioning works as intended.

note that community pipelines can also be loaded as such https://huggingface.co/docs/diffusers/v0.14.0/en/using-diffusers/custom_pipeline_overview#loading-official-community-pipelines

inpainting

import numpy as np
import torch
from PIL import Image
from stable_diffusion_controlnet_inpaint import StableDiffusionControlNetInpaintPipeline

from transformers import AutoImageProcessor, UperNetForSemanticSegmentation
from diffusers import ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image

def ade_palette():
    return [[120, 120, 120], [180, 120, 120], [6, 230, 230], [80, 50, 50],
            [4, 200, 3], [120, 120, 80], [140, 140, 140], [204, 5, 255],
            [230, 230, 230], [4, 250, 7], [224, 5, 255], [235, 255, 7],
            [150, 5, 61], [120, 120, 70], [8, 255, 51], [255, 6, 82],
            [143, 255, 140], [204, 255, 4], [255, 51, 7], [204, 70, 3],
            [0, 102, 200], [61, 230, 250], [255, 6, 51], [11, 102, 255],
            [255, 7, 71], [255, 9, 224], [9, 7, 230], [220, 220, 220],
            [255, 9, 92], [112, 9, 255], [8, 255, 214], [7, 255, 224],
            [255, 184, 6], [10, 255, 71], [255, 41, 10], [7, 255, 255],
            [224, 255, 8], [102, 8, 255], [255, 61, 6], [255, 194, 7],
            [255, 122, 8], [0, 255, 20], [255, 8, 41], [255, 5, 153],
            [6, 51, 255], [235, 12, 255], [160, 150, 20], [0, 163, 255],
            [140, 140, 140], [250, 10, 15], [20, 255, 0], [31, 255, 0],
            [255, 31, 0], [255, 224, 0], [153, 255, 0], [0, 0, 255],
            [255, 71, 0], [0, 235, 255], [0, 173, 255], [31, 0, 255],
            [11, 200, 200], [255, 82, 0], [0, 255, 245], [0, 61, 255],
            [0, 255, 112], [0, 255, 133], [255, 0, 0], [255, 163, 0],
            [255, 102, 0], [194, 255, 0], [0, 143, 255], [51, 255, 0],
            [0, 82, 255], [0, 255, 41], [0, 255, 173], [10, 0, 255],
            [173, 255, 0], [0, 255, 153], [255, 92, 0], [255, 0, 255],
            [255, 0, 245], [255, 0, 102], [255, 173, 0], [255, 0, 20],
            [255, 184, 184], [0, 31, 255], [0, 255, 61], [0, 71, 255],
            [255, 0, 204], [0, 255, 194], [0, 255, 82], [0, 10, 255],
            [0, 112, 255], [51, 0, 255], [0, 194, 255], [0, 122, 255],
            [0, 255, 163], [255, 153, 0], [0, 255, 10], [255, 112, 0],
            [143, 255, 0], [82, 0, 255], [163, 255, 0], [255, 235, 0],
            [8, 184, 170], [133, 0, 255], [0, 255, 92], [184, 0, 255],
            [255, 0, 31], [0, 184, 255], [0, 214, 255], [255, 0, 112],
            [92, 255, 0], [0, 224, 255], [112, 224, 255], [70, 184, 160],
            [163, 0, 255], [153, 0, 255], [71, 255, 0], [255, 0, 163],
            [255, 204, 0], [255, 0, 143], [0, 255, 235], [133, 255, 0],
            [255, 0, 235], [245, 0, 255], [255, 0, 122], [255, 245, 0],
            [10, 190, 212], [214, 255, 0], [0, 204, 255], [20, 0, 255],
            [255, 255, 0], [0, 153, 255], [0, 41, 255], [0, 255, 204],
            [41, 0, 255], [41, 255, 0], [173, 0, 255], [0, 245, 255],
            [71, 0, 255], [122, 0, 255], [0, 255, 184], [0, 92, 255],
            [184, 255, 0], [0, 133, 255], [255, 214, 0], [25, 194, 194],
            [102, 255, 0], [92, 0, 255]]


image_processor = AutoImageProcessor.from_pretrained("openmmlab/upernet-convnext-small")
image_segmentor = UperNetForSemanticSegmentation.from_pretrained("openmmlab/upernet-convnext-small")

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-seg", torch_dtype=torch.float16)

pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16
)

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_xformers_memory_efficient_attention()
pipe.enable_model_cpu_offload()

def image_to_seg(image):
    pixel_values = image_processor(image, return_tensors="pt").pixel_values
    with torch.no_grad():
        outputs = image_segmentor(pixel_values)
    seg = image_processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
    color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8)  # height, width, 3
    palette = np.array(ade_palette())
    for label, color in enumerate(palette):
        color_seg[seg == label, :] = color
    color_seg = color_seg.astype(np.uint8)
    seg_image = Image.fromarray(color_seg)
    return seg_image

image = load_image(
    "https://github.com/CompVis/latent-diffusion/raw/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
)

mask_image = load_image(
    "https://github.com/CompVis/latent-diffusion/raw/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
)

controlnet_conditioning_image = image_to_seg(image)

image = pipe(
    "Face of a yellow cat, high resolution, sitting on a park bench",
    image,
    mask_image,
    controlnet_conditioning_image,
    num_inference_steps=20,
).images[0]

image.save("out.png")

inpainting + image variation

import numpy as np
import torch
from PIL import Image
from stable_diffusion_controlnet_inpaint_img2img import StableDiffusionControlNetInpaintImg2ImgPipeline

from transformers import AutoImageProcessor, UperNetForSemanticSegmentation
from diffusers import ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image

def ade_palette():
    return [[120, 120, 120], [180, 120, 120], [6, 230, 230], [80, 50, 50],
            [4, 200, 3], [120, 120, 80], [140, 140, 140], [204, 5, 255],
            [230, 230, 230], [4, 250, 7], [224, 5, 255], [235, 255, 7],
            [150, 5, 61], [120, 120, 70], [8, 255, 51], [255, 6, 82],
            [143, 255, 140], [204, 255, 4], [255, 51, 7], [204, 70, 3],
            [0, 102, 200], [61, 230, 250], [255, 6, 51], [11, 102, 255],
            [255, 7, 71], [255, 9, 224], [9, 7, 230], [220, 220, 220],
            [255, 9, 92], [112, 9, 255], [8, 255, 214], [7, 255, 224],
            [255, 184, 6], [10, 255, 71], [255, 41, 10], [7, 255, 255],
            [224, 255, 8], [102, 8, 255], [255, 61, 6], [255, 194, 7],
            [255, 122, 8], [0, 255, 20], [255, 8, 41], [255, 5, 153],
            [6, 51, 255], [235, 12, 255], [160, 150, 20], [0, 163, 255],
            [140, 140, 140], [250, 10, 15], [20, 255, 0], [31, 255, 0],
            [255, 31, 0], [255, 224, 0], [153, 255, 0], [0, 0, 255],
            [255, 71, 0], [0, 235, 255], [0, 173, 255], [31, 0, 255],
            [11, 200, 200], [255, 82, 0], [0, 255, 245], [0, 61, 255],
            [0, 255, 112], [0, 255, 133], [255, 0, 0], [255, 163, 0],
            [255, 102, 0], [194, 255, 0], [0, 143, 255], [51, 255, 0],
            [0, 82, 255], [0, 255, 41], [0, 255, 173], [10, 0, 255],
            [173, 255, 0], [0, 255, 153], [255, 92, 0], [255, 0, 255],
            [255, 0, 245], [255, 0, 102], [255, 173, 0], [255, 0, 20],
            [255, 184, 184], [0, 31, 255], [0, 255, 61], [0, 71, 255],
            [255, 0, 204], [0, 255, 194], [0, 255, 82], [0, 10, 255],
            [0, 112, 255], [51, 0, 255], [0, 194, 255], [0, 122, 255],
            [0, 255, 163], [255, 153, 0], [0, 255, 10], [255, 112, 0],
            [143, 255, 0], [82, 0, 255], [163, 255, 0], [255, 235, 0],
            [8, 184, 170], [133, 0, 255], [0, 255, 92], [184, 0, 255],
            [255, 0, 31], [0, 184, 255], [0, 214, 255], [255, 0, 112],
            [92, 255, 0], [0, 224, 255], [112, 224, 255], [70, 184, 160],
            [163, 0, 255], [153, 0, 255], [71, 255, 0], [255, 0, 163],
            [255, 204, 0], [255, 0, 143], [0, 255, 235], [133, 255, 0],
            [255, 0, 235], [245, 0, 255], [255, 0, 122], [255, 245, 0],
            [10, 190, 212], [214, 255, 0], [0, 204, 255], [20, 0, 255],
            [255, 255, 0], [0, 153, 255], [0, 41, 255], [0, 255, 204],
            [41, 0, 255], [41, 255, 0], [173, 0, 255], [0, 245, 255],
            [71, 0, 255], [122, 0, 255], [0, 255, 184], [0, 92, 255],
            [184, 255, 0], [0, 133, 255], [255, 214, 0], [25, 194, 194],
            [102, 255, 0], [92, 0, 255]]


image_processor = AutoImageProcessor.from_pretrained("openmmlab/upernet-convnext-small")
image_segmentor = UperNetForSemanticSegmentation.from_pretrained("openmmlab/upernet-convnext-small")

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-seg", torch_dtype=torch.float16)

pipe = StableDiffusionControlNetInpaintImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16
)

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_xformers_memory_efficient_attention()
pipe.enable_model_cpu_offload()

def image_to_seg(image):
    pixel_values = image_processor(image, return_tensors="pt").pixel_values
    with torch.no_grad():
        outputs = image_segmentor(pixel_values)
    seg = image_processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
    color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8)  # height, width, 3
    palette = np.array(ade_palette())
    for label, color in enumerate(palette):
        color_seg[seg == label, :] = color
    color_seg = color_seg.astype(np.uint8)
    seg_image = Image.fromarray(color_seg)
    return seg_image

image = load_image(
    "https://github.com/CompVis/latent-diffusion/raw/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
)

mask_image = load_image(
    "https://github.com/CompVis/latent-diffusion/raw/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
)

controlnet_conditioning_image = image_to_seg(image)

image = pipe(
    "Face of a yellow cat, high resolution, sitting on a park bench",
    image,
    mask_image,
    controlnet_conditioning_image,
    num_inference_steps=20,
).images[0]

image.save("out.png")

cc @Suhail @haofanwang

HuggingFaceDocBuilderDev · 2023-03-06T08:57:10Z

The documentation is not available anymore as the PR was closed or merged.

patrickvonplaten

Thanks!

pcuenca

Thanks! I made a comment about citing the original authors.

examples/community/stable_diffusion_controlnet_inpaint.py

yiyixuxu

awesome!

@pcuenca

* community controlnet inpainting pipelines * add community member attribution re: @pcuenca

@pcuenca

* community controlnet inpainting pipelines * add community member attribution re: @pcuenca

CrazyBoyM · 2023-04-26T17:16:46Z

Hi, happy to see your great work! but can you tell me what's the different with inpainting + image variation

CrazyBoyM · 2023-04-26T17:35:27Z

And will the inpaint pipe support the 'MultiControlNetModel' in pipeline_stable_diffusion_controlnetand and stable_diffusion_controlnet_img2img ?

patrickvonplaten · 2023-04-27T12:07:27Z

@williamberman @yiyixuxu I wonder whether controlnet is popular enough to actually move those two pipelines into src/diffusers

RustyKettle · 2023-05-27T23:01:06Z

@williamberman @yiyixuxu I wonder whether controlnet is popular enough to actually move those two pipelines into src/diffusers

I am using the controlnet+inpainting+img2img extensively, and would love to see it moved into being officially supported. I would like to see all the recent Lora updates from the other pipelines worked into controlnet+img2img+inpaint.

patrickvonplaten · 2023-05-30T10:14:44Z

Hey @RustyKettle,

We've merged exactly this PR a couple of days ago: #3533

RustyKettle · 2023-05-30T21:53:33Z

Hey @RustyKettle,

We've merged exactly this PR a couple of days ago: #3533

It looks like it is just the inpaint pipeline. I cannot find the combination inpaint/img2img pipeline. Is this something that is planned? I'm using a version I dug up on google, but it would be great if it was officially supported with Loras. It would simplify my project significantly if I can figure this out.

@pcuenca

* community controlnet inpainting pipelines * add community member attribution re: @pcuenca

community controlnet inpainting pipelines

cc72daa

williamberman force-pushed the controlnet_stable_diffusion_inpainting branch from 68d0a2d to cc72daa Compare March 6, 2023 09:03

williamberman mentioned this pull request Mar 6, 2023

Add a ControlNet model & pipeline #2407

Merged

5 tasks

williamberman requested review from yiyixuxu, sayakpaul, patrickvonplaten and pcuenca March 6, 2023 09:07

patrickvonplaten approved these changes Mar 6, 2023

View reviewed changes

pcuenca approved these changes Mar 6, 2023

View reviewed changes

examples/community/stable_diffusion_controlnet_inpaint.py Show resolved Hide resolved

takuma104 mentioned this pull request Mar 6, 2023

Adding support for ControlNet pipeline for img2img flows #2562

Closed

yiyixuxu approved these changes Mar 6, 2023

View reviewed changes

williamberman added 2 commits March 6, 2023 11:02

add community member attribution re: @pcuenca

0faebd0

Merge branch 'main' into controlnet_stable_diffusion_inpainting

d02d5f3

williamberman merged commit ca7ca11 into huggingface:main Mar 6, 2023

yiyixuxu mentioned this pull request Mar 6, 2023

diffusers community pipeline based on your repo haofanwang/ControlNet-for-Diffusers#27

Closed

mikegarts mentioned this pull request Mar 7, 2023

community stablediffusion controlnet img2img pipeline #2584

Merged

yiyixuxu mentioned this pull request Mar 23, 2023

Create a img2img (+ inpainting) controlnet pipeline #2783

Closed

mengfei25 pushed a commit to mengfei25/diffusers that referenced this pull request Mar 27, 2023

community controlnet inpainting pipelines (huggingface#2561)

b18553e

* community controlnet inpainting pipelines * add community member attribution re: @pcuenca

w4ffl35 pushed a commit to w4ffl35/diffusers that referenced this pull request Apr 14, 2023

community controlnet inpainting pipelines (huggingface#2561)

5a683ae

* community controlnet inpainting pipelines * add community member attribution re: @pcuenca

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

community controlnet inpainting pipelines #2561

community controlnet inpainting pipelines #2561

Uh oh!

williamberman commented Mar 6, 2023 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Mar 6, 2023 •

edited

Loading

Uh oh!

patrickvonplaten left a comment

Uh oh!

pcuenca left a comment

Uh oh!

Uh oh!

yiyixuxu left a comment

Uh oh!

CrazyBoyM commented Apr 26, 2023

Uh oh!

CrazyBoyM commented Apr 26, 2023

Uh oh!

patrickvonplaten commented Apr 27, 2023

Uh oh!

RustyKettle commented May 27, 2023

Uh oh!

patrickvonplaten commented May 30, 2023

Uh oh!

RustyKettle commented May 30, 2023

Uh oh!

Uh oh!

community controlnet inpainting pipelines #2561

community controlnet inpainting pipelines #2561

Uh oh!

Conversation

williamberman commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

inpainting

inpainting + image variation

Uh oh!

HuggingFaceDocBuilderDev commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

CrazyBoyM commented Apr 26, 2023

Uh oh!

CrazyBoyM commented Apr 26, 2023

Uh oh!

patrickvonplaten commented Apr 27, 2023

Uh oh!

RustyKettle commented May 27, 2023

Uh oh!

patrickvonplaten commented May 30, 2023

Uh oh!

RustyKettle commented May 30, 2023

Uh oh!

Uh oh!

williamberman commented Mar 6, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Mar 6, 2023 •

edited

Loading