AnimateDiff + ControlNet

[yiyixuxu]

Model/Pipeline/Scheduler description

Anyone interested in adding a AnimateDiffControlNetPipeline?

The expected behavior is to allow user to pass a list of conditions (e.g. pose) and use them to condition the generation for each frame.

see comfy UI example workflow here
https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved#txt2img-w-initial-controlnet-input-using-openpose-images--latent-upscale-w-full-denoise

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[charchit7]

Hey @yiyixuxu, I'm excited to give this a try. Do you have an estimated time frame for how long it might take?

[staghado]

Interested as well!

[yiyixuxu]

@charchit7 @staghado

welcome!! and feel free to collaborate

There is no official reference implementation for this pipeline. You can dig into the source code of comfy UI or auto1111 extension if you want. Or you can just experiment on your own to come up something that works well and fit in our design:) so it's a little bit hard to estimate the timeframe.

I suggest starting with our ControlNet pipeline and AnimateDiff pipeline code and try to combine the logics in a way so that this example would work. The first draft should not take long :)

[yiyixuxu]

cc @DN6 here

[a-r-r-o-w]

@charchit7 @staghado If it's alright, I'd like to jump in as well if you're collaborating.

[charchit7]

@a-r-r-o-w, @staghado happy to collaborate :)

[EdoardoBotta]

I'm happy to collaborate on this as well.

[a-r-r-o-w]

@charchit7 @staghado @EdoardoBotta Awesome! Let's set up a discord server to discuss more on this and distribute work?

[hackpk]

@charchit7 @staghado @EdoardoBotta @a-r-r-o-w Can I join too?