Skip to content

AnimateDiff Video to Video #6328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 59 commits into from
Jan 24, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
df1b6c4
begin animatediff img2video and video2video
a-r-r-o-w Dec 25, 2023
4be3068
revert animatediff to original implementation
a-r-r-o-w Dec 26, 2023
06b427f
add img2video as pipeline
a-r-r-o-w Dec 26, 2023
aaf9194
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Dec 26, 2023
2bc77c6
update
a-r-r-o-w Dec 27, 2023
d0b3893
add vid2vid pipeline
a-r-r-o-w Dec 27, 2023
466d92a
update imports
a-r-r-o-w Dec 31, 2023
fc815c8
update
a-r-r-o-w Dec 31, 2023
315daad
remove copied from line for check_inputs
a-r-r-o-w Dec 31, 2023
cc55f3d
update
a-r-r-o-w Dec 31, 2023
d7a85be
update examples
a-r-r-o-w Dec 31, 2023
7dd73ac
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Dec 31, 2023
a831a5e
add multi-batch support
a-r-r-o-w Jan 3, 2024
b5b5a3a
fix __init__.py files
a-r-r-o-w Jan 3, 2024
8bb0855
move img2vid to community
a-r-r-o-w Jan 3, 2024
26d3145
update community readme and examples
a-r-r-o-w Jan 3, 2024
4038d9a
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 3, 2024
3196a79
fix
a-r-r-o-w Jan 3, 2024
5a4f2ee
make fix-copies
a-r-r-o-w Jan 3, 2024
7fad71a
add vid2vid batch params
a-r-r-o-w Jan 4, 2024
71e8770
apply suggestions from review
a-r-r-o-w Jan 4, 2024
068e9d7
add test for animatediff vid2vid
a-r-r-o-w Jan 4, 2024
da4c308
torch.stack -> torch.cat
a-r-r-o-w Jan 4, 2024
be2bb21
make style
a-r-r-o-w Jan 4, 2024
43b4410
docs for vid2vid
a-r-r-o-w Jan 4, 2024
4ce5bae
update
a-r-r-o-w Jan 4, 2024
f895be8
fix prepare_latents
a-r-r-o-w Jan 4, 2024
2cb3267
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 4, 2024
2b0533d
fix docs
a-r-r-o-w Jan 4, 2024
cf2b1b3
remove img2vid
a-r-r-o-w Jan 9, 2024
f6f4079
update README to :main
a-r-r-o-w Jan 9, 2024
193edcd
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 9, 2024
b1c5db9
remove slow test
a-r-r-o-w Jan 10, 2024
3fc8623
refactor pipeline output
a-r-r-o-w Jan 10, 2024
817b44e
update docs
a-r-r-o-w Jan 10, 2024
caa423e
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 10, 2024
6543079
update docs
a-r-r-o-w Jan 11, 2024
df602b3
merge community readme from :main
a-r-r-o-w Jan 11, 2024
897dfd2
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 11, 2024
5042b7d
final fix i promise
a-r-r-o-w Jan 11, 2024
9a2d8ba
add support for url in animatediff example
a-r-r-o-w Jan 12, 2024
78fa5a8
update example
a-r-r-o-w Jan 12, 2024
967fb6c
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 12, 2024
fe871f7
update callbacks to latest implementation
a-r-r-o-w Jan 17, 2024
600e414
Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video…
a-r-r-o-w Jan 17, 2024
254ea67
Update src/diffusers/pipelines/animatediff/pipeline_animatediff_video…
a-r-r-o-w Jan 17, 2024
c4bf30c
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 17, 2024
ed37cae
fix merge
a-r-r-o-w Jan 17, 2024
6b84aef
Apply suggestions from code review
patrickvonplaten Jan 19, 2024
1c645ed
remove callback and callback_steps as suggested in review
a-r-r-o-w Jan 19, 2024
54b21c0
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 19, 2024
fdbb68f
Update tests/pipelines/animatediff/test_animatediff_video2video.py
a-r-r-o-w Jan 19, 2024
39a7628
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 23, 2024
5674a71
fix import error caused due to unet refactor in #6630
a-r-r-o-w Jan 23, 2024
032c24f
fix numpy import error after tensor2vid refactor in #6626
a-r-r-o-w Jan 23, 2024
41ac862
make fix-copies
a-r-r-o-w Jan 23, 2024
c3a70eb
fix numpy error
a-r-r-o-w Jan 23, 2024
8b820a0
fix progress bar test
a-r-r-o-w Jan 23, 2024
872dee6
Merge branch 'main' into animatediff-img2video
a-r-r-o-w Jan 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update examples
  • Loading branch information
a-r-r-o-w committed Dec 31, 2023
commit d7a85be76b5410ec908b56afca8ef8e06919baf2
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,15 @@
Examples:
```py
>>> import torch
>>> from diffusers import MotionAdapter, AnimateDiffPipeline, DDIMScheduler
>>> from diffusers.utils import export_to_gif
>>> from diffusers import MotionAdapter, AnimateDiffImg2VideoPipeline, DDIMScheduler
>>> from diffusers.utils import export_to_gif, load_image

>>> adapter = MotionAdapter.from_pretrained("diffusers/motion-adapter")
>>> pipe = AnimateDiffPipeline.from_pretrained("frankjoshua/toonyou_beta6", motion_adapter=adapter)
>>> pipe.scheduler = DDIMScheduler(beta_schedule="linear", steps_offset=1, clip_sample=False)
>>> output = pipe(prompt="A corgi walking in the park")
>>> pipe = AnimateDiffImg2VideoPipeline.from_pretrained("SG161222/Realistic_Vision_V5.1_noVAE", motion_adapter=adapter).to("cuda")
>>> pipe.scheduler = DDIMScheduler(beta_schedule="linear", steps_offset=1, clip_sample=False, timespace_spacing="linspace")

>>> img = load_image("snail.png")
>>> output = pipe(image=image, prompt="A snail moving on the ground", strength=0.8, latent_interpolation_method="slerp")
>>> frames = output.frames[0]
>>> export_to_gif(frames, "animation.gif")
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,25 @@
EXAMPLE_DOC_STRING = """
Examples:
```py
>>> import imageio
>>> import torch
>>> from diffusers import MotionAdapter, AnimateDiffPipeline, DDIMScheduler
>>> from diffusers.utils import export_to_gif
>>> from diffusers import MotionAdapter, AnimateDiffVideo2VideoPipeline, DDIMScheduler
>>> from diffusers.utils import export_to_gif, load_image

>>> adapter = MotionAdapter.from_pretrained("diffusers/motion-adapter")
>>> pipe = AnimateDiffPipeline.from_pretrained("frankjoshua/toonyou_beta6", motion_adapter=adapter)
>>> pipe.scheduler = DDIMScheduler(beta_schedule="linear", steps_offset=1, clip_sample=False)
>>> output = pipe(prompt="A corgi walking in the park")
>>> pipe = AnimateDiffVideo2VideoPipeline.from_pretrained("SG161222/Realistic_Vision_V5.1_noVAE", motion_adapter=adapter).to("cuda")
>>> pipe.scheduler = DDIMScheduler(beta_schedule="linear", steps_offset=1, clip_sample=False, timespace_spacing="linspace")

>>> def load_video(file_path):
>>> images = []
>>> vid = imageio.get_reader(file_path)
>>> for i, frame in enumerate(vid):
>>> pil_image = Image.fromarray(frame)
>>> images.append(pil_image)
>>> return images

>>> video = load_image("animation_fireworks.png")
>>> output = pipe(video=video, prompt="Closeup of a woman, fireworks in the background", strength=0.7)
>>> frames = output.frames[0]
>>> export_to_gif(frames, "animation.gif")
```
Expand Down