animatediff result is worse than origin Repo

[akk-123]

Describe the bug

It's seems that animatediff result is bad, generate image is blur, however when I use the sdwebui or origin animatediff, it can get good result

diffuser result
https://github.com/huggingface/diffusers/assets/98469560/994c50d7-0568-4798-8e33-61251c77cd36

sdwebui or origin repo result
https://github.com/huggingface/diffusers/assets/98469560/dee390c2-7e95-4cee-a041-086d95d03cc8

Reproduction

adapter = MotionAdapter.from_pretrained("xx/animatediff-motion-adapter-v1-5-2")
model_id = "xx/majicmixRealistic_v7" 

animate_pipe = AnimateDiffPipeline.from_pretrained(model_id, motion_adapter=adapter).to('cuda')
scheduler = DDIMScheduler.from_pretrained(
    model_id, subfolder="scheduler", clip_sample=False, timestep_spacing="linspace", steps_offset=1
)
animate_pipe.scheduler = scheduler

prompt="1girl, offshoulder, light smile, shiny skin best quality, masterpiece, photorealistic"
seed = 42
frames = animate_pipe(
    prompt=prompt,
    num_frames=16,
    guidance_scale=7.5,
    num_inference_steps=30,
    generator=torch.Generator("cpu").manual_seed(seed),
).frames[0]

Logs

No response

System Info

• diffusers version: 0.23.0
• Platform: Linux-4.9.70-040970-generic-x86_64-with-glibc2.23
• Python version: 3.10.13
• PyTorch version (GPU?): 2.0.1+cu118 (True)
• Huggingface_hub version: 0.17.3
• Transformers version: 4.30.2
• Accelerate version: 0.21.0
• xFormers version: 0.0.20

Who can help?

No response

[DN6]

@akk-123 Looking into it. Just to confirm, the results used the exact same prompts, seeds and schedulers? I ask because the subjects seem to be different in the two results you shared.

[akk-123]

@DN6 yes, I use the same config

[akk-123]

@DN6 Any progress about it?

[cfanyyx]

@DN6 Hi, Any progress about it? I found the same problem. I think maybe the implementation of animatediff in diffusers has something wrong?

[WuTao-CS]

I meet the same problem, and the result seemed to be significantly different from the original repo.

[DN6]

Hi @akk-123 taking a look into it this week. Do you notice the same quality difference when you use other checkpoints?

[akk-123]

Hi @akk-123 taking a look into it this week. Do you notice the same quality difference when you use other checkpoints?

yes, all checkpoints is worse

@DN6 Hi, Any progress about it? I found the same problem. I think maybe the implementation of animatediff in diffusers has something wrong?

I agree, maybe the implementation of animatediff in diffusers has something wrong

[DN6]

Hi @akk-123 I think the issue seems to be with the beta_schedule set by the checkpoint. Changing it to linear removed the artifacts from the generated gif.

import torch
from diffusers import AnimateDiffPipeline, MotionAdapter, DDIMScheduler
from diffusers.utils import export_to_gif

adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2")
model_id = "pagebrain/majicmix-realistic-v7"

pipe = AnimateDiffPipeline.from_pretrained(model_id, motion_adapter=adapter).to('cuda')
scheduler = DDIMScheduler.from_pretrained(
    model_id,
    subfolder="scheduler",
    beta_schedule="linear",
    clip_sample=False,
    timestep_spacing="linspace",
    steps_offset=1
)
pipe.scheduler = scheduler
pipe.to("cuda")
pipe.enable_vae_slicing()

prompt="1girl, offshoulder, light smile, shiny skin best quality, masterpiece, photorealistic"
seed = 42
frames = pipe(
    prompt=prompt,
    num_frames=16,
    guidance_scale=7.5,
    num_inference_steps=30,
    generator=torch.Generator("cpu").manual_seed(seed),
).frames[0]
export_to_gif(frames, "output.gif")

[akk-123]

@DN6 thanks, I try it, It's work, why does this parameter(beta_schedule) have such a big impact on the results?

[DN6]

I think the original repository trained the model using the linear schedule with DDIM. It could be that the motion checkpoint is sensitive to it.

[a-r-r-o-w]

@DN6 Kindly requesting your suggestions to be reflected in blogs like this. The quality change is quite significant and it would help a lot of people like me that were wondering about it.