[refactor] create modeling blocks specific to AnimateDiff

[a-r-r-o-w]

What does this PR do?

Based on internal conversation with @DN6. I have left some inline questions as well

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@DN6

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[DN6]

Looking good. I think the issues with the failing tests need to be addressed.

[a-r-r-o-w]

@DN6 WDYT about this?

[DN6]

Could you give me an example here?

[a-r-r-o-w]

So, something like this:

init_kwargs = {
    "in_channels": input_channel,
    "prev_output_channel": prev_output_channel,
    "out_channels": output_channel,
    "temb_channels": time_embed_dim,
    "resolution_idx": i,
    "num_layers": reversed_layers_per_block[i] + 1,
    "resnet_eps": norm_eps,
    "resnet_act_fn": act_fn,
    "resnet_groups": norm_num_groups,
    "add_upsample": add_upsample,
    "temporal_num_attention_heads": reversed_motion_num_attention_heads[i],
    "temporal_max_seq_length": motion_max_seq_length,
    "temporal_transformer_layers_per_block": reverse_temporal_transformer_layers_per_block[i],
}

if up_block_type == "CrossAttnUpBlockMotion":
    up_block = CrossAttnUpBlockMotion(
        transformer_layers_per_block=reverse_transformer_layers_per_block[i],
        num_attention_heads=reversed_num_attention_heads[i],
        cross_attention_dim=reversed_cross_attention_dim[i],
        use_linear_projection=use_linear_projection,
        **init_kwargs,
    )
elif up_block_type == "UpBlockMotion":
    up_block = UpBlockMotion(**init_kwargs)
else:
    raise ValueError(
        "Invalid `up_block_type` encountered. Must be one of `CrossAttnUpBlockMotion` or `UpBlockMotion`"
    )

[DN6]

Hmm so the large list of arguments is a problem with the UNet models and associated blocks.

This approach could work, but I would argue that the arguments to these blocks could actually be made much smaller by just setting defaults for certain parameters that are just unlikely to change. Perhaps we revisit when doing the UNet redesign?

[a-r-r-o-w]

Yes sure, let's roll with what we have right now for arguments. Can take this up later and look for suggestions

[DN6]

There are some projects that might import these blocks directly.
https://github.com/search?q=%22from+diffusers.models.unets.unet_3d_blocks+import%22+language:Python&type=code

It would be good to create something similar to

diffusers/src/diffusers/models/transformers/transformer_2d.py

Line 32 in ea1b4ea

class Transformer2DModelOutput(Transformer2DModelOutput):

diffusers/src/diffusers/loaders/lora_pipeline.py

Line 1743 in ea1b4ea

class LoraLoaderMixin(StableDiffusionLoraLoaderMixin):

dummy blocks that inherit from the real blocks and raise a deprecation warning.

[a-r-r-o-w]

ah i see, yes will do that

[a-r-r-o-w]

I've added the deprecation blocks. Is this something to be handled in get_down_block and get_up_block as well?

[DN6]

No that's fine.

[DN6]

Since transfomer is now a block, we don't need to define an output class for it.

[DN6]

Just one minor nit to fix regarding the transformer output. But LGTM otherwise 👍🏽