Can anyone provide some suggestions to make tora's I2V model run on an RTX 4090? (not T2V model)

[zenmequmingzia]

I accidentally closed the previous issue.
I want to automate this I2V model into my workflow, so when I try to run Tora's I2V model example locally on an RTX 4090, I always encounter CUDA out of memory issues. Can I modify some parameters in Tora/sat/sample_video.py or Tora/sat/configs/tora/inference_sparse.yaml or Tora/sat/configs/tora/model/cogvideox_5b_tora_i2v.yaml to make this model run on an RTX 4090 with 24GB VRAM?

Although I am aware that the tora I2V model can successfully generate corresponding videos in the comfyui-tora, I still want to run it on my local machine through Python code to better integrate this model into my workflow.

You could try the diffusers-version with "--enable_model_cpu_offload" option. https://github.com/alibaba/Tora/blob/main/diffusers-version/README.md

If still OOM, try "--enable_sequential_cpu_offload" instead.

Originally posted by @Leojc in #29

According to Leojc's reply, I checked the Tora diffusers-version. However, the diffusers-version only supports T2V and not I2V? Or did I miss something?

[Leojc]

Indeed the weight file of I2V diffusers version is not available. But the i2v_pipeline code is available and diffusers-version/inference.py supports I2V. One needs to convert sat version weights to diffusers version. You can refer to https://github.com/THUDM/CogVideo/blob/main/tools/convert_weight_sat2hf.py in https://github.com/THUDM/CogVideo/#tools to do this.

[zenmequmingzia]

When I tried using the code at https://github.com/THUDM/CogVideo/blob/main/tools/convert_weight_sat2hf.py with modified parameters—based on the comments in the code, since I wanted to convert the sat version of Tora's i2v to the diffusers version—I only changed a few argument defaults: --num_layers=42, --num_attention_heads=48, --use_rotary_positional_embeddings=True, --scaling_factor=0.7, --snr_shift_scale=1.0, --i2v=True, and --version=1.0.

After running the code, I encountered the following errors:

• Size mismatch for patch_embed.proj.weight: copying a parameter with shape torch.Size([3072, 32, 2, 2]) from checkpoint while current model expects shape torch.Size([3072, 128]).
• Size mismatch for proj_out.weight: copying a parameter with shape torch.Size([64, 3072]) from checkpoint while current model expects shape torch.Size([128, 3072]).
• Size mismatch for proj_out.bias: copying a parameter with shape torch.Size([64]) from checkpoint while current model expects torch.Size([128]).

A similar issue was also mentioned here: #26 (comment).

Could you provide suggestions on how to resolve this problem? Additionally, if converting i2v to diffusers version is necessary for execution—aside from whether the transformer checkpoint (transformer_ckpt) needs conversion—do existing VAE checkpoints (vae_ckpt) and text encoders also require conversion using this script? Or does Alibaba plan to release an official diffusers-compatible version of i2v?

Looking forward to your reply.

Indeed the weight file of I2V diffusers version is not available. But the i2v_pipeline code is available and diffusers-version/inference.py supports I2V. One needs to convert sat version weights to diffusers version. You can refer to https://github.com/THUDM/CogVideo/blob/main/tools/convert_weight_sat2hf.py in https://github.com/THUDM/CogVideo/#tools to do this.