
I used the "GRADUAL RGB" model trained on the ParallelDomain-4D dataset provided by you, and set the following code in the configuration file:
- input_key: scaled_relative_pose
is_trainable: False
target: sgm.modules.encoders.modules.CameraEmbedder
params:
embed_dim: 128
zero_init: False
However, in the video Unet module, the encoding of the condition is based on a 768 input dimension, while using a custom camera perspective will change the condition to 896 dimensions. Through debugging, I found that your video Unet's aux_emb-dim is 0, but it should actually be defined as 128. I am not sure if my change is correct
I used the "GRADUAL RGB" model trained on the ParallelDomain-4D dataset provided by you, and set the following code in the configuration file:
However, in the video Unet module, the encoding of the condition is based on a 768 input dimension, while using a custom camera perspective will change the condition to 896 dimensions. Through debugging, I found that your video Unet's aux_emb-dim is 0, but it should actually be defined as 128. I am not sure if my change is correct