Inquiry Regarding Removal of Adaptive Audio Modulation (AAM)

[chiuwhsin]

In the previous technical report, Adaptive Audio Modulation (AAM) was included as part of the training pipeline. However, I notice this technique is absent from both the latest v1.5 technical report and the code implementation.
Could you provide insight into the rationale for removing AAM? It appeared to be a promising technique based on earlier documentation. Alternatively, is it still being utilized but simply not explicitly mentioned in the current documentation?
Thank you for any clarification you can provide.

[aidenyzhang]

Due to space limits and topic relevance, the v1.5 technical report focuses on new findings and improvements.
In our previous experiments, we analyzed the number of audio segments per video frame (T) in AAM, finding T=5 optimal for our previous SyncNet.

However, LatentSync trained larger SyncNet and proved that T=16 is the best.
We ran some AAM experiments on this SyncNet and found that the best T indeed differed from our previous SyncNet.
The reason might be that the larger SyncNet can handle larger temporal information.
In this version of MuseTalk_v1.5, we released the checkpoint trained with LatentSync's SyncNet.

We also think that combining our sampling strategy in AMM (sampling from only one individual within a batch) with the larger SyncNet model of LatentSync could yield better results, which is still under experiments.