Where is the task decoder during the training process?

[AzureRossi]

In the released training code, I don't see a trainable task decoder to preserve the semantics of the latent.
Is this open-sourced code an implementation of the pipeline diagram shown in the paper?
Furthermore, if there is no task decoder during training to maintain the latent's semantics, doesn't the entire training process just become a representation trade-off? That is, adding a residual branch enhances the ability to reconstruct details, but at the cost of sacrificing semantics."

[howlin-wang]

You are correct that we did not add a trainable task decoder during the training of the generative model. Based on the results reported in Table 4 of the paper, adding the residual branch does not lead to any noticeable change in downstream task performance. Therefore, we believe that the semantics of the latent are largely unaffected.

[AzureRossi]

Thank you very much for your reply. I observed that in Table 4 of the original paper, you only report the semantic probe results after adding both the residual branch and distribution alignment, but do not report the linear probe results for using only the residual branch. I would like to know whether distribution alignment turned out to play an unexpectedly large role in preserving semantics. In addition, when the number of residual channels is increased, does this distribution alignment effect on semantic preservation break down or become less effective?

[howlin-wang]

Please refer to #4 . Feel free to reach out if you have any further questions.