Problems of IWAE ELBO Loss

[GloryyrolG]

Hi Anand and all,

As weighting of samples, weight should be detached from the current computational graph for the expected optimization objective, right? See

PyTorch-VAE/models/iwae.py

Line 155 in 8700d24

weight = F.softmax(log_weight, dim = -1)

[GloryyrolG]

Actually I saw there is a detach statement but in the annotation.

PyTorch-VAE/models/iwae.py

Line 152 in 8700d24

log_weight = (log_p_x_z + kld_weight * kld_loss) #.detach().data

[GloryyrolG]

Besides, as the original paper said, "Vanilla VAE separated out the KL divergence in the bound in order to achieve a
simpler and lower-variance update. Unfortunately, no analogous trick applies for k > 1" (Y. Burda et al., 2016). How are we still able to compute KL Divergence?

PyTorch-VAE/models/iwae.py

Line 152 in 8700d24

log_weight = (log_p_x_z + kld_weight * kld_loss) #.detach().data

[tongdaxu]

I also found this change very suspicious.

In the original paper Eq 14, we have:

this obviously requires the grad w to be detached. or else the grad will be equals to:

\Sum (w * \nabla \log(ELBO) + \nabla w * \log(ELBO))

which has an additional term due to taking derivative wrt to \sum(w * ELBO)

[tongdaxu]

Besides, as the original paper said, "Vanilla VAE separated out the KL divergence in the bound in order to achieve a simpler and lower-variance update. Unfortunately, no analogous trick applies for k > 1" (Y. Burda et al., 2016). How are we still able to compute KL Divergence?

PyTorch-VAE/models/iwae.py

Line 152 in 8700d24

log_weight = (log_p_x_z + kld_weight * kld_loss) #.detach().data

I think you are also right here, the SGVB 2 estimator separate out KL divergence out of the monte carlo sampling of reparameterized noise. Here, we should use SGVB 1 instead and use Monte Carlo to compute the whole log p(x, y) - q(y|h).

[tongdaxu]

Kindly refers to PR: #53