Just Relax It

Discrete Variables Relaxation

📬 Assets

1. Technical Meeting 1 - Presentation
2. Technical Meeting 2 - Jupyter Notebook
3. Blog Post
4. Documentation
5. Tests

💡 Motivation

For lots of mathematical problems we need an ability to sample discrete random variables. The problem is that due to continuos nature of deep learning optimization, the usage of truely discrete random variables is infeasible. Thus we use different relaxation method. One of them, Concrete distribution or Gumbel-softmax (this is one distribution proposed in parallel by two research groups) is implemented in different DL packages. In this project we implement different alternatives to it.

🗃 Algorithms to implement (from simplest to hardest)

• Relaxed Bernoulli
• Correlated relaxed Bernoulli
• Gumbel-softmax TOP-K
• Straight-Through Bernoulli, distribution (don't mix with Relaxed distribution from pyro)
• Invertible Gaussian reparametrization with KL implemented
• Hard concrete
• REINFORCE (not a distribution actually, think how to integrate it with other distributions)
• Logit-normal distribution and Laplace-form approximation of Dirichlet

📚 Recommended stack

Some of the alternatives for GS were implemented in pyro, so it might be useful to play with them also.

🧩 Problem details

To make to library constistent, we integrate imports of distributions from pyro and pytorch into the library, so that all the categorical distributions can be imported from one entrypoint.

👥 Contributors

• Daniil Dorin (Basic code writing, Final demo, Algorithms)
• Igor Ignashin (Project wrapping, Documentation writing, Algorithms)
• Nikita Kiselev (Project planning, Blog post, Algorithms)
• Andrey Veprikov (Tests writing, Documentation writing, Algorithms)

🔗 Useful links

• About top-k GS
• VAE implementation with different latent distributions
• KL divergence between Dirichlet and Logistic-Normal implemented in R
• About score function (SF) and pathwise derivate (PD) estimators, VAE and REINFORCE