For lots of mathematical problems we need an ability to sample discrete random variables. The problem is that due to continuos nature of deep learning optimization, the usage of truely discrete random variables is infeasible. Thus we use different relaxation method. One of them, Concrete distribution or Gumbel-softmax (this is one distribution proposed in parallel by two research groups) is implemented in different DL packages. In this project we implement different alternatives to it.
<img src="assets/overview.png"/>
- ☒ Relaxed Bernoulli
- ☐ Correlated relaxed Bernoulli
- ☐ Gumbel-softmax TOP-K
- ☒ Straight-Through Bernoulli, distribution (don’t mix with Relaxed distribution from pyro)
- ☐ Invertible Gaussian reparametrization with KL implemented
- ☒ Hard concrete
- ☐ REINFORCE (not a distribution actually, think how to integrate it with other distributions)
- ☐ Logit-normal distribution with KL implemented and Laplace-form approximation of Dirichlet
Some of the alternatives for GS were implemented in pyro, so it might be useful to play with them also.
To make to library constistent, we integrate imports of distributions from pyro and pytorch into the library, so that all the categorical distributions can be imported from one entrypoint.
- Daniil Dorin (Basic code writing, Final demo, Algorithms)
- Igor Ignashin (Project wrapping, Documentation writing, Algorithms)
- Nikita Kiselev (Project planning, Blog post, Algorithms)
- Andrey Veprikov (Tests writing, Documentation writing, Algorithms)