This is a repo for vector quantization and modern self-organizing maps (SOMs). Additionally, I added a differential version of these SOMs based on this. I also include a test on CIFAR which makes pretty pictures, visualized below!
codebook.mp4
Here is a similar visualization of the SOM codebook usage when reconstructing audio (which was admittedly not as cool as I hoped it would be):
audio_codebook.mp4
Here, each color represents a different codebook, so the plot is displaying the entries producing the sound at that instant in the video. I was hoping it would not look so random. Nonetheless, I hope that SOMs will serve to make attention mechanisms on the codebook more robust (since the codebook has a natural notion of neighborhood and proximity among entries), as well as giving structure to "interpolation" between codebook entries.
To install this package, run:
pip install git+https://github.com/LumenPallidium/quantization-maps.git
The only requirements are Pytorch (>=2.0) and einops. The above PIP install command will install Pytorch, but I would reccomend installing on your own independently, so you can configure any neccesary environments, CUDA tools, etc.