learned-tokenization

Here are 2 public repositories matching this topic...

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

My attempts at applying Soundstream design on learned tokenization of text and then applying hierarchical attention to text generation

Add a description, image, and links to the learned-tokenization topic page so that developers can more easily learn about it.

To associate your repository with the learned-tokenization topic, visit your repo's landing page and select "manage topics."