support modern tokenizers for NLP

[martinjaggi]

if we can support recent standard tokenizers (byte-pair, sentencepiece) etc such as used for modern transformer models, we'll have much better compatibility with current NLP models. (and the tokenizers can later be used for any kind of small/big word2vec/lstm/transformer etc models)

the only thing needed at first will be the ability to load and use a predefined (pretrained) tokenizer, such as the ones for say GPT-style models, or from huggingface.

here is a suitable codebase in typescript
https://github.com/botisan-ai/gpt3-tokenizer

let's evaluate if we can use it and if it can load existing pretrained tokenizers (a nice vocabulary size is 2^15 = 32k, but can start smaller for a proof of concept)

[martinjaggi]

this will help for #491

[martinjaggi]

are you guys looking towards allowing general huggingface tokenizers? maybe like https://praeclarum.org/2022/08/25/transformers-js.html

(no need to support the actual transformers yet in this issue, we'll make a follow-up issue later)

[walidabn]

Yes, that's exactly the one we were looking into ( the latest web version from https://github.com/praeclarum/web-transformers), to find a way to implement it in TFJS rather than ONNX as they did.

[martinjaggi]

and this one as well:
https://github.com/xenova/transformers.js

but same problem again is that it usually does conversion pytorch->js via ONNX, which doesn't allow training. but as transformer architecture is simpler than CNNs (and same for all layers), it should be easy to later have the transformer trainable in tfjs.

but anyway, we only need tokenizers at first. standard LSTMs etc can also run nicely on top of tokenized input, which we can support as preprocessing for all NLP tasks