Skip to content

Demon-Sheriff/shared_bias_tokenizer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

cohere extension experiment to the paper : One Tokenizer to Rule Them All

training run for the universal tokenizer finetune :

image
image image image
image image image image image image image image image image

alt text

alt text

alt text

alt text

alt text

About

biasing the universal tokenizer and an attempt to optimize compression rates in multilingual compression

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages