vocab: refactor tokenizer to reduce the overhead of creating multi times tokenizer#9449
Merged
ggerganov merged 12 commits intoggerganov:masterfrom kylo5aby:refactor-tokenizerSep 28, 2024
+239-142
Commits
Commits on Sep 20, 2024
- committed
- committed
- committed
Merge branch 'gg/tokenizer-cleanup' of https://github.com/ggerganov/llama.cpp into ggerganov-gg/tokenizer-cleanup
committed- committed
- committed
- committed
Commits on Sep 23, 2024
Commits on Sep 24, 2024
- committed
Commits on Sep 25, 2024
- authored
- authored