Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vocab: refactor tokenizer to reduce the overhead of creating multi times tokenizer #9449

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

vocab: refactor tokenizer to reduce the overhead of creating multi times tokenizer #9449

wants to merge 9 commits into from
+234 −137

Commits on Sep 20, 2024

  1. refactor tokenizer

    @kylo5aby
    kylo5aby committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d949c58 View commit details
    Browse the repository at this point in the history

  2. llama : make llm_tokenizer more private 

    ggml-ci
    @ggerganov
    ggerganov committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    6e873e5 View commit details
    Browse the repository at this point in the history

  3. refactor tokenizer

    @kylo5aby
    kylo5aby committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    403758f View commit details
    Browse the repository at this point in the history

  4. Merge branch 'gg/tokenizer-cleanup' of https://github.com/ggerganov/l… 

    …lama.cpp into ggerganov-gg/tokenizer-cleanup
    @kylo5aby
    kylo5aby committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d653d25 View commit details
    Browse the repository at this point in the history

  5. refactor tokenizer

    @kylo5aby
    kylo5aby committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    2ec25db View commit details
    Browse the repository at this point in the history

  6. llama : make llm_tokenizer more private 

    ggml-ci
    @ggerganov @kylo5aby
    ggerganov authored and kylo5aby committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    02629d9 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'ggerganov-gg/tokenizer-cleanup' into refactor-tokenizer

    @kylo5aby
    kylo5aby committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    8be5d11 View commit details
    Browse the repository at this point in the history

  8. remove unused files

    @kylo5aby
    kylo5aby committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    25d4599 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. remove unused fileds to avoid unused filed build error

    @kylo5aby
    kylo5aby committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    768c43f View commit details
    Browse the repository at this point in the history