Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support reading tiktoken tokenizer.model file #31656

Merged
merged 62 commits into from
Sep 6, 2024
Merged

Support reading tiktoken tokenizer.model file #31656

merged 62 commits into from
Sep 6, 2024

Commits on Aug 22, 2024

  1. use existing TikTokenConverter to read tiktoken tokenizer.model file

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ce21f32 View commit details
    Browse the repository at this point in the history

  2. del test file

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    744ab3d View commit details
    Browse the repository at this point in the history

  3. create titktoken integration file

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ef5df3f View commit details
    Browse the repository at this point in the history

  4. adding tiktoken llama test

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    341db42 View commit details
    Browse the repository at this point in the history

  5. ALTNATIVE IMPLEMENTATION: supports llama 405B

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    cc11541 View commit details
    Browse the repository at this point in the history

  6. fix one char

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    418d3d7 View commit details
    Browse the repository at this point in the history

  7. remove redundant line

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    66f38a7 View commit details
    Browse the repository at this point in the history

  8. small fix

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    09f05f5 View commit details
    Browse the repository at this point in the history

  9. rm unused import

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    6426cfb View commit details
    Browse the repository at this point in the history

  10. flag for converting from tiktokeng

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    0b12788 View commit details
    Browse the repository at this point in the history

  11. remove unneeded file

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d0017c4 View commit details
    Browse the repository at this point in the history

  12. ruff

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    0dd2fda View commit details
    Browse the repository at this point in the history

  13. remove llamatiktokenconverter, stick to general converter

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    61279bc View commit details
    Browse the repository at this point in the history

  14. tiktoken support v2

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    31f526d View commit details
    Browse the repository at this point in the history

  15. update test

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    4ac4cd0 View commit details
    Browse the repository at this point in the history

  16. remove stale changes

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c77224b View commit details
    Browse the repository at this point in the history

  17. udpate doc

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c7e7fa1 View commit details
    Browse the repository at this point in the history

  18. protect import

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    9e045d5 View commit details
    Browse the repository at this point in the history

  19. use is_protobuf_available

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d77883e View commit details
    Browse the repository at this point in the history

  20. add templateprocessor in tiktokenconverter

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    e69c709 View commit details
    Browse the repository at this point in the history

  21. reverting templateprocessor from tiktoken support

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    266826b View commit details
    Browse the repository at this point in the history

  22. update test

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    b520024 View commit details
    Browse the repository at this point in the history

  23. add require_tiktoken

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    07a7ce9 View commit details
    Browse the repository at this point in the history

  24. dev-ci

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    3820238 View commit details
    Browse the repository at this point in the history

  25. trigger build

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c994e35 View commit details
    Browse the repository at this point in the history

  26. trigger build again

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ece867a View commit details
    Browse the repository at this point in the history

  27. dev-ci

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    e2cb91a View commit details
    Browse the repository at this point in the history

  28. [build-ci-image] tiktoken

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    0f58a9c View commit details
    Browse the repository at this point in the history

  29. dev-ci

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    97b5099 View commit details
    Browse the repository at this point in the history

  30. dev-ci

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    94f4e00 View commit details
    Browse the repository at this point in the history

  31. dev-ci

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    83ec4bf View commit details
    Browse the repository at this point in the history

  32. dev-ci

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    b4764c4 View commit details
    Browse the repository at this point in the history

  33. change tiktoken file name

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    03bf949 View commit details
    Browse the repository at this point in the history

  34. feedback review

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    1d2cb00 View commit details
    Browse the repository at this point in the history

  35. feedback rev

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    2ac47df View commit details
    Browse the repository at this point in the history

  36. applying feedback, removing tiktoken converters

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    cd979e3 View commit details
    Browse the repository at this point in the history

  37. conform test

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    b5c0618 View commit details
    Browse the repository at this point in the history

  38. adding docs for review

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    40982ea View commit details
    Browse the repository at this point in the history

  39. add doc file for review

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ec14290 View commit details
    Browse the repository at this point in the history

  40. add doc file for review

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    cba0340 View commit details
    Browse the repository at this point in the history

  41. add doc file for review

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    4ef4040 View commit details
    Browse the repository at this point in the history

  42. support loading model without config.json file

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    728252a View commit details
    Browse the repository at this point in the history

  43. Revert "support loading model without config.json file" 

    This reverts commit 2753602.
    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    e6e08ed View commit details
    Browse the repository at this point in the history

  44. remove dev var

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ad53d2c View commit details
    Browse the repository at this point in the history

  45. updating docs

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c08c80f View commit details
    Browse the repository at this point in the history

  46. safely import protobuf

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    55ee7e5 View commit details
    Browse the repository at this point in the history

  47. fix protobuf import error

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    3ff09bf View commit details
    Browse the repository at this point in the history

  48. fix protobuf import error

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    815f1b6 View commit details
    Browse the repository at this point in the history

  49. trying isort to fix ruff error

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    6b568c8 View commit details
    Browse the repository at this point in the history

  50. fix ruff error

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d0ca96f View commit details
    Browse the repository at this point in the history

  51. try to fix ruff again

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    7b34bed View commit details
    Browse the repository at this point in the history

  52. try to fix ruff again

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    4be52ba View commit details
    Browse the repository at this point in the history

  53. try to fix ruff again

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d647025 View commit details
    Browse the repository at this point in the history

  54. doc table of contents

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    a8de4ca View commit details
    Browse the repository at this point in the history

  55. add fix for consistency.dockerfile torchaudio

    @itazap
    itazap committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    747f6f1 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Merge branch 'main' into tiktoken_file_support

    @itazap
    itazap authored Aug 26, 2024
    Configuration menu
    Copy the full SHA
    5ede660 View commit details
    Browse the repository at this point in the history

  2. ruff

    @itazap
    itazap committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    eab392d View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. applying feedback

    @itazap
    itazap committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    97b1e23 View commit details
    Browse the repository at this point in the history

  2. minor typo

    @itazap
    itazap committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    1cdd8ae View commit details
    Browse the repository at this point in the history

  3. merging with push-ci-image

    @itazap
    itazap committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    1c9ee23 View commit details
    Browse the repository at this point in the history

  4. clean up imports

    @itazap
    itazap committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    bf3e795 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. revert dockerfile consistency

    @itazap
    itazap committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    08dde7a View commit details
    Browse the repository at this point in the history