Code

HF Tokenizers#11

Code

Merged

larryliu0820 merged 16 commits intopytorch-labs:mainpytorch-labs/tokenizers:mainfrom

gabe-l-hart:HFTokenizersgabe-l-hart/tokenizers:HFTokenizersCopy head branch name to clipboard

Dec 21, 2024

Commits

Commits on Dec 18, 2024

• feat: Add json submodule

  Show description for 31b8380
  gabe-l-hart

  committed
  31b8380

  Copy full SHA for 31b8380

• feat: Add BPETokenizerBase

  Show description for 46cf28f
  gabe-l-hart

  committed
  46cf28f

  Copy full SHA for 46cf28f

• refactor: Rebase Tiktoken onto BPETokenizerBase

  Show description for e61f5ed
  gabe-l-hart

  committed
  e61f5ed

  Copy full SHA for e61f5ed

• build: Add bpe_tokenizer_base to build

  Show description for e31e16d
  gabe-l-hart

  committed
  e31e16d

  Copy full SHA for e31e16d

• build/ci/test: Reactor unit test build

  Show description for 1bbb9bd
  gabe-l-hart

  committed
  1bbb9bd

  Copy full SHA for 1bbb9bd

• build: Use absolute paths and glob all source files

  Show description for aac4404
  gabe-l-hart

  committed
  aac4404

  Copy full SHA for aac4404

• feat: Add vendored copy of llama.cpp's unicode module

  Show description for 4c4d957
  gabe-l-hart

  committed
  4c4d957

  Copy full SHA for 4c4d957

• build: Add build support for private headers in src/include

  Show description for 78bc122
  gabe-l-hart

  committed
  78bc122

  Copy full SHA for 78bc122

• feat: Add pre_tokenizer suite

  Show description for 2df9c5c
  gabe-l-hart

  committed
  2df9c5c

  Copy full SHA for 2df9c5c

• feat: Add token_decoder abstraction and byte level implementation

  Show description for bcbfcc2
  gabe-l-hart

  committed
  bcbfcc2

  Copy full SHA for bcbfcc2

• feat: Add HFTokenizer implementation

  Show description for acad6fe
  gabe-l-hart

  committed
  acad6fe

  Copy full SHA for acad6fe

• feat: Add tokenize_tool with build support

  Show description for 681df89
  gabe-l-hart

  committed
  681df89

  Copy full SHA for 681df89

• style: Run linting on all authored code

  Show description for 3328824
  gabe-l-hart

  committed
  3328824

  Copy full SHA for 3328824

• refactor: Move vendored llama.cpp unicode module into third-party

  Show description for 766c364
  gabe-l-hart

  committed
  766c364

  Copy full SHA for 766c364

• style: Add missing trailing newlines

  Show description for bcfd996
  gabe-l-hart

  committed
  bcfd996

  Copy full SHA for bcfd996

Commits on Dec 19, 2024

• fix: Add missing include for linux build

  Show description for 887ebed
  gabe-l-hart

  committed
  887ebed

  Copy full SHA for 887ebed