Code

Refactor: GGUF metadata tokenizer#389

Code

Merged

EricLBuehler merged 25 commits intoEricLBuehler:masterEricLBuehler/mistral.rs:masterfrom polarathene:refactor/gguf-metadata-tokenizerpolarathene/mistral.rs:refactor/gguf-metadata-tokenizerCopy head branch name to clipboardJun 7, 2024

+619-280

Commits

Commits on Jun 4, 2024

• tests: Use cfg(test) attribute to avoid dead_code warnings

  Show description for 1545395
  polarathene

  committed
  1545395

• tests: DRY codec test cases

  Show description for ad6ca10
  polarathene

  committed
  ad6ca10

• chore: Add TODO note regarding test remote data dependency

  polarathene

  committed
  f3ba6d9

Commits on Jun 5, 2024

• refactor: DRY metadata extraction

  Show description for 18f1567
  polarathene

  committed
  18f1567

• refactor: Extract unigram tokenizer out of match statement

  Show description for c9651fa
  polarathene

  committed
  c9651fa

• chore: rustfmt adjustments + notes

  polarathene

  committed
  5417221

• refactor: GGUF Unigram Tokenizer Vocab construction

  polarathene

  committed
  fe24df7

• Merge branch 'master' into refactor/gguf-metadata-tokenizer

  polarathene

  authored
  0c78b31

• Update gguf_tokenizer.rs

  polarathene

  authored
  ea4fd54

Commits on Jun 6, 2024

• chore: Rename MetadataContext => ContentMetadata

  polarathene

  committed
  fa70ffc

• chore: verify_sanity_gguf() => verify_arch()

  Show description for bbe4d00
  polarathene

  committed
  bbe4d00

• chore: Expand GGUF Value enum types support

  Show description for 4ee563a
  polarathene

  committed
  4ee563a

• refactor: GGUF metadata - quantized_llama.rs

  polarathene

  committed
  ec16212

• refactor: GGUF metadata - quantized_phi2.rs

  polarathene

  committed
  4cf25e5

• refactor: GGUF metadata - quantized_phi3.rs

  polarathene

  committed
  c4dfe68

• refactor: GGUF metadata - X-LoRA llama + phi3

  Show description for bbea097
  polarathene

  committed
  bbea097

• tests: Skip encoder test case for special tokens

  Show description for 86f538c
  polarathene

  committed
  86f538c

• Update mistralrs-core/src/pipeline/gguf_tokenizer.rs

  polarathene

  authored
  8bdc736

Commits on Jun 7, 2024

• refactor: Use convenience enums for Decoder and Normalizer inputs

  Show description for b3705c3
  polarathene

  committed
  b3705c3

• chore: Add a tokenizer builder workaround

  Show description for 130b1ac
  polarathene

  committed
  130b1ac

• chore: MetadataContent path_prefix to &str

  polarathene

  committed
  dba3024

• tests: Skip Decoder with special tokens

  Show description for 4b8d775
  polarathene

  committed
  4b8d775

• fix: Decoder tests

  Show description for 67e972f
  polarathene

  committed
  67e972f

• tests: Replace web request with hard-coded string

  polarathene

  committed
  74b3319

• docs: Add maintenance reference comment

  Show description for fe48b9c
  polarathene

  committed
  fe48b9c