Commits

Breadcrumbs

History for

onmaster

User selector

All users

Datepicker

All time

Commit History

Commits on Mar 1, 2023

• Consider escaped characters as single characters in BPE (#322)

  guillaumekln

  authored
  e8cf860

  Copy full SHA for e8cf860

• Cleanup signature of function utf8_to_cp

  guillaumekln

  committed
  e7b96f4

  Copy full SHA for e7b96f4

Commits on Feb 28, 2023

• Add tokenization option allow_isolated_marks (#316)

  Show description for d5c1908
  guillaumekln

  authored
  d5c1908

  Copy full SHA for d5c1908

Commits on Dec 6, 2022

• Implement pickle support for Vocab objects (#303)

  guillaumekln

  authored
  d3b5ad8

  Copy full SHA for d3b5ad8

Commits on Jul 21, 2022

• Expose token counters from Vocab class (#287)

  guillaumekln

  authored
  16f6952

  Copy full SHA for 16f6952

Commits on Mar 7, 2022

• Add a Vocab class and related functions (#281)

  Show description for 45fc569
  guillaumekln

  authored
  45fc569

  Copy full SHA for 45fc569

Commits on Sep 16, 2021

• Remove the SpaceTokenizer class (#263)

  guillaumekln

  authored
  9281191

  Copy full SHA for 9281191

• Allow setting a custom tokens delimiter when writing to files (#261)

  Show description for 0302fa0
  guillaumekln

  authored
  0302fa0

  Copy full SHA for 0302fa0

Commits on May 31, 2021

• Fix application of subword vocabulary for tokens with "preserve" flag (#248)

  guillaumekln

  authored
  251d5ee

  Copy full SHA for 251d5ee

Commits on Apr 22, 2021

• Require ICU 60 or greater to use lang option (#245)

  guillaumekln

  authored
  d696bad

  Copy full SHA for d696bad

Commits on Apr 12, 2021

• Add lang argument and apply locale-dependent recasing (#240)

  Show description for a35a48f
  guillaumekln

  authored
  a35a48f

  Copy full SHA for a35a48f

Commits on Mar 15, 2021

• Add training flag to enable/disable subword regularization (#233)

  Show description for 8176188
  guillaumekln

  authored
  8176188

  Copy full SHA for 8176188

Commits on Feb 12, 2021

• Add verbose flag to log tokenization progress (#226)

  guillaumekln

  authored
  560ec01

  Copy full SHA for 560ec01

Commits on Jan 18, 2021

• Add 'options' property to Python Tokenizer instances (#224)

  Show description for d8ef499
  guillaumekln

  authored
  d8ef499

  Copy full SHA for d8ef499

Commits on Jan 14, 2021

• Add missing export symbol and suppress dll-interface warnings with MSVC (#221)

  panosk

  authored
  a7a2f74

  Copy full SHA for a7a2f74

Commits on Jan 11, 2021

• feat: Adds a argument to control if keep sentencepiece training data. (#217)

  innerNULL

  authored
  aa07284

  Copy full SHA for aa07284

Commits on Dec 29, 2020

• Fix space tokenization on protected sequences with spaces (#214)

  guillaumekln

  authored
  490a3fb

  Copy full SHA for 490a3fb

Commits on Dec 17, 2020

• Improve support of Common and Inherited script properties (#212)

  Show description for 18ef771
  guillaumekln

  authored
  18ef771

  Copy full SHA for 18ef771

Commits on Dec 16, 2020

• Split characters into CharInfo objects (#207)

  Show description for a69b7ee
  guillaumekln

  authored
  a69b7ee

  Copy full SHA for a69b7ee

Commits on Dec 4, 2020

• Forward declare SentencePieceProcessor to avoid subclassing (#202)

  guillaumekln

  authored
  5ba01a6

  Copy full SHA for 5ba01a6

Commits on Nov 19, 2020

• Move static constant outside of class definition

  guillaumekln

  committed
  30a5bb6

  Copy full SHA for 30a5bb6

• Precompute set of alphabet codes to segment

  guillaumekln

  committed
  82aa249

  Copy full SHA for 82aa249

Commits on Nov 18, 2020

• Small optimizations (#194)

  Show description for f65b78e
  guillaumekln

  authored
  f65b78e

  Copy full SHA for f65b78e

Commits on Oct 26, 2020

• Add getters for options and subword encoder

  guillaumekln

  committed
  46868ea

  Copy full SHA for 46868ea

• Add pointer to options documentation

  guillaumekln

  committed
  5e108a9

  Copy full SHA for 5e108a9

• Remove comment that is no longer accurate

  guillaumekln

  committed
  aab02ff

  Copy full SHA for aab02ff

• Declare tokenization options when setting a subword vocabulary (#186)

  Show description for 35aaea7
  guillaumekln

  authored
  35aaea7

  Copy full SHA for 35aaea7

• Replace flags by a structure of options (#185)

  guillaumekln

  authored
  caf67f1

  Copy full SHA for caf67f1

• Add lowercase method to Token class

  guillaumekln

  committed
  e94f049

  Copy full SHA for e94f049

• Minor style harmonization

  guillaumekln

  committed
  68334be

  Copy full SHA for 68334be

• Remove mapMode, use str_to_mode function instead

  guillaumekln

  committed
  c92ea3d

  Copy full SHA for c92ea3d

• Deprecate subword encoder caching (#184)

  Show description for 3a8912e
  guillaumekln

  authored
  3a8912e

  Copy full SHA for 3a8912e

• Add set_random_seed function (#182)

  guillaumekln

  authored
  0f5e254

  Copy full SHA for 0f5e254

Commits on Oct 22, 2020

• Remove ambiguous tokenization methods (#181)

  guillaumekln

  authored
  11c8b55

  Copy full SHA for 11c8b55

• Avoid some copies in BPE loading methods

  guillaumekln

  committed
  cd1fcfe

  Copy full SHA for cd1fcfe

Pagination

Previous

Next