Skip to content

Tokenizer 1.22.1
Compare
Choose a tag to compare
@guillaumekln guillaumekln released this 30 Oct 11:37
· 164 commits to master since this release
v1.22.1
83e60f6

Fixes and improvements

  • Fix error when enabling vocabulary restriction with SentencePiece and spacer_annotate is not explicitly set
  • Fix backward compatibility with Kangxi and Kanbun scripts (see segment_alphabet option)
Assets 2
Loading