Tokenizers: additional Normalizers, PreTokenizers, PostProcessors

[pcuenca]

So far I've ported the components I needed to support the models I tested, but there are many more in transformers and tokenizers. For example:

• swift-transformers/Sources/Tokenizers/PostProcessor.swift

  Line 27 in 595c5cf

  // case RobertaProcessing

• swift-transformers/Sources/Tokenizers/PreTokenizer.swift

  Line 41 in 595c5cf

  // Several more to be supported

In addition to checking the source code of the tokenizers library, I recommend taking a look at the JavaScript implementation by @xenova in transformers.js – it's a single file and easy to follow!

[jkrukowski]

Added WhitespacePreTokenizer here #20

[011235813]

Adding BertPreTokenizer to support BERT-type encoders would be great.

Tokenizers/PreTokenizer.swift:61: Fatal error: Unsupported PreTokenizer type: BertPreTokenizer

[pcuenca]

Thanks @011235813, could you share the tokenizer you are trying to load?

[ptsochantaris]

@pcuenca

let tokenizer = try await AutoTokenizer.from(pretrained: "sentence-transformers/all-MiniLM-L6-v2")

For example gives an error:

Tokenizers/PreTokenizer.swift:66: Fatal error: Unsupported PreTokenizer type: BertPreTokenizer

[jkrukowski]

@pcuenca

let tokenizer = try await AutoTokenizer.from(pretrained: "sentence-transformers/all-MiniLM-L6-v2")

For example gives an error:

Tokenizers/PreTokenizer.swift:66: Fatal error: Unsupported PreTokenizer type: BertPreTokenizer

should be fixed in #137