Tokenizers questions and ... proposals?

[JoanFM]

Hello @ggerganov ,

Thanks for the great project. I have been trying to include the Jina Embedding models into llama.cpp as you can see in #6826.

I have been successful on having it run for most of the models that Jina offers, (English, Spanish and German) but I cannot have it for Chinese.

I havee seen that the issue comes from the tokenization part of the model and I have been digging more into the code for llama.cpp as the one from tokenizers in HuggingFace.

I have some questions that I will try to place here.

1st. - How to know which tokenizer needs to be used for each model. For instance, I see that the SPM and BPE tokenizers here seems to work quite similarly but there are some discrepancies.

2nd. - I have seen that the problem from the Chinese model when it comes to the differences in output compared to the usage of transformers comes from the fact that the model uses some Normalizers and PreTokenizers that are very hard to configure in llama.cpp.

I wonder if there would be need to do some refactoring in the tokenizer to enable the decoupling of the tokenizing logic with the surrouding normalization code, plus some options to have a reacher mapping of the tokenizer options in transformers and in llama.cpp.

I am not sure if my observations here make any sense or I am just missusing the project or missunderstanding some of the concepts.

Thank you for the great work and happy to bring some help.

[ggerganov]

On master there is no way to support correct tokenization for BPE/WPM tokenizers

Please take a look at the description in #6920 - this will be merged soon and it will introduce a pre-tokenizer field that llama.cpp can use to do pre-tokenization correctly. I've focused only on BPE tokenizers in that PR. AFAICT the Jina tokenizer falls in the WPM category - probably similar handling would be necessary, but I think it should be relatively simple to add. First step is to add the model as explained in the PR and create tokenizer tests for it and then we'll see what is necessary

[JoanFM]

Okey I will follow closely the developments, thank you very much

[JoanFM]

Hey @ggerganov ,

I have been checking the PR and the pretokenization closely and I have some questions and doubts.

• In the model which I was trying to add, it was a Chinese model where NFC Unicode Normalization and LowerCase as a list of Normalizers from HF (https://huggingface.co/docs/tokenizers/components). I am not sure how I can map this to a regex, or I can do the normalization without regex?

• I have another model which I would like to apply which has a PreCompiled HF tokenizer with a precompiled_charsmap attribute which I am failing to understand. (I was wondering if that has in fact any relationship with the regex you set in the llama : improve BPE pre-processing + LLaMA 3 and Deepseek support #6920 PR.

Thanks for all the help and attention I am receiving in my PRs and inquiries.

[ggerganov]

I'm not sure about NFC normalization - need to understand what it is first. Lowercase should be easy to apply in the tokenize() method before or after the regex split based on vocab.type_pre

Not familiar with pre-compiled char maps

[JoanFM]

Another question.

In order to decide the pretokenizer we do this:

        chktok = tokenizer.encode(chktxt)
        chkhsh = sha256(str(chktok).encode()).hexdigest()

and we pretend to identify the pretrokenization behavior based on this.

But does this actually make sense?

The chktok may differ even for models with the same pretokenziation behavior but just a different vocabulary. Shouldn't there be a way to have this independent of the actual vocabulary?

Wdyt @ggerganov ?

[JoanFM]

@ggerganov @slaren @teleprint-me I have a question.

If I were to consider adding support for NFC normalization, would you be open to include other dependencies as Boost or ICU to depend on this, or is including external dependencies very much avoided regarding the nature of llama.cpp to intend to be used for edge devices?

[JoanFM]

Also, Perhaps a nice option to better scale the efforts to handle tokenization logic could be to take a look at https://github.com/mlc-ai/tokenizers-cpp which seems to be a port from tokenizers and been linked in this issue huggingface/tokenizers#185.

[slaren]

Large external dependencies like boost are out of the question. Small, self-contained libraries that can be bundled in the repository have a higher chance of being accepted by @ggerganov, but the preference is still to reduce dependency on external libraries.

[ggerganov]

The chktok may differ even for models with the same pretokenziation behavior but just a different vocabulary. Shouldn't there be a way to have this independent of the actual vocabulary?

Yes, the resulting hashes would be different, but if you observe that the pre-tokenizer configs are the same, you can assign the same "name" for both models in order to reuse the pre-tokenizer. But also adding a duplicate one is fine.

If I were to consider adding support for NFC normalization, would you be open to include other dependencies as Boost or ICU to depend on this

From quick look, is NFC normalization similar to NFD normalization? We have the latter already implemented:

https://github.com/ggerganov/llama.cpp/blob/3275e60f575fb1141e4706da2e9dc810d7d47457/unicode.cpp#L472-L485

Seems simple enough to implement from scratch

[JoanFM]

Hey @ggerganov ,

To be honest, I know NFC is related to NFD but not sure how wasy is to implement, I am trying to understand some implementations but I am quite new to this. (I believe is a little more complex as it combines symbols more than decomposing). I will try to dig deeper, thanks

I know you can map to the same name from different hashes, but would be nice if it could actually detect it in a better way. (I have a small proposal in #7039 for some imrpovement).

[teleprint-me]

NFC simply merges similar characters together to reduce redundancy, but the importance and practicality of such an implementation is debatable. It really depends on the use case and needs of the project. I have no say here, but I agree with @ggerganov on implementing a function from scratch if it really is desired for some reason. There's a nifty online tool with a brief overview on what the concept is.

Edit: It's important to keep in mind that there's a definite potential for data loss with the use of NFC, but everything is compressed here. It's all compression, so I suppose it's more practical to think in terms of how lossy it is in comparison to other methods.

[JoanFM]

Hey @teleprint-me ,

As per wether it i should be used or not, I believe this is a discussion to be had when training the tokenizer, in the case of the library, I believe that if a model is trained with a tokenizer requiring this normalization process, we should be able to produce this normalization.

As per wether we should implement ourselves, I agree and I am working in that direction. However I am finding it very hard to get the data needed for the mapping. Where did @ggerganov get the required data to complete the unicode_map_nfd?

The algorithm I understand it goes as:

• Decompose the unicode_cpts recursively following the mapping rules that can be extracted (I will see how later) from https://www.unicode.org/Public/13.0.0/ucd/UnicodeData.txt
• Reorder adjacent unicode_cpts according to its Unicode Canonical Combining Class (also extracted potentially from same file)
• Apply the Canonical Composition Algorithm (as described here https://unicode.org/reports/tr15/#Description_Norm). I guess this takes a map of pairs (or more) of unicode_pts and merges them into a single unicode_cpts. The problem is I have no clue where to extract the mappings to obtain such behavior.

Is there any hint you could give me as per where I could look for?

Thanks

[JoanFM]

@iamlemec may be able to provide more details about the current implementation.

Plus I have a question,

In theory NFC should be equivalent somehow to using NFD plus a Normalization step, but I am doubting why the NFD implementation right now uses find for the multi_map instead of range_equals as in the first implementation (#5799). It seems to me that find will return only the first occurecne and thus the decomposition would be wrong?

Thanks for all the help