Not generating emojis correctly

[AshD]

Prompt: Write a Christmas message with emojis

Using the InstructExecutor.InferAsync() with the Mistral Instruct 7B model

Output: ������🎉Merry Christmas!���🏻���

Looks like the output is not getting tokenized correctly.

Thanks,
Ash

[martindevans]

This is a known issue, and is actually a surprisingly common issue across tokenizers in many projects! For example try this tool in text mode and paste in some emoji. It's caused when one single character is represented by multiple tokens.

This has been fixed in the StatelessDecoder by using the StreamingTokenDecoder. That same fix will need to be applied to the other executors at some point.

[AshD]

Thanks Martin.

Quick question - Would this cause an issue with Chinese and Japanese characters too?

Thanks,
Ash

[martindevans]

Would this cause an issue with Chinese and Japanese characters too?

Yes! In fact the issue report that got me working on this issue was from someone using Korean, but there were other related issues from people using Cyrillic and Chinese characters (and I'd spotted it myself in English, but with emoji).

It's not every character in those languages, just ones which the tokenizer cannot represent in a single token (which seems to be fairly rare).

[AshD]

Thanks Martin.
I tried the StatelessExecutor.
It did give me correct emojis but then continued to output nonsense text. Pasting a small snippet below.

🎁🎅🎈 Merry Christmas! 💙🎊industriëšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšćîšìîšćîšìîšćîšćìšćîššìšćîshć****š️šćîšćìšćîshccššćîshć~\šćìšćîshćìšćîshćìšćîshćìšćîshćìšćîshćìšćîshćìšćîshćìšćîshćìšćîshćššćîshćìšćîshć~\ishšćîshćìšćîshćìšćîshćìšćîshćìšćîshćìšćîshćìšćîshćššćîshćìšćîshćìšćîshćìšćîshćìšćîshćššćîshćìšćîshćîshćîshć Industîshć îshîshćindustîshćİshš️šīshćîshćìšćîshćîshćîshćîshćîshćîshćîshćîshćîshćîshćîshćîshćîshćîshćîshšishšćîshšš️ìšćîshćîshćîshćîshćîshćîshšìšćîshšìšćîshćîshšìšćîshšìšćîshšìšćîshšìšćîshšìš️šššīshšìšćîshšš️ìšćîshšishšćîshšìšćîshšš️ìššìšćîshšìšćîshšìšćîshšìšćîshšīshšš️ìššš️ìšîshšıshšìš️6îshšìšćîshšìšćîshšìšćîshšìš️shîshćîshšìšćîshšīshšishšćîshšìšćîshšıshšìš-----------------š--------------------îshš---------------------,6š--------------------îshš----------------îshć Industîshš️5š--------------------îshšīshš

[martindevans]

What did you have set as your antiprompts?

[AshD]

List<string> { "<END>", "User:", ">" }

[martindevans]

From the antiprompts it looks like it's "correct" that it never stopped - it never generated any of the antiprompts. Could you try exactly the same model and prompt in the main llama.cpp project to see if it produces the same output? It may just be that the model is getting confused by the emoji and gets stuck outputting nonsense.

[hswlab]

I have just tried the current version of the Master Branch with emoji in the text, unfortunately they still seem to be displayed incorrectly.

Do I need to activate the StreamingTokenDecoder feature somehow?
I have used the avx2 binaries for testing, which I downloaded here:
https://github.com/SciSharp/LLamaSharp/actions/runs/6762323560

[martindevans]

Was this using the StatelessExecutor? The StreamingTokenDecoder fixes the problem, but it's only being used in that one executor right now. Someone is going to have to modify the other executors to use it.

[hswlab]

@martindevans Do you mean the Inference Type "Stateless"?
This seem to work with emoji, but it is writing some strange text :D

Edit: It seems that this strange text is only generated after an emoji is displayed. The output can then no longer be canceled.

[hswlab]

Oh, I think this problem is caused by a different model. Previously I tested with vicuna-7b-v1.5.Q3_K_M.gguf which was caused this strange behaviour in the text output. but now I tried llama-2-7b-chat.Q5_K_M.gguf, which gives me a much better result :)

[martindevans]

That still looks like the model is running on without stopping. What are the antiprompts set to?

[hswlab]

The Antiprompt is set to default "User:"

[martindevans]

Ah I guess we're not looking at the raw text here, it's presumably stripping out the antiprompt so we can't see it.