tokenization: double EOS tokens

[Azirine]

The llama.cpp tokenizer currently adds an EOS token unconditionally. However, adding an EOS token to the end of the system prompt is necessary to prevent generation before user input. This leads to two EOS tokens before the user prompt, potentially causing suboptimal performance.

llama.cpp-b2972, MacOS 14.5

./main -m Meta-Llama-3-8B-Instruct-Q8_0.gguf --temp 0 -i -e -p "<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant.<|eot_id|>" -r "<|eot_id|>" --in-prefix "<|start_header_id|>user<|end_header_id|>\n\n" --in-suffix "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

Current behavior:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|>
<|eot_id|>
<|start_header_id|>user<|end_header_id|>

Expected behavior:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>

[arch-btw]

I can't reproduce the first prompt. Which version of llama.cpp are you using?

[arch-btw]

Also -ins is not used for llama3, it's designed to work with Alpaca model

-ins, --instruct: Run the program in instruction mode, which is specifically designed to work with Alpaca models that excel in completing tasks based on user instructions.

https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md