Gemma-7b-it quantized still not working

[nxphi47]

Hi, I install the latest llama.cpp on macos and tried gemma-7b-it-GGUF-Q4, but the model still generate nonsense either in chat mode or completion mode. I have added the <bos> token to the prompt.

 ./main -m  ./mlabonne/gemma-7b-it-GGUF/gemma-7b-it.Q4_0.gguf -n 256  --color -i

Results:

Please help check or instruct how to fix this problem.

[dranger003]

This model is quite sensitive, take a look at the model page here and try the following:

./main -p "<start_of_turn>user\nWrite an essay about AI.<end_of_turn>\n<start_of_turn>model\n" -e --repeat-penalty 1.0 -ngl 29 -m ggml-gemma-7b-it-iq3_m.gguf

[nxphi47]

@dranger003 thanks ! It turns out that —repeat-penalty is the culprit https://huggingface.co/google/gemma-7b-it/discussions/38

@ggerganov Could you consider setting —repeat-penalty to 1.0 by default instead ? Would that damage other models performance? Usually repeat-penalty is not enabled by default in other frameworks.