Flaky server responses with llama 3

[kurnevsky]

I noticed that some of the responses I got from llama-cpp server (latest master) are unnaturally fast for 70b model, and it happens randomly. And when this happens the response has worse quality. The model I'm using is https://huggingface.co/NousResearch/Meta-Llama-3-70B-Instruct-GGUF/blob/main/Meta-Llama-3-70B-Instruct-Q5_K_M.gguf with the command line llama-server -m Meta-Llama-3-70B-Instruct-Q5_K_M.gguf -c 0 -t 24 -ngl 24. It's only partially offloaded to gpu (with rocm on linux) so maybe somehow llama-cpp doesn't use all layers when it responds quickly.

[phymbert]

Llama 3 is not yet supported, please wait for:

• Support Llama 3 conversion #6745
• Added llama-3 chat template #6751

[kurnevsky]

It works perfectly fine when it's responding slowly. I do not use chat template, I use my own client that calls /completion endpoint.

[phymbert]

Feel free to reopen once those 2 PR are merged.

[kurnevsky]

I have no rights to reopen issues in this repo. Also I looked through those MRs - they have nothing to do with this problem.

[phymbert]

llama3 is not supported, could you understand the GGUF you are using is probably just wrong ?

[phymbert]

If your issue persists once you have converted again the HF model and run the latest server code with those PRs merged. Please ping. I will reopen