Quantitative measurement of model perplexity for different models and model quantization modes

[noughtmare]

llama.cpp seems to give bad results compared to Facebook's implementation.

Here's an example simple reading comprehension prompt:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book

LLaMA 7B with Facebook's implementation yields:

Seed 1:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book with pictures."
Asked by lone wolf 1788 days ago.

Seed 2 (to show that the above is not just a fluke):

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book with pictures."
Question: "Tom, Mark, and Paul bought books: two with pictures and

While llama.cpp without quantization (so still float16) generates (with --seed 0 -t 8):

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book that is neither as good nor bad."
This solution breaks down the problem into its simple parts; then using those components we can see what each component means by itself, in order to solve this logic puzzle. 1) Tom and Mark had different kinds of books...and so did Paul! (Therefore one out three were without pictures). ... [end of text]

It even has a grammatical error at the end: "one out [of] three"

As you can see the quality of 7B is higher in Facebook's implementation. So, I think you may still have bugs in your implementation or the default parameters could be improved.

[noughtmare]

With quantization the result is also bad:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book WITHOUT PICTURES."
This is just an example question; I can't figure out how to post actual questions that are being asked in my classroom! So, here we go . . . We recently had our first day back after Thanksgiving break. One of the things

[gjmulder]

You might not be comparing apples to apples. e.g. are the --top_p and other parameters identical between implementations?

[noughtmare]

I'm using the default settings, so for the Python code it is:

    temperature: float = 0.8,
    top_p: float = 0.95,

And for llama.cpp:

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000

So I think only the repeat penalty and top_k could be different?

[noughtmare]

If I disable the repeat penalty (I assume --repeat_penalty 0 does that) then I still get low quality results:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book about cars (p1). He chose to read and look "up Hepa", with only p3 one "out did it one time with his different". The time look only only What look and time about a a it about Answer and Mark What?out his one pout the car it. Tom' his bought

repeat_penalty 1 gives me a more sensible but still wrong result:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book without pictures. Paul had a different kind of book than Tom and Mark."
Question: "Jeffrey, Jennifer, and Jackson bought books: three with pictures and one without. Jackson and Jennifer had different kinds of books. What kind did Jeffrey buy?" Answer: "Jeffrey bought a book

(These results are from the quantized 7B model with seed 0 and default parameters except for the repeat penalty)

[gjmulder]

top_k looks to be currently broken, as I recently reported in issue #56. I just now realised that due to #95 identical seeds across implementations are unlikely to produce identical results as per @ggerganov's correction to my comment in that issue.

It does then look like llama.cpp is of lower quality. You've tried other prompts and got similar results?

[noughtmare]

I haven't tested the Python implementation extensively, because Facebook's implementation takes very long to run on my CPU. But I generally feel that running 7B and even 13B with llama.cpp gives results that are below the quality that Facebook has claimed.

[beiller]

Try the following parameters, gives me good quality output:

--temp 0.7 --top_k 40 --top_p 0.5 --repeat_last_n 256 --repeat_penalty 1.17647

Also repeat_penalty = 1.0 means disable. Maybe its not named as it should be 😇

[Urammar]

If 1 means disable whats the point of higher than 1 values? Also its good to let it repeat itself a little, sometimes that makes sense in conversation, but tighter lets it break loops before they begin.

[noughtmare]

Try the following parameters

Still gives me a wrong result with the quantized model:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book with no pictures."
Answer to Question 1739: "The three students were going on an outing. They needed shoes for the trip. Each student owned a pair of shoes that was not his own. Which student wore tennis shoes? (Hint: The answer is in the question.)

With the fp16 model it is also wrong:

Question: "Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book that was not in the picture."
"Tom, Mark, and Paul bought books: two with pictures and one without. Tom and Mark had different kinds of books. What kind did Paul buy?" Answer: "Paul bought a book that was not in the picture." [end of text]

I think the problem is more fundamental than just a change of the parameters.

[gjmulder]

I haven't tested the Python implementation extensively, because Facebook's implementation takes very long to run on my CPU. But I generally feel that running 7B and even 13B with llama.cpp gives results that are below the quality that Facebook has claimed.

It may be simply a case of the project management triangle, i.e. choose any two of:

1. Performance
2. Quality
3. Self-hosting

[noughtmare]

That might be so, but I don't see an obvious reason why the quality would be lower. Quantization could have been a logical cause, but I think I have shown that even the fp16 model has a lower quality.

[Urammar]

If its simply a straight up c++ implementation then it should be the same, but an install step in the github states it must be quantized, which means even if you are running it in fp16 its still been crunched in precision to run better, which naturally means its outputs will slightly differ.

You wouldnt expect a mile long road at 18.2 degrees to end up at the same place as one rebuilt at 18.0 degrees, right?

As you just said as I was typing this, quantization made its brain just that little more crispy, and that clearly slightly effects it. Thats probably not solvable.

[noughtmare]

but an install step in the github states it must be quantized,

I don't think that step is required. The model runs fine without the quantization step. And the readme also claims llama.cpp has "Mixed F16/F32 precision". Edit: there's an example of running without quantization here: #2 (comment)

[beiller]

@Urammar higher than 1 starts to penalize the predicted next token if it occurred in previous N tokens. It will multiply the likelihood by 1/penalty.