Mixtral output quality drop after 4553

[psmitsu]

I've checked q3_K_M Mixtral and it seems like it got "dumber" after the prompt processing speed up #4553 Below are example runs for pre-merge and merge cases with temperature 0.

Prompt:
[INST] In a store, when an item is sold, the seller applies a sales tax of 10% on it and a 15% discount. In order to save most money for the buyer, what is better to calculate first: a tax or a discount? Calculate the final price for both cases (don't be verbous, write concisely) first, and then explain what property of multiplication operation is the reason for what's happening. [/INST]

Output before the merge:

First, let's calculate the final price in each case.

Case 1: Discount first, then tax:
Final Price = Original Price * (1 - Discount Rate) * (1 + Tax Rate)
= Original Price * (1 - 0.15) * (1 + 0.1)

Case 2: Tax first, then discount:
Final Price = Original Price * (1 + Tax Rate) * (1 - Discount Rate)
= Original Price * (1 + 0.1) * (1 - 0.15)

Now, observe that the sequence of operations inside the parentheses is different between the two cases. This difference arises from the commutative but not associative property of multiplication. The order in which we perform calculations matters when there are multiple operations involved.

However, if you look closely at the expressions, you will notice that they simplify down to the same value regardless of the order. That's because the product of a number and its inverse under multiplication results in unity (1). In this case, subtracting the discount rate followed by adding the tax rate or vice versa cancels out due to their opposite effects on the original price.

So, whether you apply the discount or tax first doesn't actually affect the final outcome. It won't save more money for the buyer either way.

Output after merge:
First, let's calculate the final price in each case.

Case 1: Discount first, then tax:

Final Price = Original Price - Discount * Original Price + Tax * Final Price
FP = OP - D*OP + T*FP
FP - T*FP = OP - D\*OP
FP(1-T) = OP(1-D)
FP = OP(1-D)/(1-T)

Case 2: Tax first, then discount:
Final Price = Original Price \* (1+Tax) \* (1-Discount)
FP = OP \* (1+T) \* (1-D)

Now, we can see that the final price calculation results are not equal due to the different sequence of operations. However, if there was no tax or discount, they would be equal. This illustrates the commutative property of multiplication, which states that changing the order of factors does not change the product. But here, since subtraction is involved, it doesn't follow this property anymore.

However, if both the tax and discount were constant percentages, it wouldn't matter because either way you get the same relative difference from the original price. In practice though, sellers usually apply taxes before discounts, so mathematically speaking, applying a discount first gives more savings for the buyer.

In both cases I run identical settings:

GEN_OPTIONS=(--temp 0.0
  --keep -1
  --repeat_penalty 1.17647
  --repeat_last_n 256
  --min-p 0.02
  --top-p 1.0
  --top-k 0
  --typical 1.0
  --mirostat 2
  --mirostat-lr 0.05
  --mirostat-ent 3.0 
  --seed -1) 

Idk if other options, sampling in particular, play a role when temp is 0.0, so I've included them just in case. Also other options:

RUN_OPTIONS=(--color 
  --batch_size 1024
  --ctx_size 4096
  --threads 4
  --gpu-layers 14
  --log-enable) 

@JohannesGaessler could you please take a look?

[JohannesGaessler]

From what I can tell you're not even using the same seed so why would you expect the output to stay the same?

[psmitsu]

@JohannesGaessler thank you for checking! I assumed that with temperature 0 the seed should not matter (I get the exact same output rerunning). Anyways, I have done running with the seed set to 777 and got the exact same outputs.

[JohannesGaessler]

That being said, I do observe a change in perplexity but I would need to run the calculation for more than a few chunks to determine whether this is beyond rounding error.

[kalomaze]

That being said, I do observe a change in perplexity but I would need to run the calculation for more than a few chunks to determine whether this is beyond rounding error.

It has been observed in the past that batch size causes a change in the logits (before Mixtral even existed). Batch size 512 also errors out no matter the GPU layers offloaded for someone at 8192 ctx, but only if the batch size is 512 or bigger (256 and 128 worked for them). Could have something to do with it?

[BadisG]

I think you need to increase your batch size, it's a known bug at this point
abetlen/llama-cpp-python#1018

[psmitsu]

@BadisG Setting batch size to 4096 or 8192 doesn't change the output for me at all. I haven't found a way to print token probabilities in llama.cpp (server should do that, but retuns completion_probabilities: [] when I set e.g. n_probs=10)

@kalomaze I don't know if that information helps, but on new version with batch sizes 1 to 16 the model produces a correct answer to the prompt, after that batch_size it begins to hallucinate. The output it gives (on batch sizes 1-16) matches the output of the older version, but the latter give the correct output all the time, just rephrases it after batch size 128. By "older version" I mean the one with #4538

[psmitsu]

@JohannesGaessler I have run perplexity on wikitext-2-raw-v1.zip with gpu offloading and results for both builds are roughly the same: with 4553 I've got 4.0422 +/- 0.02040 and with 4538 I've got 4.0440 +/- 0.02042 . So the change in output is probably related to what @kalomaze has mentioned, i.e. batching changes the logits.