Skip to content

Latest commit

 

History

History
193 lines (171 loc) · 19.5 KB

README.md

File metadata and controls

193 lines (171 loc) · 19.5 KB

Perplexity

The perplexity example can be used to calculate the so-called perplexity value of a language model over a given text corpus. Perplexity measures how well the model can predict the next token with lower values being better. Note that perplexity is not directly comparable between models, especially if they use different tokenizers. Also note that finetunes typically result in a higher perplexity value even though the human-rated quality of outputs increases.

Within llama.cpp the perplexity of base models is used primarily to judge the quality loss from e.g. quantized models vs. FP16. The convention among contributors is to use the Wikitext-2 test set for testing unless noted otherwise (can be obtained with scripts/get-wikitext-2.sh). When numbers are listed all command line arguments and compilation options are left at their defaults unless noted otherwise. llama.cpp numbers are not directly comparable to those of other projects because the exact values depend strongly on the implementation details.

By default only the mean perplexity value and the corresponding uncertainty is calculated. The uncertainty is determined empirically by assuming a Gaussian distribution of the "correct" logits per and then applying error propagation.

More statistics can be obtained by recording the logits from the FP16 version of a model. To do this, supply perplexity with --kl-divergence-base path/to/logit/binary/file.kld. The program will then record all logits and save them to the provided path in binary format. The logit file will be very large, 11 GiB for LLaMA 2 or 37 GiB for LLaMA 3 when using the Wikitext-2 test set. Once you have the file, supply perplexity with the quantized model, the logits file via --kl-divergence-base, and finally the --kl-divergence argument to indicate that the program should calculate the so-called Kullback-Leibler divergence. This is a measure of how similar the FP16 and the quantized logit distributions are with a value of 0 indicating that the distribution are the same. The uncertainty on the mean KL divergence is calculated by assuming the KL divergence per token follows a Gaussian distribution.

In addition to the KL divergence the following statistics are calculated with --kl-divergence:

  • Ratio of mean FP16 PPL and quantized PPL. Uncertainty is estimated on logits, then propagated. The logarithm of this metric is also calculated and printed, it is 0 if the logit distributions are the same.
  • Difference of mean FP16 PPL and quantized PPL. Uncertainty is estimated on logits, then propagated.
  • Mean change in "correct" token probability. Positive values mean the model gets better at prediction, negative values mean it gets worse.
  • Pearson correlation coefficient of the "correct" token probabilites between models.
  • Percentiles of change in "correct" token probability. Positive values mean the model gets better at prediction, negative values mean it gets worse. Can be used to judge noise vs. quality loss from quantization. If the percentiles are symmetric then the quantization is essentially just adding noise. If the negative values are significantly larger than the positive values then this indicates that the model is actually becoming worse from the quantization.
  • The root mean square of the change in token probabilities. If you were to assume that the quantization simply causes Gaussian noise on the token probabilities then this would be the standard deviation of said noise. The uncertainty on the value is calculated that the change in token probabilities follows a Gaussian distribution. Related discussion: ggerganov#2875 .
  • Same top p: Percentage of how often the token was assigned the highest probabilites by both models. The uncertainty is calculated from the Gaussian approximation of the binomial distribution.

LLaMA 3 8b Scoreboard

Revision f364eb6f
Backend CUDA
CPU AMD Epyc 7742
GPU 1x NVIDIA RTX 4090

Results were generated using the CUDA backend and are sorted by Kullback-Leibler divergence relative to FP16. The "WT" importance matrices were created using varying numbers of Wikitext tokens and can be found here. Note: the FP16 logits used for the calculation of all metrics other than perplexity are stored in a binary file between runs. In order to save space this file does not contain the exact same FP32 logits but instead casts them to 16 bit unsigned integers (with some scaling). So the "f16" results are to be understood as the difference resulting only from this downcast.

Quantization imatrix Model size [GiB] PPL ΔPPL KLD Mean Δp RMS Δp
f16 None 14.97 6.233160 ± 0.037828 0.001524 ± 0.000755 0.000551 ± 0.000002 0.001 ± 0.002 % 0.787 ± 0.004 %
q8_0 None 7.96 6.234284 ± 0.037878 0.002650 ± 0.001006 0.001355 ± 0.000006 -0.019 ± 0.003 % 1.198 ± 0.007 %
q6_K None 6.14 6.253382 ± 0.038078 0.021748 ± 0.001852 0.005452 ± 0.000035 -0.007 ± 0.006 % 2.295 ± 0.019 %
q5_K_M None 5.33 6.288607 ± 0.038338 0.056974 ± 0.002598 0.010762 ± 0.000079 -0.114 ± 0.008 % 3.160 ± 0.031 %
q5_K_S None 5.21 6.336598 ± 0.038755 0.104964 ± 0.003331 0.016595 ± 0.000122 -0.223 ± 0.010 % 3.918 ± 0.036 %
q5_1 None 5.65 6.337857 ± 0.038677 0.106223 ± 0.003476 0.018045 ± 0.000139 -0.287 ± 0.011 % 4.123 ± 0.039 %
q5_0 None 5.21 6.363224 ± 0.038861 0.131591 ± 0.003894 0.022239 ± 0.000166 -0.416 ± 0.012 % 4.634 ± 0.043 %
q4_K_M WT 10m 4.58 6.382937 ± 0.039055 0.151303 ± 0.004429 0.028152 ± 0.000240 -0.389 ± 0.014 % 5.251 ± 0.049 %
q4_K_M None 4.58 6.407115 ± 0.039119 0.175482 ± 0.004620 0.031273 ± 0.000238 -0.596 ± 0.014 % 5.519 ± 0.050 %
q4_K_S WT 10m 4.37 6.409697 ± 0.039189 0.178064 ± 0.004744 0.031951 ± 0.000259 -0.531 ± 0.015 % 5.645 ± 0.051 %
iq4_NL WT 10m 4.35 6.455593 ± 0.039630 0.223959 ± 0.005201 0.035742 ± 0.000288 -0.590 ± 0.016 % 5.998 ± 0.054 %
iq4_XS WT 10m 4.14 6.459705 ± 0.039595 0.228071 ± 0.005207 0.036334 ± 0.000284 -0.668 ± 0.016 % 6.044 ± 0.054 %
q4_K_S None 4.37 6.500529 ± 0.039778 0.268895 ± 0.005638 0.043136 ± 0.000314 -0.927 ± 0.017 % 6.562 ± 0.055 %
q4_1 None 4.78 6.682737 ± 0.041285 0.451103 ± 0.008030 0.071683 ± 0.000505 -0.927 ± 0.017 % 8.512 ± 0.063 %
q4_0 None 4.34 6.700147 ± 0.041226 0.468514 ± 0.007951 0.071940 ± 0.000491 -1.588 ± 0.022 % 8.434 ± 0.061 %
q3_K_L WT 10m 4.03 6.671223 ± 0.041427 0.439590 ± 0.008154 0.073077 ± 0.000529 -0.940 ± 0.023 % 8.662 ± 0.064 %
q3_K_M WT 10m 3.74 6.734255 ± 0.041838 0.502622 ± 0.008901 0.084358 ± 0.000588 -1.198 ± 0.024 % 9.292 ± 0.065 %
q3_K_L None 4.03 6.787876 ± 0.042104 0.556242 ± 0.009171 0.087176 ± 0.000614 -1.532 ± 0.025 % 9.432 ± 0.067 %
q3_K_M None 3.74 6.888498 ± 0.042669 0.656864 ± 0.010071 0.101913 ± 0.000677 -1.990 ± 0.026 % 10.203 ± 0.068 %
iq3_M WT 10m 3.53 6.898327 ± 0.041643 0.666694 ± 0.009449 0.102534 ± 0.000663 -3.178 ± 0.026 % 10.513 ± 0.066 %
iq3_S WT 10m 3.42 6.965501 ± 0.042406 0.733867 ± 0.010245 0.111278 ± 0.000710 -3.066 ± 0.027 % 10.845 ± 0.068 %
iq3_XS WT 10m 3.28 7.163043 ± 0.043772 0.931409 ± 0.012084 0.138693 ± 0.000857 -3.667 ± 0.031 % 12.148 ± 0.070 %
iq3_XXS WT 10m 3.05 7.458436 ± 0.046404 1.226803 ± 0.015234 0.183625 ± 0.001042 -3.918 ± 0.035 % 13.836 ± 0.074 %
q3_K_S WT 10m 3.41 7.602878 ± 0.046848 1.371244 ± 0.015688 0.199821 ± 0.001008 -5.046 ± 0.037 % 14.980 ± 0.070 %
q3_K_S None 3.41 7.863786 ± 0.048885 1.632152 ± 0.017733 0.228217 ± 0.001079 -5.604 ± 0.038 % 15.541 ± 0.070 %
iq2_M WT 10m 2.74 8.600799 ± 0.055124 2.369166 ± 0.025244 0.325989 ± 0.00160 -6.463 ± 0.046 % 18.519 ± 0.080 %
q2_K WT 10k 2.96 8.652290 ± 0.055572 2.420657 ± 0.025587 0.331393 ± 0.001562 -6.606 ± 0.046 % 18.790 ± 0.078 %
q2_K WT 100k 2.96 8.641993 ± 0.055406 2.410359 ± 0.025495 0.331672 ± 0.001569 -6.628 ± 0.047 % 18.856 ± 0.078 %
q2_K WT 10m 2.96 8.647825 ± 0.055610 2.416191 ± 0.025683 0.332223 ± 0.001572 -6.500 ± 0.047 % 18.881 ± 0.078 %
q2_K WT 1m 2.96 8.674365 ± 0.055743 2.442732 ± 0.025843 0.335308 ± 0.001576 -6.634 ± 0.047 % 19.009 ± 0.079 %
q2_K WT 1k 2.96 8.682605 ± 0.055916 2.450972 ± 0.026069 0.337093 ± 0.001596 -6.596 ± 0.047 % 18.977 ± 0.079 %
q2_K_S WT 10m 2.96 9.323778 ± 0.061551 3.092145 ± 0.031914 0.403360 ± 0.001787 -7.131 ± 0.049 % 20.050 ± 0.081 %
q2_K_S WT 1m 2.96 9.329321 ± 0.061378 3.097688 ± 0.031816 0.403590 ± 0.001797 -7.289 ± 0.049 % 20.123 ± 0.081 %
q2_K_S WT 100k 2.96 9.362973 ± 0.061740 3.131339 ± 0.032169 0.408367 ± 0.001802 -7.198 ± 0.050 % 20.132 ± 0.081 %
q2_K_S WT 10k 2.96 9.376479 ± 0.062045 3.144846 ± 0.032464 0.408662 ± 0.001819 -7.141 ± 0.050 % 20.120 ± 0.081 %
q2_K_S WT 1k 2.96 9.415200 ± 0.062475 3.183567 ± 0.032993 0.415865 ± 0.001846 -7.153 ± 0.050 % 20.311 ± 0.082 %
iq2_S WT 10m 2.56 9.650781 ± 0.063209 3.419148 ± 0.034017 0.439197 ± 0.001976 -8.319 ± 0.052 % 21.491 ± 0.083 %
q2_K None 2.96 9.751568 ± 0.063312 3.519934 ± 0.033863 0.445132 ± 0.001835 -9.123 ± 0.051 % 21.421 ± 0.079 %
iq2_XS WT 10m 2.43 10.761424 ± 0.071056 4.529791 ± 0.042229 0.546290 ± 0.002133 -10.576 ± 0.056 % 23.872 ± 0.082 %
iq2_XXS WT 10m 2.24 14.091782 ± 0.098396 7.860148 ± 0.070752 0.812022 ± 0.002741 -14.363 ± 0.065 % 28.576 ± 0.084 %
iq1_M WT 10m 2.01 25.493722 ± 0.177903 19.262089 ± 0.152396 1.393084 ± 0.003529 -24.672 ± 0.077 % 38.287 ± 0.084 %
iq1_S WT 1m 1.88 58.097760 ± 0.438604 51.866126 ± 0.416604 2.211278 ± 0.004688 -32.471 ± 0.087 % 46.418 ± 0.085 %
iq1_S WT 1k 1.88 58.267851 ± 0.446208 52.036218 ± 0.424373 2.214858 ± 0.004778 -31.880 ± 0.089 % 46.330 ± 0.086 %
iq1_S WT 100k 1.88 58.581498 ± 0.453145 52.349864 ± 0.431360 2.220834 ± 0.004818 -32.261 ± 0.089 % 46.002 ± 0.086 %
iq1_S WT 10m 1.88 60.694593 ± 0.471290 54.462959 ± 0.449644 2.254554 ± 0.004868 -31.973 ± 0.088 % 46.271 ± 0.086 %
iq1_S WT 10k 1.88 63.221324 ± 0.493077 56.989691 ± 0.471423 2.293527 ± 0.004885 -32.261 ± 0.089 % 46.562 ± 0.086 %

There seems to be no consistent improvement from using more Wikitext tokens for the importance matrix. K-quants score better on mean Δp than the legacy quants than e.g. KL divergence would suggest.

LLaMA 2 vs. LLaMA 3 Quantization comparison

Revision f364eb6f
Backend CUDA
CPU AMD Epyc 7742
GPU 1x NVIDIA RTX 4090
Metric L2 7b q2_K L3 8b q2_K L2 7b q4_K_M L3 8b q4_K_M L2 7b q6_K L3 8b q6_K L2 7b q8_0 L3 8b q8_0
Mean PPL 5.794552 ± 0.032298 9.751568 ± 0.063312 5.877078 ± 0.032781 6.407115 ± 0.039119 5.808494 ± 0.032425 6.253382 ± 0.038078 5.798542 ± 0.032366 6.234284 ± 0.037878
Mean PPL ratio 1.107955 ± 0.001427 1.564849 ± 0.004525 1.014242 ± 0.000432 1.028160 ± 0.000723 1.002406 ± 0.000191 1.003490 ± 0.000296 1.000689 ± 0.000107 1.000425 ± 0.000161
Mean ΔPPL 0.625552 ± 0.008725 3.519934 ± 0.033863 0.082526 ± 0.002530 0.175482 ± 0.004620 0.013941 ± 0.001110 0.021748 ± 0.001852 0.003990 ± 0.000624 0.002650 ± 0.001006
PPL correlation 97.36% 89.62% 99.71% 99.34% 99.94% 99.88% 99.98% 99.96%
Mean KLD 0.108903 ± 0.000645 0.445132 ± 0.001835 0.012686 ± 0.000079 0.031273 ± 0.000238 0.002098 ± 0.000014 0.005452 ± 0.000035 0.000369 ± 0.000007 0.001355 ± 0.000006
Mean Δp -2.710 ± 0.023 % -9.123 ± 0.051 % -0.416 ± 0.008 % -0.596 ± 0.014 % -0.035 ± 0.003 % -0.007 ± 0.006 % -0.005 ± 0.002 % -0.019 ± 0.003 %
Maximum Δp 85.136% 94.268% 45.209% 95.054% 23.593% 53.601% 43.925% 28.734%
99.9% Δp 37.184% 50.003% 17.461% 27.084% 7.798% 13.613% 3.387% 6.402%
99.0% Δp 18.131% 25.875% 7.798% 12.084% 3.838% 6.407% 1.867% 3.544%
Median Δp -0.391% -2.476% -0.026% -0.024% -0.001% 0.000% -0.000% -0.000%
1.0% Δp -39.762% -87.173% -11.433% -19.567% -4.222% -6.767% -1.862% -3.698%
0.1% Δp -79.002% -98.897% -26.433% -56.054% -9.091% -16.584% -3.252% -6.579%
Minimum Δp -99.915% -99.965% -83.383% -98.699% -43.142% -68.487% -9.343% -24.301%
RMS Δp 9.762 ± 0.053 % 21.421 ± 0.079 % 3.252 ± 0.024 % 5.519 ± 0.050 % 1.339 ± 0.010 % 2.295 ± 0.019 % 0.618 ± 0.011 % 1.198 ± 0.007 %
Same top p 85.584 ± 0.086 % 71.138 ± 0.119 % 94.665 ± 0.055 % 91.901 ± 0.072 % 97.520 ± 0.038 % 96.031 ± 0.051 % 98.846 ± 0.026 % 97.674 ± 0.040 %

LLaMA 3 BF16 vs. FP16 comparison

Revision 83330d8c
Backend CPU
CPU AMD Epyc 7742
GPU N/A

Results were calculated with LLaMA 3 8b BF16 as --kl-divergence-base and LLaMA 3 8b FP16 as the --model for comparison.

Metric Value
Mean PPL(Q) 6.227711 ± 0.037833
Mean PPL(base) 6.225194 ± 0.037771
Cor(ln(PPL(Q)), ln(PPL(base))) 99.990%
Mean ln(PPL(Q)/PPL(base)) 0.000404 ± 0.000086
Mean PPL(Q)/PPL(base) 1.000404 ± 0.000086
Mean PPL(Q)-PPL(base) 0.002517 ± 0.000536
Mean KLD 0.00002515 ± 0.00000020
Maximum KLD 0.012206
99.9% KLD 0.000799
99.0% KLD 0.000222
99.0% KLD 0.000222
Median KLD 0.000013
10.0% KLD -0.000002
5.0% KLD -0.000008
1.0% KLD -0.000023
Minimum KLD -0.000059
Mean Δp -0.0000745 ± 0.0003952 %
Maximum Δp 4.186%
99.9% Δp 1.049%
99.0% Δp 0.439%
95.0% Δp 0.207%
90.0% Δp 0.125%
75.0% Δp 0.029%
Median Δp 0.000%
25.0% Δp -0.030%
10.0% Δp -0.126%
5.0% Δp -0.207%
1.0% Δp -0.434%
0.1% Δp -1.016%
Minimum Δp -4.672%
RMS Δp 0.150 ± 0.001 %
Same top p 99.739 ± 0.013 %

Old Numbers

Llama 2 70B Scoreboard
Quantization Model size (GiB) Perplexity Delta to fp16
Q4_0 36.20 3.5550 3.61%
Q4_1 40.20 3.5125 2.37%
Q5_0 44.20 3.4744 1.26%
Q2_K 27.27 3.7339 8.82%
Q3_K_S 27.86 3.7019 7.89%
Q3_K_M 30.83 3.5932 4.72%
Q3_K_L 33.67 3.5617 3.80%
Q4_K_S 36.39 3.4852 1.57%
Q4_K_M 38.54 3.4725 1.20%
Q5_K_S 44.20 3.4483 0.50%
Q5_K_M 45.41 3.4451 0.40%
Q6_K 52.70 3.4367 0.16%
fp16 128.5 3.4313 -