Replies: 14 comments 34 replies
-
|
A wider view of progress over time using the HF leaderboard data. I curled the Conclusion: Continuous and continuing progress upwards with the 7B models improving twice as fast as 65B, likely due to compute availability. |
Beta Was this translation helpful? Give feedback.
-
|
did you combine llama v1 7B with v2 7B ? |
Beta Was this translation helpful? Give feedback.
-
|
If you take the log(n_params) and compare it to the hellaswag scores, you can get a picture of how much models of different sizes can actually learn common-sense reasoning. Calculating the linear regression at different points in time, we could possibly use the function constants to plot a general chart of how the hellaswag capacity of LLMs changes over time.
|
Beta Was this translation helpful? Give feedback.
-
Yes. The plot was intended to provide a six month view of the consistent "small-model" incremental LLM progress as a function of model size. Small, specialist models can now outperform generalist models for specific use cases, e.g. coding, math, creative writing, etc. These models can be iterated and improved on in weeks, and deployed easily using inference frameworks like llama.cpp. Compare this to Bard, GPT, etc. where progress has tended to be discontinuous, i.e. prior to this year, every year or so, a large tech company has moved SOTA forward with typically some innovation and a lot of scaling - at the cost of 10's of millions of dollars of compute. The recently leaked GPT-4 architecture and compute costs indicate that scaling is probably not the way forward. GPT-4 is a somewhat larger GPT-3, and then scaled horizontally by using a Mixture of Experts (MoE), while still costing significantly more in compute to train and run. If MoE is the "way forward" (unlikely), we can conceivably run a MoE of specialist smaller models. |
Beta Was this translation helpful? Give feedback.
-
|
The scores of the LLaMA models on the Open LLM Leadboard have been recalculated (several times) due to tokenizer problems. EleutherAI/lm-evaluation-harness#531 (comment) Problems with the tokenizer may be one of the reasons why the scores in our HellaSwag test is lower than the Leaderboard scores. The other reason is that we are currently doing 0-shot tests and not 10-shot. There is some discussion about this here #2389. |
Beta Was this translation helpful? Give feedback.
-
Yes, I seem to remember this was a significant problem with the OpenLLM family of base models. Of course the HL benchmark inconsistency has educated people on how important the tokenizer is in terms of model output quality. I'm plotting the HL average across all the EleutherAI benchmarks and hopefully by looking at 400 or so models a lot of the noise from problematic tokenization and other issues is averaged out. What will be interesting is comparing scores in another 6 months time and seeing if the trends for each model size continue improving. |
Beta Was this translation helpful? Give feedback.
-
|
The HellSwag calculation is broken on current master: I get a score of 71.00 after 400 tasks for LLaMA-v2-7B (see attached output). The change from the score of 77.5 we find in the table above occurred after PR 6381d4e. I have checked that the result is also different for LLaMA-v1-7B (68.50 vs 75.25 after 400 tasks). |
Beta Was this translation helpful? Give feedback.
-
|
Is the perplexity on the model you used different? |
Beta Was this translation helpful? Give feedback.
-
|
All my results in the table above was done using cublas without offloading before the custom cuda mat mul was made default in PR #2683. Test disabling the mmq by using the |
Beta Was this translation helpful? Give feedback.
-
|
@klosax When someone says that commit |
Beta Was this translation helpful? Give feedback.
-
The GGUF commit seems like a bit of a special case since it involved a lot of changes from different authors and also administrative decisions like how tokenizing is performed, vocabulary stored, etc. I think it would be really hard to solve a regression in this specific case by going through the changes and trying to read them. Also, can't blame me in this case. :) I'm an "author" of the patch technically but all I did was contribute a stand-alone script for converting GGML files to GGUF and I doubt you're testing with a model converted using that. Something related to the tokenizing/vocab changes seems like it's probably most likely to be causing this. If I was debugging it, I'd probably try to run hellaswag on a specific set of test cases and dump exactly the tokens are getting sent to the model to prompt it. It should be possible to confirm or rule out tokenizing stuff that way (or other similar issues like BOS getting included or not). |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I must be the one who is confused by the commit numbers and the problems with mmq in Falcon. Anyway,. the GGUF commit was massive and one thing that comes to my mind that could be related to the current HellaSwag scores is the tokenizer fixes from PR #2549. |
Beta Was this translation helpful? Give feedback.
-
|
Let me try again. |
Beta Was this translation helpful? Give feedback.
-
Yes, and there were major tokenizer changes in 6381d4e and the way vocabulary gets stored/processed for the model files also changed a lot. That's what I was talking about. If you can compare the token ids that get sent to the model before and after the GGUF patch you can either confirm or rule out those changes as causing this effect. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, in commit 6381d4e (PR #2398 aka |
Beta Was this translation helpful? Give feedback.
-
Excellent! This gives me the opportunity to rant against the policy of sqaush-merging massive changes to the code base, with 6381d4e being a great example why that is not good. Do I want to have the 200+ commits that lead from dadbed9 to 6381d4e in the history? Definitely not. Would it have been better to restructure the 200+ commit in #2398 into |
Beta Was this translation helpful? Give feedback.
-
|
It's still possible to see the individual unsquashed commits that made up the pull: https://github.com/ggerganov/llama.cpp/pull/2398/commits
That's certainly true, but it's way easier said than done. There are a large number of commits that are interdependent, and even depend on specific versions of other commits. Maybe it would be possible to go through everything and bundle sets of changes into logic chunks but it would be both difficult and time consuming. No one volunteered to do that, and let's face it most people really don't find that kind tedious administrative task very fun. As open source contributes, we mostly contribute the stuff that we enjoy working on or that scratches an itch, etc. Just coordinating the changes in the GGUF pull was a pretty big task (and the transition was surprisingly smooth). Should we say Mr. GG should have just buckled down and put in the work to try to organize the pull better once it reached a mergeable state? Or if not him, who should have been assigned to do it? edit: Just want to add, I don't mean this as a rebuke or criticism. I'm also not saying it's unreasonable to be frustrated in this situation, it's definitely not ideal. My point is only that there isn't really a simple/easy solution to the problem. |
Beta Was this translation helpful? Give feedback.
-
|
So, the HellaSwag score change after #2398 is due to this line: Line 3030 in bae5c5f If I comment it out, I recover pre-GGUF scores. If I replace it with I get a score very close to pre-GGUF but not quite the same (see #2805) |
Beta Was this translation helpful? Give feedback.
-
|
Here are some HellaSwag 10-shot scores for LLaMA-v2-7B. The test cases along with the 10-shot context added that I have used are attached (see at the end of the post). I tried a few variations and this version gave the highest score for the The dataset consists of 5001 tasks. These are the tasks from activities that have training data in the HellaSwag training dataset, and the context added before each task was taken from there. I verified that for 0-shot evaluation with this 5001-task subset I get the exact same 0-shot HellaSwag score of The following table shows the difference in HellaSwag score for different quantization types. If you want to have the actual HellaSwag score for a given quantization, just use
Here is a plot of the data in the above table. The x-axis is
|
Beta Was this translation helpful? Give feedback.
-
How did you choose the tasks? The tasks should be fully randomized and not hand picked for a specific model. Does this 10-shot test work equally well on other models like Falcon? IMO the training data should not be used at all when testing. To measure how good the model generalizes outside of training, only the validation dataset should be used as a test target. The OpenLLM 10-shot are highly linearly correlated to the 400 task 0-shot. The 0-shot being about 1.5 lower then the 10-shot.
The strong correlation is only valid for models trained on the exact same dataset. LLaMA-7b perplexity is much lower than Falcon-7b on the |
Beta Was this translation helpful? Give feedback.
-
As explained, those 5001 tasks out of the 10042 tasks in the HellaSwag validation dataset were taken where training data of the given activity were available. The following activities did not have training examples: Because I expected that someone may object to selecting these 5001 tasks, I specifically pointed out that the 0-shot HellaSwag score for these 5001 tasks is exactly the same as the 0-shot HellaSwag score of the excluded 5041 tasks. I.e., I did not add any kind of bias into the evaluation by picking these 5001 tasks.
Have not tried. The calculations with the full data set take a very long time. Evaluations with just 400 tasks are basically useless as statistical uncertainty is in the range of
And how do people do 10-shot HellaSwag in your opinion? The 10 examples added to the context along with the actual task must come from somewhere. In general, as long as the data being used for prediction on a given validation/test dataset does not contain traces of that validation/test dataset, one can do whatever one likes with the training data. One can wipe the floor with it, cook coffee with it, paint the walls with it, you name it. And, among all the things one can do with the training data, one can add it as a context when predicting the continuation of a sentence in the HellaSwag test. Saying that one cannot use the training data when running inference on a dataset that did not participate in training is equivalent to saying that we can not use inference to test a model because, you know, the training data is encoded and present in some form in the model weights. If still in doubt, I can give you an example from $work that is completely unrelated to LLMs. The task is to generate a CT image from a given MR image of a patient. One of the algorithms we use is based on deformable image registration. Basically, you have a training dataset consisting of registered MR-CT image pairs of a set of patients. To generate the CT image of a new patient, one picks the
Yes, I saw the plot, and yes, I see there is correlation. But I also see in the very same plot that two models with the same 0-shot-400-tasks score can differ by 5+ points from their respective OpenLLM 10-shot score (and from the linear fit of the data). 5 HellaSwag points error is just too big to disambiguate between modern models, so not really useful for anything but distinguishing a modern model from GPT-2.
Sure. But in the table above I see columns for |
Beta Was this translation helpful? Give feedback.
-
I do not know how it is supposed to work. That is why I settled on doing 0-shot only using the validation dataset. Maybe you are right that > 0 shot is supposed to be using training data to make the model better understand the activity/context before predicting the endings.
Perhaps the problem here is also the statistical uncertainty when evaluated using only 400 tasks. The only variable that correlates well with HellaSwag over different models and sizes seems to be the (log of) the number of model parameters. |
Beta Was this translation helpful? Give feedback.
-
|
The datafile for 10-shot contain I guess tokenizers should not resolve the special tokens by default, but I think we should have that option to get things like this to work properly. |
Beta Was this translation helpful? Give feedback.
-
|
Tbh, I don't understand this test. The test ran for 400 tasks. Edit: It looks something like this: |
Beta Was this translation helpful? Give feedback.
-
|
In each task you have a sentence plus the beginning of the next sentence. There are 4 possible continuations given, one of the 4 is the expected continuation. The algorithm evaluates the probability of the 4 continuations and picks the one with the highest probability. If the selected continuation is the expected one, +1 is added to the score. After evaluating |
Beta Was this translation helpful? Give feedback.
-
|
I see. Thank you. |
Beta Was this translation helpful? Give feedback.
-
I ran Falcon-7b now. I get 10-shot HellaSwag = 77.904 after 5001 tasks, which is almost as good as the score of 78.13 posted on the OpenLLM leader board. So, I guess, we could say that it works for Falcon as well as, if not better than, LLaMA models. Interestingly enough, in 0-shot evaluation Falcon-7b beats LLaMA-v2-7b: 76.97 vs 75.8 after 10042 tasks. I used a different 10-shot data file, which gets 10-shot HellaSwag = 78.24 for LLaMA-v2-7B (so very slightly better than the 78.16 I had before for LLaMA-2). I'm attaching it below. What is the difference between the two versions? In the original version that I used in my post above, I used the 10 longest examples from the training data that belong to the given activity. In this version, the training examples are selected randomly using a probability proportional to the example length squared. |
Beta Was this translation helpful? Give feedback.
-
|
The |
Beta Was this translation helpful? Give feedback.
-
|
But token id 1 in Falcon models is something different: {
"id": 1,
"content": ">>ABSTRACT<<",
"single_word": false,
"lstrip": false,
"rstrip": false,
"normalized": false,
"special": true
},Token id 11 is both BOS and EOS: {
"id": 11,
"content": "<|endoftext|>",
"single_word": false,
"lstrip": false,
"rstrip": false,
"normalized": false,
"special": true
}Pasted from https://huggingface.co/tiiuae/falcon-7b |
Beta Was this translation helpful? Give feedback.
-
|
Those should be saved by the conversion scripts to the gguf model file and be accessible in C (llama.cpp) by using the variables Falcon does not seem to have a differentiation of BOS and EOS like LLaMA models does. |
Beta Was this translation helpful? Give feedback.
-
|
If you convert using #2842 does it help? Assuming you're using a venv you'll need to |
Beta Was this translation helpful? Give feedback.
-
|
No the problem is in the llama.cpp tokenizer code. It will not tokenize the special tokens string values to the special token ids and I think it should not normally do that since |
Beta Was this translation helpful? Give feedback.
-
|
Input: add |
Beta Was this translation helpful? Give feedback.
-
|
The purpose of these |
Beta Was this translation helpful? Give feedback.
-
|
Does the The hellaswag implementation could parse the input and separate the tokenization and correctly insert BOS or EOS when |
Beta Was this translation helpful? Give feedback.
-
|
Running 10-shot HellaSwag for Falcon-7b with 5000 tasks on an RTX-4080 takes 1.5 hours. So, I think, I'll let someone else to sort out if there are better example separators for Falcon. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry to bring this up again. Why my hellaswag score so low? for example, both 400 randomized tasks and full 10042 tasks give me 50 ish score using MIstral-7B-v0.1, which is supposed to be in the 80 ish. |
Beta Was this translation helpful? Give feedback.
-
|
So, if memory serves me correctly, there were changes to the tokenizer since I last used this tool. Again, if I remember correctly, one of the changes was to automatically add a bos token in |
Beta Was this translation helpful? Give feedback.
-
|
From the logs, it seems |
Beta Was this translation helpful? Give feedback.
-
|
Looking at it, there are quite a few changes since my last commit on Aug 26. There were also the KV handling related changes, this could also be causing it. |
Beta Was this translation helpful? Give feedback.
-
|
Check if #4981 helps - we forgot to clear the KV cache when we switched to the new API |
Beta Was this translation helpful? Give feedback.
-
|
OK, it is KV cache related. I submitted a PR that fixes it (#4982) |
Beta Was this translation helpful? Give feedback.
-
The HellaSwag test
The test is done by using examples of sentences that makes sense and others which do not make sense. The sentences are presented to the model in groups (tasks) of four with the same start but different endings. One of the endings in each group is labeled as the one that makes most sense. The probability of the model to predict the different endings are computed and if the labeled ending got the highest probability it it considered a correct prediction. The resulting accuracy score is the percent of groups with correct predictions.
More info can be found in the paper https://arxiv.org/abs/1905.07830
In order to compute the HellaSwag scores you need to download the datafile here: klosax/hellaswag_text_data.
Simple usage that runs the test on 400 random tasks in the datafile:
./perplexity --hellaswag -f hellaswag_val_full.txt -m modelfile.ggufFor a more accurate test you can specify the number of tasks to use in the computation:
./perplexity --hellaswag --hellaswag-tasks N -f hellaswag_val_full.txt -m modelfile.ggufThe table below is showing the score (0-shot acc_norm) for various models using 400 randomized HellaSwag tasks. The numbers in the "Leaderboard" column are taken from HuggingFace Open LLM Leaderboard. Note that the Leaderboard scores are about 1.5 higher because they are 10-shot. The "Params" column is the total number of elements in the model file and "Train tokens" are the total number of tokens the model was trained on.
The second part of the table contains models not yet supported in llama.cpp, but support may be added in the future.
The HellaSwag scores are correlated to the number of model parameters:
The 400 task 0-shot HellaSwag scores are highly correlated to the OpenLLM Leaderboard 10-shot HellaSwag scores:
(This post will be updated)
Beta Was this translation helpful? Give feedback.
All reactions