Potential degredation in sampling/too repetitive

[syskn]

Moving from Discussions #471

Hi guys, great work!

I have been experimenting with the library for several weeks, and immediately noticed that sampled tokens (with the same temperature and such) are significantly more deterministic with Vllm vs. HF Transformers using the same models - with temperature lower than 0.7, often the first 5-10 sampled tokens are exactly same across few different generations, even recreating the original text in the datasets verbatim, like there is some greedy decoding going on (when it is not). This unfortunately leads to a significant repetition issue I've never seen with HF.

The issue is not related to special tokens such as <s></s>. I modified vllm so that it never generates those special tokens like HF's bad_words_ids, but the issue persists (those special tokens will also make the inference quality significantly worse especially with non-chat prompts, but it is a different issue).

In the mean time, I have also been checking and modifying codebase to see if there is any discrepancy between the sampling process, but I am not sure about the difference. I'm suspecting either cuda kernels or partitioning/block in PagedAttention.

#590: Related topic with actual examples. The topic is about GPTJ, but the issue is there with other architectures such as Llama and NeoX.

[BaiMoHan]

Yes.If my memory serves right, this problem arose when I was using version 0.1.1.

[luohao123]

I got exactly same issue here!

the speed are faster, but the performance dropped a lot

the final output are really not like hf even genering params are same!!

[ehartford]

I had the same experience. I set up a client with VLLM and they saw totally different (and very disappointing) output from VLLM vs the original huggingface code, using the exact same parameters - temperature, repetition penalty, etc.

[Sanster]

I had the same experience. I set up a client with VLLM and they saw totally different (and very disappointing) output from VLLM vs the original huggingface code, using the exact same parameters - temperature, repetition penalty, etc.

Hi, I noticed that the default value of repetition_penalty in HF is 1.0, but in vllm the default value of presence_penalty is 0. So should we set presence_penalty = repetition_penalty - 1.0? Thanks

[syskn]

I had the same experience. I set up a client with VLLM and they saw totally different (and very disappointing) output from VLLM vs the original huggingface code, using the exact same parameters - temperature, repetition penalty, etc.

Hi, I noticed that the default value of repetition_penalty in HF is 1.0, but in vllm the default value of presence_penalty is 0. So should we set presence_penalty = repetition_penalty - 1.0? Thanks

This should not be the case here. HF-style repetition penalty works like this (it's a bit strange implementation for historical reason):
value == 1.0 bypass
value > 1.0 penaltize repetition
value < 1.0 "promote" repetition

Whereas OpenAI-style frequency penalty and presence penalty are:
value == 0 bypass
value > 0 penaltize repetition
value < 0 "promote" repetition

[Sanster]

I had the same experience. I set up a client with VLLM and they saw totally different (and very disappointing) output from VLLM vs the original huggingface code, using the exact same parameters - temperature, repetition penalty, etc.

Hi, I noticed that the default value of repetition_penalty in HF is 1.0, but in vllm the default value of presence_penalty is 0. So should we set presence_penalty = repetition_penalty - 1.0? Thanks

This should not be the case here. HF-style repetition penalty works like this (it's a bit strange implementation for historical reason): value == 1.0 bypass value > 1.0 penaltize repetition value < 1.0 "promote" repetition

Whereas OpenAI-style frequency penalty and presence penalty are: value == 0 bypass value > 0 penaltize repetition value < 0 "promote" repetition

Yes, I also encountered the issue of sampling. In my testing, I used HF-style repetition_penalty. However, even after changing presence_penalty to repetition_penalty-1, the problem still persists.

[syskn]

Actually, there is also a difference between how each repetition penalties are implemented (HF is exponential, OpenAI/vLLM is linear, presence penalty is one-off). However, this is not directly related to the potential degradation issue from what I understand.

[Linzecong]

I got exactly same issue here!

the speed are faster, but the performance dropped a lot

the final output are really not like hf even genering params are same!!

Subjectively feel that the HF generation will be better. I'm trying to adjust various parameters to align at low temperature.

[luohao123]

On my side, I didn't care about which params, I just want aligned with hf, since hf result are good, but vllm worse..

[kechunFIVE]

On my side, I didn't care about which params, I just want aligned with hf, since hf result are good, but vllm worse..

vllm also seems to be getting worse in my experiments, but your avatar is so cool.

[Sanster]

Share some of my experimental results.

LongEval result

Here is the result of my running the longeval Fine-grained Line Retrieval task, using same model and same parameters.

• temperature: 0.01
• top_p: 0.95
• vllm max_num_batched_tokens: 4096

Ceval

However, the same model, tested on the ceval task using HF and VLLM, did not show a significant difference.

[syskn]

Sanster,

Wow. Just to be sure, did you merge RoPE interpolation support PR (either #555 or #690 respectively) which is not yet added to the latest release?

AFAIK, if a rope-encoded model is 2048 ctx length, eval starts to significantly degrade from around 2400 tokens without interpolation methods.

[creatorrr]

We are also seeing the same effect for longer sequences. What could be the issue @zhuohan123 ?