[Performance]: The performance of bge-rerank model on vllm and huggingface is inconsistent

[yumingmin88]

Proposal to improve performance

No response

Report of performance regression

No response

Misc discussion on performance

The script loaded using vllm is:

from vllm import LLM
model_path = "/data/reranker/bge-reranker-large"
# Create an LLM.
# You should pass task="score" for cross-encoder models
model = LLM(
    model=model_path,
    task="score",
    enforce_eager=True,
    dtype="float32"
)

huggingface scripts mainly use default parameters for，float32

 CrossEncoder(
            model, max_length=512, device=device
        )

After running 200 sets of data, in which 100 paragraphs were rearranged, 19 sets were found to be in inconsistent order after rearrangement
I also looked up some information online and there is the following information：
https://zhuanlan.zhihu.com/p/658780653

I do not know whether the parameter of my vllm is wrong, or what causes it?

Your current environment (if you think it is necessary)

Current environment
linux: ubuntu 18
NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 Tesla T4
vllm==0.6.6

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Can you provide a minimum reproducible example where the outputs are inconsistent?

[cafeii]

hello @DarkLight1337 , it seems that different prompt inputs can result the different embedding results.
e.g.:

llm = LLM(
    model='/data1/model/llama-7b-base/',
    task='embed',
    enforce_eager=True
)

output1 = llm.embed(['hello world', 'hello world'])
output2 = llm.embed(['hello world'])

And the results of the two embeddings are like:

output1[0].outputs.embedding: 
[0.0108795166015625,
 -0.01033782958984375,
 0.0052490234375,
 -0.033599853515625,
 0.0019931793212890625,
 -0.005802154541015625,
 -0.0067291259765625,
 -0.00399017333984375,
...]
output2[0].outputs.embedding:
[0.0109100341796875,
 -0.01035308837890625,
 0.00524139404296875,
 -0.033599853515625,
 0.001972198486328125,
 -0.005802154541015625,
 -0.006740570068359375,
 -0.003993988037109375,
 -0.00960540771484375,
 -0.00354766845703125,
...]

The difference between emb1 and emb2 is typically small.

diff = np.array(emb1) - np.array(emb2)
np.mean(np.abs(diff)), np.std(diff)
(1.1552008800208569e-05, 1.6885600893505396e-05)

Do you know the possible reason that may lead to this difference? I think this may be the same reason of the inconsistence mentioned by @yumingmin88

[noooop]

I also found that reranker results will lead to different results because different tokens are batched

When verifying whether the model is consistent with hf, it is necessary to compare the results after softmax to ensure that the difference is within 0.01

test_bge-reranker-v2-m3

Of course, I think the model should handle this kind of dynamic batch.
If you don’t use dynamic batch inference, it will be too slow.

@yumingmin88

Try using softmax on the rerank model output and it to see if the order can be maintained.

[DarkLight1337]

hello @DarkLight1337 , it seems that different prompt inputs can result the different embedding results
Do you know the possible reason that may lead to this difference? I think this may be the same reason of the inconsistence mentioned by @yumingmin88

Please see #11526 (comment)

[yumingmin88]

Can you provide a minimum reproducible example where the outputs are inconsistent?

Hello, do you think such an example is OK? I was learning RAG, so I ran 200 sets of examples like this from the knowledge base, 19 sets with different candidates length and number of shards, and 181 sets with the same index and only a slight difference in score
Thank you for your reply
`
query, candidates = 'J2智能扫地机器人能吸得干净猫毛吗？',['请勿使用清洁勾刀、刷子等物品直接清理滤网)\n\n安装回干净滤网并盖上上盖板，完成滤网清洁和替换。\n(注意\n滤网水洗后需24小时自然晾干后才能使用，建议两块滤网交替使用)\n\n边刷\n定期清理边刷，可以保证机器人扫地清洁的清洁效率，避免机器人工作时边刷转动可能发出的异响。\n保养周期 \n建议两周清理一次毛发，三到六个月更换一次\n保养步骤 \n清理毛发缠绕\n\n清理扫地模块上的其他脏污。 (扫地模块可直接用水冲洗，水洗后晾千才可以继续使用)\n如需更换新的边刷，双手扯住刷毛往上提拉\n\n滚刷\n保养周期 \n建议两周清理一次，六个月更换一次\n保养步骤 \n取下滚刷壳\n\n取下滚刷\n\n取下两端堵头，清理两端毛发\n\n勾刀或剪刀清理中间缠绕\n\n装回滚刷和壳体\n\n万向轮\n', ' Q:边刷炸毛严重\nA:机器人在地毯等材质上面清洁，可能导致出现容易炸毛的情况。边刷炸毛对扫地功能、清洁效果影响不大，可定期更换边刷使用\n', ' Q:边刷炸毛严重\nA:机器人在地毯等材质上面清洁，可能导致出现容易炸毛的情况。边刷炸毛对扫地功能、清洁效果影响不大，可定期更换边刷使用。\n']

from sentence_transformers import CrossEncoder
model_path = "/data/reranker/bge-reranker-large"
model = CrossEncoder(model_path, max_length=512)

pairs = [[query, candidate] for candidate in candidates]
scores = model.predict(pairs)

ranked_candidates = [(idx, score) for idx, score in enumerate(scores)]
ranked_candidates.sort(key=lambda x: x[1], reverse=True)
print("Ranked Candidates:")
for idx, score in ranked_candidates:
print(f"index:{idx} socre:{score}")

from vllm import LLM
model_path = "/data/reranker/bge-reranker-large"
model = LLM(
model=model_path,
task="score",
enforce_eager=True,
dtype="float32"
)
outputs = model.score(query, candidates, truncate_prompt_tokens=512)
outputs_list = sorted([(idx, output.outputs.score) for idx, output in enumerate(outputs)], key= lambda x: -x[1])
for idx, score in outputs_list:
print(f"index:{idx} socre:{score}")

`

[yumingmin88]

I also found that reranker results will lead to different results because different tokens are batched

When verifying whether the model is consistent with hf, it is necessary to compare the results after softmax to ensure that the difference is within 0.01

test_bge-reranker-v2-m3

Of course, I think the model should handle this kind of dynamic batch. If you don’t use dynamic batch inference, it will be too slow.

@yumingmin88

Try using softmax on the rerank model output and it to see if the order can be maintained.

Thank you very much for your reply. I will try it like this

[yumingmin88]

你好@DarkLight1337，似乎不同的提示输入可能会导致不同的嵌入结果。 例如：

llm = LLM(
    model='/data1/model/llama-7b-base/',
    task='embed',
    enforce_eager=True
)

output1 = llm.embed(['hello world', 'hello world'])
output2 = llm.embed(['hello world'])

两次嵌入的结果如下：

output1[0].outputs.embedding: 
[0.0108795166015625,
 -0.01033782958984375,
 0.0052490234375,
 -0.033599853515625,
 0.0019931793212890625,
 -0.005802154541015625,
 -0.0067291259765625,
 -0.00399017333984375,
...]
output2[0].outputs.embedding:
[0.0109100341796875,
 -0.01035308837890625,
 0.00524139404296875,
 -0.033599853515625,
 0.001972198486328125,
 -0.005802154541015625,
 -0.006740570068359375,
 -0.003993988037109375,
 -0.00960540771484375,
 -0.00354766845703125,
...]

emb1 和 emb2 之间的差异通常很小。

diff = np.array(emb1) - np.array(emb2)
np.mean(np.abs(diff)), np.std(diff)
(1.1552008800208569e-05, 1.6885600893505396e-05)

你知道导致这种差异的可能原因是什么吗？我认为这可能是@yumingmin88

你好@DarkLight1337，似乎不同的提示输入可能会导致不同的嵌入结果。 例如：

llm = LLM(
    model='/data1/model/llama-7b-base/',
    task='embed',
    enforce_eager=True
)

output1 = llm.embed(['hello world', 'hello world'])
output2 = llm.embed(['hello world'])

两次嵌入的结果如下：

output1[0].outputs.embedding: 
[0.0108795166015625,
 -0.01033782958984375,
 0.0052490234375,
 -0.033599853515625,
 0.0019931793212890625,
 -0.005802154541015625,
 -0.0067291259765625,
 -0.00399017333984375,
...]
output2[0].outputs.embedding:
[0.0109100341796875,
 -0.01035308837890625,
 0.00524139404296875,
 -0.033599853515625,
 0.001972198486328125,
 -0.005802154541015625,
 -0.006740570068359375,
 -0.003993988037109375,
 -0.00960540771484375,
 -0.00354766845703125,
...]

emb1 和 emb2 之间的差异通常很小。

diff = np.array(emb1) - np.array(emb2)
np.mean(np.abs(diff)), np.std(diff)
(1.1552008800208569e-05, 1.6885600893505396e-05)

你知道导致这种差异的可能原因是什么吗？我认为这可能是@yumingmin88

I don't know what the underlying principle is. Is the cumulative accuracy different due to the different calculation methods of KQV?

[noooop]

@yumingmin88

我们一件事一件事的说：

1. 首先这种随机性来自于使用fp16或者bf16，它的精度真的很低。

科学计算需要用双精度浮点，最差也用单精度浮点，现在使用的是半精度浮点。

这还不够，现在开始使用fp8。

通常 fp8做推理已经认为是无损的，deepseek v3 甚至使用 fp8 训练模型。

所以完全相同的结果是不可能的，只要在一个大数据集上统计意义上精度差不多就行了。

2. 具体随机结果是怎么来的

计算attention时，将多个序列放在一个批次执行时，用 mask 来将不同序列的attention区分开。

但是乘法的精度比较低，所以不是一个序列的中间向量的值也会互相影响

具体的看 Is Flash Attention Stable?

3. 对于解码模型 使用 check_logprobs_close 对比结果是否正确

4. 对于 embedding 模型，对比 Cosine Similarity， 当然绝对值也算吧，1.1552008800208569e-05 已经足够精确了

5. 对于 rerank 模型，我也发现如果不使用 softmax ，误差偶尔会超过 0.01，有时确实不可接受，你试试 加 softmax 效果会好吗

[DarkLight1337]

Here are the scores I got using your script

[Sentence Transformers]
index:0 socre:0.43743935227394104
index:2 socre:0.16435925662517548
index:1 socre:0.14919064939022064
[vLLM]
index:0 socre:0.4374387860298157
index:2 socre:0.16435886919498444
index:1 socre:0.14919109642505646

The results look close enough to me.

[yumingmin88]

@yumingmin88

我们一件事一件事的说：

1. 首先这种随机性来自于使用fp16或者bf16，它的精度真的很低。

科学计算需要用双精度浮点，最差也用单精度浮点，现在使用的是半精度浮点。

这还不够，现在开始使用fp8。

通常 fp8做推理已经认为是无损的，deepseek v3 甚至使用 fp8 训练模型。

所以完全相同的结果是不可能的，只要在一个大数据集上统计意义上精度差不多就行了。

2. 具体随机结果是怎么来的

计算attention时，将多个序列放在一个批次执行时，用 mask 来将不同序列的attention区分开。

但是乘法的精度比较低，所以不是一个序列的中间向量的值也会互相影响

具体的看 Is Flash Attention Stable?

3. 对于解码模型 使用 check_logprobs_close 对比结果是否正确
4. 对于 embedding 模型，对比 Cosine Similarity， 当然绝对值也算吧，1.1552008800208569e-05 已经足够精确了
5. 对于 rerank 模型，我也发现如果不使用 softmax ，误差偶尔会超过 0.01，有时确实不可接受，你试试 加 softmax 效果会好吗

谢谢您的回复，我正在一条条理解，些许精度差异我也认为是正常，也有提前搜过网上别人的解释。
尝试rerank时是发现有个别数据会出现原来在Sentence Transformers排序位置处于top1，在vllm处在top17
我正在尝试按照5看看效果

[yumingmin88]

Here are the scores I got using your script

[Sentence Transformers]
index:0 socre:0.43743935227394104
index:2 socre:0.16435925662517548
index:1 socre:0.14919064939022064
[vLLM]
index:0 socre:0.4374387860298157
index:2 socre:0.16435886919498444
index:1 socre:0.14919109642505646

The results look close enough to me.

Ok, thank you, the index is inconsistent when I run it, I can accept the slight difference in the score, maybe I need to carefully find the problem，The sentence_transformers version is 3.3.1
[Sentence Transformers]
index:1 socre:0.8325262665748596
index:0 socre:0.771981418132782
index:2 socre:0.6860415935516357
[vLLM]
index:0 socre:0.7844522595405579
index:1 socre:0.7098560929298401
index:2 socre:0.7048576474189758

[noooop]

@yumingmin88

如果你想尽可能减少vllm 和 hf 的差距，可以尝试;

1. 所有依赖版本库保持一致
2. 使用 fp32
3. 使用 max_num_seqs = 1

当然这样不能保证结果完全一样，不同的硬件，驱动版本，cuda版本，依赖库版本都可能对结果造成影响

我觉得误差百分之一到千分之一都是可以接受的

[yumingmin88]

Here are the scores I got using your script

[Sentence Transformers]
index:0 socre:0.43743935227394104
index:2 socre:0.16435925662517548
index:1 socre:0.14919064939022064
[vLLM]
index:0 socre:0.4374387860298157
index:2 socre:0.16435886919498444
index:1 socre:0.14919109642505646

The results look close enough to me.

I am even more confused as to why my results are more different from yours on Sentence Transformers, both index and score.
Is it because the T4 graphics card I use is slightly weak? Or is the rerank model used different? I use the rearrangement of model source: https://huggingface.co/BAAI/bge-reranker-large/tree/main

[DarkLight1337]

I used bge-reranker-v2-m3 model for this test as I was unsure which model you used. Let me try again using bge-reranker-large.