[Bug]: Numerics of Embedding Models

[robertgshaw2-redhat]

Your current environment

vLLM CI (L4)

🐛 Describe the bug

We have been seeing some numerical differences in embedding models and have had to increase the tolerance in the CI recently (though locally on H100 machines we have not seen any issues)

e.g. #22844

It would be great if someone could look into the source of these numerics and see if there is a recommended attention backend that can resolve the problem or if there is some other fundamental problem that we have with embedding models.

Thanks!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

cc @noooop

[DarkLight1337]

cc @maxdebayser @LucasWilkinson as well

[ccs96307]

If it’s okay, I can start by reproducing the tests on different architectures and checking where the discrepancies occur.

[DarkLight1337]

Sure, anyone can help!

[huachenheli]

I think @luccafong is looking into general numerical differences debugging.

[noooop]

Since #22878 Pooling models mteb test uses enforce_eager, mteb_test_embed_models has become less (never) flaky.

I'm not 100% sure that flaky is caused by torch.compile. It is very likely caused by it.

• x-axis has 19 models sorted by diff, so the overall graph roughly goes from lower left to upper right
• y-axis daily statistics for each model's diff compared to st fp32
• y-axis range is set to the threshold MTEB_EMBED_TOL = 1e-4
• diff from different models on the same day are connected into a line

I'll reset the threshold back to MTEB_EMBED_TOL = 2e-4 next month. (The previous was 1e-4, perhaps 2e-4 is a more suitable tolerance.
I'll reset the threshold back to MTEB_EMBED_TOL = 1e-4 next month. The part with a negative diff indicates that the default dtype of vllm is higher than sentence-transformers fp32 in mteb test, (Is it not entirely impossible, right?) , so we don't need to worry too much about this part.

---

I am not familiar with torch.compile, need experts to investigate further.