[Bug]: Different GPU architecture generate different evaluation results

[Lynnzake]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

A800: 0.644
H200: 0.592

I ran an evaluation with H200 and A800, but it get the different evaluation results. I have set seed and cuda deterministic and torch deterministic and other things to control the reproductivity. I now can assure the evaluation results can be the same on same GPU, but it differs when i use another type of GPU. How to solve this issue, and why this happened.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[syedia1]

The output of collect_env.py is missing in the post. Please add that so complete about the environment (hardware, software versions) is available.
If possible, also mention the evaluation being run. (with code/links)