Fix remaining issues when running on H100 machines

[jerryzh168]

when running pytest test there are many issues, some of them are specific to H100 (need to skip tests), we should follow up and fix them

[jainapurva]

I've a draft PR which fixes a lot of these issues. Didn't merge it as we were not running tests on H100 in CI
#1792