Running multiple cards in parallel is slower(nearly twice) than a single card

[Amanda-Barbara]

Hi, when I am running four A100 with parameter tensor_parallel_size is 4 in parallel, I found that the speed is slower(nearly twice) than a single card. can you explain what causes this and how to solve it. Thank you.

[dutsc]

I use 3090 and had the same problem as you.

[hmellor]

Closing as duplicate of #689