[Question] Why does using multiple gpus lead to slower performance?

[nutmilk10]

When I set tensor_parallel_size to > 1, the wall time increases though everything else is down. Am I doing something wrong in my setup where using multiple gpus is actually slower than using one?

vllm = LLM(
    model="mosaicml/mpt-7b-instruct", 
    trust_remote_code=True,
    dtype="float16",
    tensor_parallel_size=1,
    gpu_memory_utilization=.95,
)

CPU times: user 3.66 s, sys: 262 ms, total: 3.93 s
Wall time: 1.11 s

vllm = LLM(
    model="mosaicml/mpt-7b-instruct", 
    trust_remote_code=True,
    dtype="float16",
    tensor_parallel_size=2,
    gpu_memory_utilization=.95,
)

CPU times: user 65.5 ms, sys: 32.2 ms, total: 97.7 ms
Wall time: 1.27 s

[Cyberes]

I'm facing the same issue. Seems like there isn't much discussion going on so I'm going to link a bunch of issues together to hopefully get the ball rolling.
#42
#689
#935
#732
#367

[sleepwalker2017]

I also got the same problem !

[hmellor]

The cost of inter-GPU communication outweighs the parallelisation of the compute for smaller models

[njhill]

There has also been significant reduction in the TP overhead since this issue was opened. Would suggest trying the same comparison with the latest version, and also with #3466.

[Deno-V]

i also notice this problem. Use one card i take 15 mins a round. While use 4 card, my time consumption increases 30%.