4 GPUs cluster (g5.12xlarge) is doing batch processing faster than 8 GPUs cluster (g5.48xlarge)

[orellavie1212]

Hey,
I am using vllm==0.1.3 with ray >= 2.5.1 on aws sagemaker, with cuda 11.8 and llama2-13b-chat-hf.
When I am using the batch processing function (model.generate , with 3/6/15/30/45/60/105/120 texts as python list, all texts are the same one, only different number of them), the 4 gpus cluster (4 x a10g, ml.g5.12xlarge) is faster in all cases against the 8 gpus cluster (8 x a10g, ml.g5.48xlarge). I wonder maybe the tensor parallel is not working well with more than 4 gpus, or the cuda drivers not utilising them as it should be.
The temp params I am using are:
"temperature": 0.1,
"top_p": 0.75,
"top_k": 40,
"max_tokens": 256,
"frequency_penalty": 1.07,
"use_beam_search": False,
"stop": ["<|endoftext|>",'[INST]', '']

for example, for 105 texts, the 4gpus cluster doing it in 47secs, and the the 8gpus cluster doing it in 59.2 secs. Also, for 60 texts the 4gpus cluster doing it in 27secs, and the the 8gpus cluster doing it in 37.3 secs.
I can provide another examples if needed.
Thanks!

[naed90]

hey, shoot me an email, let's see if I can help:
dean.leitersdorf@gmail.com

[mmaaz60]

Hi @orellavie1212 , @naed90,

Please keep us updated on the development on this issue. Thanks

[yeeeqichen]

@mmaaz60 @orellavie1212 @naed90
Actually, when using TP, there is an extra cost produced by the communication between different GPUs(Because the different gpus aren't working indepently, instead they need to synchronise).

So maybe the problem you're running into is reasonable, because when the gpus grows from 4 -> 8, the cost of communation also doubles(I guess). With more gpus, the single inference operation might takes more time, but it could efficiently handling multi-requests.

[orel1212]

@yeeeqichen I talked with @naed90 about that case (throughput via mem bandwith vs comms. between gpus), it requires profiling I guess and hard to say why.
Tried with TS of 2 and got results not far from 4 ( more faster than half speed of 4)...

[batindfa]

@mmaaz60 @orellavie1212 @naed90
Hi,an error to inference Llama-70b-chat using 16 gpus. The gpus is in one machine. #930
Do you have some ideas?Thanks!

[raihan0824]

I’m experiencing the same issue, I think its weird because I thought the throughput will be linear if I double the gpu cluster. Can anybody explain this clearly? Will be super helpful

[hmellor]

Closing as duplicate of #689