多gpus如何使用？

[xxm1668]

[nkfnn]

model = LLM(model=base_model, tensor_parallel_size=2)，两个gpu

[xxm1668]

(Worker pid=2861953) [W socket.cpp:601] [c10d] The IPv6 network addresses of (internal_head, 19352) cannot be retrieved (gai error: -3 - Temporary failure in name resolution).
(Worker pid=2861953) [W socket.cpp:601] [c10d] The IPv6 network addresses of (internal_head, 19352) cannot be retrieved (gai error: -3 - Temporary failure in name resolution). [repeated 10x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
(Worker pid=2861953) [W socket.cpp:601] [c10d] The IPv6 network addresses of (internal_head, 19352) cannot be retrieved (gai error: -3 - Temporary failure in name resolution). [repeated 10x across cluster]

[xxm1668]

export NCCL_IGNORE_DISABLED_P2P=1，可以解决；但有个问题哈，为啥多卡比单卡运行还慢呢？

[nkfnn]

export NCCL_IGNORE_DISABLED_P2P=1，可以解决；但有个问题哈，为啥多卡比单卡运行还慢呢？

问下，您使用vllm后快了多少？我是用WizardCoder-15B-V1.0模型，使用vllm比hf快了一倍，但是网上都宣传快了27倍，我怀疑是不是我哪里设置没有到位。

[Sanster]

export NCCL_IGNORE_DISABLED_P2P=1，可以解决；但有个问题哈，为啥多卡比单卡运行还慢呢？

问下，您使用vllm后快了多少？我是用WizardCoder-15B-V1.0模型，使用vllm比hf快了一倍，但是网上都宣传快了27倍，我怀疑是不是我哪里设置没有到位。

某个 issue 里看到过，27 倍应该是 batch 推理的加速效果

[nkfnn]

哦哦，好的，谢谢

[xxm1668]

我快了27倍，2卡反而慢了，差不多快了14倍，不知道多卡为啥慢了

[xxm1668]

[gesanqiu]

Make sure you have already know the theory and goal of TP.
Usually, TP is use to solve the bottleneck of memory, for small size model, there is no need to use TP, multi-instances is better than use TP. Because when you use TP to a small model, you will meet the computing bottleneck of the GPU card itself. Considering of the communication cost, maybe you can only get 40-50% improvement when you use 2-GPUs(comparing to 1-GPU).

[xxm1668]

thanks，i understand

[nkfnn]

我快了27倍，2卡反而慢了，差不多快了14倍，不知道多卡为啥慢了

时间缩短了27倍嘛？可以给下您使用的模型，和hf的时间，以及缩短后的时间是多少吗？

[xxm1668]

图片不是有吗？

[guozhiyao]

我快了27倍，2卡反而慢了，差不多快了14倍，不知道多卡为啥慢了

@xxm1668
请问下，你两卡是单独推理，还是弄成tp形式推理的呢？

[zlh1992]

batch

batch 推理什么意思呀？

[Kevinddddddd]

Make sure you have already know the theory and goal of TP. Usually, TP is use to solve the bottleneck of memory, for small size model, there is no need to use TP, multi-instances is better than use TP. Because when you use TP to a small model, you will meet the computing bottleneck of the GPU card itself. Considering of the communication cost, maybe you can only get 40-50% improvement when you use 2-GPUs(comparing to 1-GPU).

Hi, I have a question. Do you know how to set the number of instance in vllms?