[llm.serving] Fix using uni executor when world size == 1 #50849
Conversation
Signed-off-by: Gene Su <e870252314@gmail.com>
|
@kouroshHakha this should fix the tp=1 throwing |
| # uni processing executor when world_size is 1. This is a bug in vllm 0.7.2 and | ||
| # is fixed by https://github.com/vllm-project/vllm/pull/12934 which is shipped | ||
| # with vllm 0.7.3. | ||
| if engine_config.parallel_config.world_size == 1: |
There was a problem hiding this comment.
maybe check the version? we specify version constraint as >=0.7.2 not ==0.7.2, so user could be using either 0.7.2, 0.7.3 or even some future version.
There was a problem hiding this comment.
nevermind, I guess world_size is the golden condition to use here. :)
maybe describe it a bit clearer in the comment on how the vllm version is related to world_size.
There was a problem hiding this comment.
This is specifically for 0.7.2. it's fixed in 0.7.3, but vllm pinned ray to 2.40.0 so that's not gonna work at least when ray 2.43.0 comes out
There was a problem hiding this comment.
why don't we always force using RayDistributedExecutor @GeneDer ?
There was a problem hiding this comment.
isn't that the case for any num_worker > 1?
There was a problem hiding this comment.
@kouroshHakha Doesn't that hurt performance based on our previous investigation?
There was a problem hiding this comment.
Not sure about the performance part, but we been using ray executor since when this is private unless user specific an executor to override
There was a problem hiding this comment.
so our prev investigation for tp=2 shows that using MQEngine is the reason for perf boost and not necessarily RayDistribuedExecutor. In tp=1 it might have different performance profile anyways. but since RayDistribuedExecutor handles these placement groups stuff internally well I think that has the most well-defined integration with ray serve. So using that seems more reasonable right now.
Signed-off-by: Gene Su <e870252314@gmail.com>
…t#50849) Signed-off-by: Gene Su <e870252314@gmail.com>
Why are these changes needed?
When using vllm 0.7.2 with world size 1 (tp=1 and pp=1), vllm will force to use
UniProcExecutorand causeNo CUDA GPUs are availableerror. This PR forces to usesRayDistributedExecutorfor any case so it will always be compatible with Ray.Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.