[RFC]: Scale the API server across multiple CPUs

[russellb]

Motivation.

Currently, the API server runs in a single process, utilizing a single CPU for its work. As GPUs continue to get faster, it is important that we scale the API server to ensure that it is able to process requests fast enough to keep GPU resources fully utilized.

Proposed Change.

From a high level, this proposal is to move from the API server being a single process to being a configurable pool of processes to ensure that a single CPU for the apiserver will not become a bottleneck in server utilization.

Design notes: https://docs.google.com/document/d/1Y2S011RKYkFKtrcz_MuEqEf3cRXORNGsVvMHCaqqc-k/edit?tab=t.0

Feedback Period.

No response

CC List.

@robertgshaw2-redhat @njhill

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

cc @WoosukKwon @simon-mo

[njhill]

Thanks @russellb, I'd already been looking into this a bit

Some ideas:

• Try out uvicorn --workers option ... this is the recommended way I think for ASGI/asyncio apps rather than using gunicorn directly
  • The following changes would be needed but hopefully it could be a fairly quick experiment:
    • Coordinating startup of the engine core process so that it's only done once (doesn't look like there's a simple way to identify a leader but a lockfile type approach might work)
    • Switch to use a ZMQ MPSC pattern
• Start experimenting with Python 3.13t (so we can just use multiple threads rather than multiple processes)
  • We can hopefully switch to 3.13 soon anyhow since it's now supported by pytorch 2.6.
  • 3.13t is not yet supported by pytorch but we could possibly try to use it just for the front-end process, or at least experiment with that in preparation for pytorch 2.7.

[russellb]

Thanks @russellb, I'd already been looking into this a bit

Cool, I'm happy to work with you on it! Or to let you run with it if you were already deep into it.

Some ideas:

• Try out uvicorn --workers option ... this is the recommended way I think for ASGI/asyncio apps rather than using gunicorn directly

  • The following changes would be needed but hopefully it could be a fairly quick experiment:

    • Coordinating startup of the engine core process so that it's only done once (doesn't look like there's a simple way to identify a leader but a lockfile type approach might work)
    • Switch to use a ZMQ MPSC pattern

Thanks for the recommendation. I'll take a look.

• Start experimenting with Python 3.13t (so we can just use multiple threads rather than multiple processes)

  • We can hopefully switch to 3.13 soon anyhow since it's now supported by pytorch 2.6.
  • 3.13t is not yet supported by pytorch but we could possibly try to use it just for the front-end process, or at least experiment with that in preparation for pytorch 2.7.

My hesitation here is that it's not a full solution until 3.13 became our minimum supported Python version. Even if we can use that, I think we need something for older Python before 3.13 will be widespread.

[njhill]

Sure, let's do it!

My hesitation here is that it's not a full solution until 3.13 became our minimum supported Python version. Even if we can use that, I think we need something for older Python before 3.13 will be widespread.

I think this is less important for the api server which is where this would be most applicable, since it would typically be containerized anyhow.

[WoosukKwon]

My hesitation here is that it's not a full solution until 3.13 became our minimum supported Python version. Even if we can use that, I think we need something for older Python before 3.13 will be widespread.

+1 on this. I think relying on Python 3.13 should be our last resort.

[WoosukKwon]

Maybe it's good to start with a few benchmarks and diagnosis?

I agree with @robertgshaw2-redhat that the current API server would be good enough for text-only models. However, I believe the current API server is not scalable enough to handle image/audio/video inputs at high QPS. It'd be nice if we can verify this hypothesis first and set up some target benchmarks (e.g., the new Qwen2.5-VL model with video inputs).

[russellb]

I've started capturing design notes in this doc: https://docs.google.com/document/d/1Y2S011RKYkFKtrcz_MuEqEf3cRXORNGsVvMHCaqqc-k/edit?tab=t.0

[gaby]

@russellb @njhill How do you fix this for vLLM running in Docker? I noticed my vLLM instances are only using 1 CPU core, when my server has over 32 cores.

Ideally uvicorn workers should be configurable via vllm serve, although I'm not sure if serve uses uvicorn.

[gaby]

Apparently you can use WEB_CONCURRENCY as noted here for --workers: https://www.uvicorn.org/deployment/#running-from-the-command-line

Question is, does this break anything in vLLM?

[njhill]

@gaby I'm working on this at the moment, hopefully will have a working PR this week. It's being done in conjunction with the multi-engine capability for data-parallel.

Apparently you can use WEB_CONCURRENCY as noted here for --workers: https://www.uvicorn.org/deployment/#running-from-the-command-line

This won't work due to how uvicorn is integrated in vllm, the front-end servers need to share one or more back-end engines.