Consider optimizing the API server

[imoneoi]

Consider optimizing the FastAPI/OpenAI API server in vLLM as the server is widely used and seems to have a lot of overhead. On 1xA100 Llama 13B, the LLM class reaches 90~100% GPU utilization, while the API server can only utilize 50%

Related: #459

[imoneoi]

Also, it would be nice to add a throughput benchmark for the API server

[WoosukKwon]

Hi @imoneoi, thanks for letting us know the issue. We've observed some slowdown when using the API server, but didn't know the slowdown is such significant. Will investigate it.

[hmellor]

@WoosukKwon has this investigation happened yet? If yes, is there any discussion/issue/PR you can link to?

[DarkLight1337]

Closing as this has been addressed by moving online server to another process.