About Concurrency and Stability

[gdw439]

Feature request

Hello! I recently came across this popular OpenAI-compatible inference framework and found it very interesting. I'd like to know more about its concurrency and stability—specifically, how it compares to vLLM.

Motivation

project feature

Your contribution

no

[michaelfeil]

https://www.baseten.co/blog/how-we-built-bei-high-throughput-embedding-inference/#performance-benefits-from-baseten-infrastructure Answer: It depends. It has much more features arond embeddings than vllm.