Feat: Router observability (Current QPS, router-side queueing delay, etc)

[sitloboi2012]

This issue dedicated to discuss about the feature:

(P1) Router observability (Current QPS, router-side queueing delay, number of pending / prefilling / decoding requests, average prefill / decoding length, etc)

[sitloboi2012]

@gaocegege @ApostaC did you guy think of any specific layout for the dashboard yet or for now let's just dump them in first and then think about the layout later, I'm currently splitting out the metrics into 3 main groups:

• Core vLLM Metrics: “Available vLLM instances”, “Request latency distribution”. “Request TTFT distribution”

• Operational Metrics: Number of Running Request, GPU KV Usage Percentage, Number of Pending Request, GPU KV cache hit rate

• Router Observability Metrics: “Current QPS”, “Router‐side Queueing Delay”, “Average Prefill Length”, “Number of Prefilling Requests”, “Number of Decoding Requests”, “Average Decoding Length”

[YuhanLiu11]

@sitloboi2012 This looks good!

Just for your reference, below are our earlier design on this:

• Overview of the system: Number of currently healthy vLLM pods, Number of requests that are processed or queuing, Average latency.
• QoS information: Timeseries of average QPS, Average TTFT, Average ITL
• Serving engine load: Timeseries of GPU KV cache usage, Number of running requests, Number of queuing requests, Number of swapped requests
• Current resource usage: Timeseries of GPU, CPU, Memory and Disk usage

(cc @ApostaC )

[sitloboi2012]

nice, I think yours @YuhanLiu11 makes more sense, mine was like guessing around based on the usage and using ChatGPT to get some suggestions 😆
I will update again based on these references, thanks for your input, appreciate it 👍