Feat: Router observability (Current QPS, router-side queueing delay, etc) Part 1

[sitloboi2012]

Feature Router observability (Current QPS, router-side queueing delay, etc) Part 1: #78

Due to the complexity and large task, I will split this feature into 2 sub-small PR for easier manage:

• First one will be the setup visualization, update router for localhost logging, etc.
• Second will be connect the Router with Prometheus and visualize the output on Grafana

Description

This PR is to create 4 main things for the first part of the Router Observability update:

• Update vLLM Dashboard Grafana to align with the reference from @YuhanLiu11 combine with add-on. The metrics include

  • Overview of the system:
    • Available vLLM instances
    • Request latency distribution
    • Number of currently healthy vLLM pods
    • Average latency
  • QoS Information:
    • Current QPS
    • Request TTFT distribution
    • Router-side Queueing Delay
    • Average Prefill Length
    • Number of Prefilling Requests
    • Number of Decoding Requests
    • Average Decoding Length
    • Average ITL (Inter-Token Latency)
  • Serving Engine Load:
    • Number of Running Requests
    • Number of Pending Requests
    • GPU KV Usage Percentage
    • GPU KV Cache Hit Rate
    • Number of swapped requests
  • Current Resource Usage:
    • GPU Usage
    • CPU Usage
    • Memory Usage
    • Disk Usage

• Update router.py to tracking information related to the vLLM Dashboard, add on logging for easier tracking

• Update README.md file to include exposing port for Prometheus dashboard as well

• Update utils/install-minikube-cluster.sh to include update minikube context, restart docker when setting up the observability, add on logging for easier interpretation

• Update engine_stats.py and request_stats.py to align with the metrics information

Minor update:

• Update requirements.txt for tests and vllm_router
• Update observe/install.sh from helm install to helm upgrade to solve the problem exist namespace when re-install
• Update base_url in tests/perftest to align with latest v1 endpoint
• Update documentation to all the files for easier readability and understand how the functions work

Next step:

• Connect router.py tracking to Prometheus and visualize it on Grafana dashboard
• Add on metrics calculation in router.py to calculate some customize metrics
• Update README.md file to explain metrics and dashboard information

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

---

PR Checklist (Click to Expand)

Thank you for your contribution to production-stack! Before submitting the pull request, please ensure the PR meets the following criteria. This helps us maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Feat] for new features in the cluster (e.g., autoscaling, disaggregated prefill, etc.).
• [Router] for changes to the vllm_router (e.g., routing algorithm, router observability, etc.).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• Pass all linter checks. Please use pre-commit to format your code. See README.md for installation.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Please include sufficient tests to ensure the change is stay correct and robust. This includes both unit tests and integration tests.

DCO and Signed-off-by

When contributing changes to this project, you must agree to the DCO. Commits must include a Signed-off-by: header which certifies agreement with the terms of the DCO.

Using -s with git commit will automatically add this header.

What to Expect for the Reviews

We aim to address all PRs in a timely manner. If no one reviews your PR within 5 days, please @-mention one of YuhanLiu11
, Shaoting-Feng or ApostaC.

[sitloboi2012]

@ApostaC @gaocegege @YuhanLiu11 please help me review the first part of the PR. I split into 2 small PRs to make it easier to manage because this feature will require quite a amount of files to be changes if do all-in-one. This would also save time to check the work as well 😃

[ApostaC]

@sitloboi2012 Thank Really appreciate your contribution!

@YuhanLiu11 @Shaoting-Feng PTAL 🙏

[YuhanLiu11]

@sitloboi2012 Thanks for your contribution!

Can you fix the pre-commit errors?

[sitloboi2012]

hi @YuhanLiu11 @Shaoting-Feng, I ran pre-commit again, please help me check this PR

[ApostaC]

The PR seems to contain a lot of unrelated stuff (the changes in previous PRs). Is it caused by the rebase?

@sitloboi2012 Can you try fixing this? Maybe cherry-pick your modifications? Or create a new PR based on the diff with the latest main?

[ApostaC]

Btw, this is really great work and we appreciate your contribution! Hopefully the PR problem can be fixed easily.

[sitloboi2012]

yep should I will cherry picking the stuff out to clean the PR

[sitloboi2012]

@ApostaC should be good now, no more redundant file changes 😃

[ApostaC]

Great! Will take a look soon! Thanks!

[YuhanLiu11]

Why using TTFT as decoding length here?

[YuhanLiu11]

Can you change this name to "Queueing Delay"?

[sitloboi2012]

hi @ApostaC @YuhanLiu11 , I updated the PR with some update related to the metrics and other minor fixes to align with the latest PR that we currently have on main for the tests part. I also updated the PR message again as well so please have a look through. Cheers 😄

[YuhanLiu11]

hi @ApostaC @YuhanLiu11 , I updated the PR with some update related to the metrics and other minor fixes to align with the latest PR that we currently have on main for the tests part. I also updated the PR message again as well so please have a look through. Cheers 😄

Thanks for your great contribution! We will take a look soon.

[Shaoting-Feng]

Thanks for your contirbution.

The Prometheus Metrics endpoint in the router looks good to me. However, could you please leave the other parts of the router unchanged? I've noticed many changes, such as comment modifications and variable redefinitions, that seem unrelated to this PR.

[Shaoting-Feng]

Could you please keep these comments?

[Shaoting-Feng]

Could you please keep these comments?

[Shaoting-Feng]

This information has already been logged in the two lines below.

[Shaoting-Feng]

Do we need to delete the conditional statement ?

[Shaoting-Feng]

Could you please keep these comments?

[Shaoting-Feng]

Do we need to delete this conditional statement?

[sitloboi2012]

hi @ApostaC @YuhanLiu11 @Shaoting-Feng , I updated the code again based on your comment, please help me review it again please, cheers team 😄

[YuhanLiu11]

hey @sitloboi2012 , thanks again for your great contribution! Can you also provide some examples showing the code can run and produce expected output?

[sitloboi2012]

yep sure, let me update the code to include your comment, test case and instruction on how to reproduce the work

[gaocegege]

Please resolve the conflicts.

[sitloboi2012]

hi @YuhanLiu11, to test this code, you can use the following file that has been available. Currently the work that connecting between router + prometheus + grafana is WIP and this PR only focusing on setting the stuff that will be use in the next step as I mentioned in the description of the PR.

• To test vllm-dashboard.json, you can first run utils/install-minikube-cluster.sh to setup K8S, Helm Chart, etc. After that run observability/install.sh to install the kube-prom-stack-grafana. After that you should be able to port-forwarding and login to Grafana and upload the vllm-dashboard.json in to view the layout
• To test router.py new metrics info, you can setup the src/tests/fake-openai-service with bash run-server.sh 9000 2. After that, setup the main src/vllm_router/router.py with bash run-router.sh 8000. Finally, trigger the fake request generator with python request_generator.py --qps 1 --num-workers 1. This would help you seeing the metrics are being log out into console log

If run successfully, you should have some output like this in your terminal:

[2025-02-18 08:31:16,075] DEBUG: Got session id: 12196 (routing_logic.py:153:vllm_router.routing_logic)
INFO:     Routing request 30a133a0-ec2f-4c4d-ba4f-e25ce547daa9 to http://localhost:9000 at 1739842276.0759501, process time = 0.0002
INFO:     Started request 30a133a0-ec2f-4c4d-ba4f-e25ce547daa9 for backend http://localhost:9000
INFO:     127.0.0.1:38004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     
==================================================
Model: fake_model_name
Server: http://localhost:9000
 Engine Stats: Running Requests: 19.0, Queued Requests: 0.0, GPU Cache Hit Rate: 0.00
 Request Stats: QPS: 1.00, Avg Latency: -1, Avg ITL: -1, Prefill Requests: 0, Decoding Requests: 29, Swapped Requests: 0, Finished: 0, Uptime: 28.75 sec
--------------------------------------------------
==================================================

[2025-02-18 08:31:16,839] INFO: Scraping metrics from 1 serving engine(s) (engine_stats.py:114:vllm_router.engine_stats)

[YuhanLiu11]

hi @YuhanLiu11, to test this code, you can use the following file that has been available. Currently the work that connecting between router + prometheus + grafana is WIP and this PR only focusing on setting the stuff that will be use in the next step as I mentioned in the description of the PR.

• To test vllm-dashboard.json, you can first run utils/install-minikube-cluster.sh to setup K8S, Helm Chart, etc. After that run observability/install.sh to install the kube-prom-stack-grafana. After that you should be able to port-forwarding and login to Grafana and upload the vllm-dashboard.json in to view the layout
• To test router.py new metrics info, you can setup the src/tests/fake-openai-service with bash run-server.sh 9000 2. After that, setup the main src/vllm_router/router.py with bash run-router.sh 8000. Finally, trigger the fake request generator with python request_generator.py --qps 1 --num-workers 1. This would help you seeing the metrics are being log out into console log

If run successfully, you should have some output like this in your terminal:

[2025-02-18 08:31:16,075] DEBUG: Got session id: 12196 (routing_logic.py:153:vllm_router.routing_logic)
INFO:     Routing request 30a133a0-ec2f-4c4d-ba4f-e25ce547daa9 to http://localhost:9000 at 1739842276.0759501, process time = 0.0002
INFO:     Started request 30a133a0-ec2f-4c4d-ba4f-e25ce547daa9 for backend http://localhost:9000
INFO:     127.0.0.1:38004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     
==================================================
Model: fake_model_name
Server: http://localhost:9000
 Engine Stats: Running Requests: 19.0, Queued Requests: 0.0, GPU Cache Hit Rate: 0.00
 Request Stats: QPS: 1.00, Avg Latency: -1, Avg ITL: -1, Prefill Requests: 0, Decoding Requests: 29, Swapped Requests: 0, Finished: 0, Uptime: 28.75 sec
--------------------------------------------------
==================================================

[2025-02-18 08:31:16,839] INFO: Scraping metrics from 1 serving engine(s) (engine_stats.py:114:vllm_router.engine_stats)

Thanks! Will take a look soon.

[YuhanLiu11]

hey @sitloboi2012 I'm able to follow the instructions you gave and get the the expected output.

Can you fix the pre-commit issues? (sorry for going back-and-forth for the tests..) and then I can merge your PR.

Thanks again!

[sitloboi2012]

hi @YuhanLiu11 just updated, merged and ran pre-commit for formatting and linting, please help me merge and close this PR 😄 thank you team

[gaocegege]

The test Functionality test for helm chart / Multiple-Models (pull_request) is flaky. I rerun it several times to pass it. Now I think it is ready to merge. /cc @YuhanLiu11

[YuhanLiu11]

LGTM now. Thanks!