Skip to content

add SLA information into comparison graph for vLLM Benchmark Suite #25525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Oct 23, 2025
Merged

add SLA information into comparison graph for vLLM Benchmark Suite #25525

merged 16 commits into from
Oct 23, 2025

Conversation

@louie-tsai
Copy link
Contributor

@louie-tsai louie-tsai commented Sep 24, 2025 •
edited by github-actions bot
Loading

Purpose

For better understand how TTFT and TPOT meet the defined SLA.
This PR label the related SLA in the diagram and highlight numbers meet SLA in green in the table.
it makes users easy to find the numbers meet the SLA.

also dump vllm environment information into vllm_env.txt

Test Plan

test manually

Test Result

with SLA TTFT 3000ms and TPOT 150 ms.
image

image
Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@mergify mergify bot added ci/build performance Performance-related issues labels Sep 24, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Sep 24, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds SLA information to the vLLM benchmark comparison graphs and tables. The changes introduce new helper functions to draw limit lines on plots and highlight values in tables that meet the SLA. New command-line arguments are added to specify SLA thresholds. My review focuses on the implementation details of these new features. I've found a couple of issues: one related to file handling that could cause problems in CI, and another related to redundant code and logic in data sorting which affects correctness and maintainability. Please see my detailed comments.

@louie-tsai louie-tsai force-pushed the SLA_Graph branch 2 times, most recently from e515299 to 162bf05 Compare October 2, 2025 06:05
@mergify
Copy link

mergify bot commented Oct 7, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @louie-tsai.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Oct 7, 2025
@louie-tsai louie-tsai force-pushed the SLA_Graph branch 2 times, most recently from 1bb8bb5 to d6d0079 Compare October 9, 2025 22:34
@mergify mergify bot removed the needs-rebase label Oct 9, 2025
@louie-tsai louie-tsai force-pushed the SLA_Graph branch 4 times, most recently from 4edceae to 7868429 Compare October 13, 2025 15:39
@louie-tsai
Copy link
Contributor Author

louie-tsai commented Oct 14, 2025

Random Dataset
128x128
128x2048
2048x128

@louie-tsai louie-tsai force-pushed the SLA_Graph branch 4 times, most recently from 3074ace to ba76dd7 Compare October 22, 2025 04:25
@bigPYJ1151
Copy link
Member

bigPYJ1151 commented Oct 22, 2025

Hi @louie-tsai looks like the DCO check needs fix.

@louie-tsai
quick fix
80a6fed 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
seperate model into different files
89d54aa 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
add SLA lines
1cd2a6f 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
highlight cell within SLA
e1bd425 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
sorted by x axis value
2500b6e 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
reduce TTFT SLA
28a696d 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
change to compare p99
b06aab2 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
make p99/median both available for latency
f80f0de 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
louie-tsai and others added 8 commits October 22, 2025 23:26
@louie-tsai @gemini-code-assist
Update .buildkite/nightly-benchmarks/scripts/compare-json-results.py
0555169 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Louie Tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
keep only 2 decimial
598edcb 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
fix for pre-commit
5c4b657 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
make the latency ratio >1
1bd8e3c 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
Add vllm collect-env
ed7d3b2 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>

Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
add TP4 test cases according to findings from AWS benchmarking
e2aaa8d 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
change serving-test-cpu.json for R8i.24xlarge
92d187b 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai
add two more use cases according to discussions
d831470 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@bigPYJ1151 bigPYJ1151 enabled auto-merge (squash) October 23, 2025 06:41
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 23, 2025
@bigPYJ1151 bigPYJ1151 merged commit 3b7bdf9 into vllm-project:main Oct 23, 2025
19 checks passed
usberkeley pushed a commit to usberkeley/vllm that referenced this pull request Oct 23, 2025
@louie-tsai @gemini-code-assist @usberkeley
add SLA information into comparison graph for vLLM Benchmark Suite (v…
cb7be55 
…llm-project#25525)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
albertoperdomo2 pushed a commit to albertoperdomo2/vllm that referenced this pull request Oct 23, 2025
@louie-tsai @gemini-code-assist @albertoperdomo2
add SLA information into comparison graph for vLLM Benchmark Suite (v…
53d8c63 
…llm-project#25525)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
845473182 pushed a commit to raindaywhu/vllm that referenced this pull request Oct 24, 2025
Merge branch 'step_forward' of https://github.com/raindaywhu/vllm int…
6f4dbd4 
…o step_forward

* 'step_forward' of https://github.com/raindaywhu/vllm: (148 commits)
  [Model] Add MoE support for NemotronH (vllm-project#25863)
  [Metrics] [KVConnector] Add connector prefix cache hit rate stats (vllm-project#26245)
  [CI] Reorganize entrypoints tests (vllm-project#27403)
  add SLA information into comparison graph for vLLM Benchmark Suite (vllm-project#25525)
  [CI/Build] Fix AMD CI: test_cpu_gpu.py (vllm-project#27388)
  [Bugfix] Fix args settings for guided decoding args (vllm-project#27375)
  [CI/Build] Fix Prithvi plugin test (vllm-project#27393)
  [Chore] Remove duplicate `has_` functions in vllm.utils (vllm-project#27372)
  [Model] Add num_cached_tokens for PoolingRequestOutput (vllm-project#27378)
  [V1][spec decode] return logprobs for spec decoding (vllm-project#26060)
  [CORE] Support Prefix Caching with Prompt Embeds (vllm-project#27219)
  [Bugfix][Core] running queue index leakage exception (vllm-project#26754)
  [Bugfix] Fix incorrect kv cache metrics in grafana.json (vllm-project#27133)
  [Bugfix] Fix SLA tuner initialization (vllm-project#27355)
  [Bugfix] Fix deepseek-ocr multi-image inference and add `merge_by_field_config=True` with tensor schema support (vllm-project#27361)
  [MLA] Bump FlashMLA (vllm-project#27354)
  [Chore] Separate out system utilities from vllm.utils (vllm-project#27201)
  [BugFix] bugfix for Flash Attention MLA with full cuda graph IMA following pr-25490 (vllm-project#27128)
  [Feature] publisher default set zmq in kv_event config (vllm-project#26915)
  [Prefix Cache] Use LoRA name for consistent KV-cache block hashing (vllm-project#27211)
  ...
kingsmad pushed a commit to kingsmad/vllm that referenced this pull request Oct 25, 2025
@louie-tsai @gemini-code-assist @kingsmad
add SLA information into comparison graph for vLLM Benchmark Suite (v…
af87cf8 
…llm-project#25525)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025
@louie-tsai @gemini-code-assist @0xrushi
add SLA information into comparison graph for vLLM Benchmark Suite (v…
a1ad387 
…llm-project#25525)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025
@louie-tsai @gemini-code-assist @0xrushi
add SLA information into comparison graph for vLLM Benchmark Suite (v…
26c6299 
…llm-project#25525)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Louie Tsai <louie.tsai@intel.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bigPYJ1151 bigPYJ1151 bigPYJ1151 approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build performance Performance-related issues ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@louie-tsai @bigPYJ1151