[V1] [Performance Benchmark] Benchmark the performance of Speculative Decoding

[LiuXiaoxuanPKU]

1. Let's start with ngram, can you collect both latency and throughput numbers on ShareGPT dataset on H100 and one low end GPU?
2. If the numbers from 1 is not expected, could you run some profiling to understand the performance bottleneck.
3. Get more performance numbers on other datasets.

[wwl2755]

Hi,

I'm really interested to work on this. But I only have access to A100s. Does the number on them useful for this? Thank you!

[LiuXiaoxuanPKU]

@wwl2755 Yeah, A100 is good for the first step. Before that, could you create a doc and list the settings you plan to benchmark, happy to take a look.

[wwl2755]

Sure. I will check through the settings and prepare a doc within the next few days.

Could you assign this to me? Thanks! @LiuXiaoxuanPKU

[LiuXiaoxuanPKU]

Keep in mind, vllm already has some predefined benchmark scripts/dataset such as shareGPT and instructcode (https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_dataset.py#L725), we can start from there.

[wwl2755]

Got it! Thank you!

[wwl2755]

@LiuXiaoxuanPKU Hi, I drafted a doc describing the settings and metrics we would like to use and observe during the benchmarking. I would be grateful if you could leave some suggestion and comments on this. Thank you!

Link: https://docs.google.com/document/d/1SbAnLNfCp04lHLJ_cF22IYc_StJ_U3jRRUSc3dQTxO0/edit?usp=sharing

[LiuXiaoxuanPKU]

Hi @wwl2755 , I just left some comments, let me know if you have any questions, otherwise go ahead! One thing I want to point out, let's benchmark v1's performance now by export VLLM_USE_V1=1 on the server side. Thanks!

[wwl2755]

@LiuXiaoxuanPKU Sure. I have addressed the comments on the doc. PTAL. And I will add the results in the doc and notify you when there is some progress. Thanks!

[wwl2755]

Hi @LiuXiaoxuanPKU, I have finished the benchmarking on the shareGPT dataset and have some initial results. If you have time, could you please take a look to make sure it is on the right track? Thank you!

[wwl2755]

And another issue is: if we switch to the v0.8.2, it does not contain the supporting code for InstructCoder. It seems that part of code was included three days ago in #15303.

[LiuXiaoxuanPKU]

Hi @wwl2755 , thanks for the great work! Yeah the results on ShareGPT mostly makes sense, could you try instructcoder as well? maybe just checkout to a newer version that includes that modification, really appreciate it!