[Performance]: The performance of version 0.6.3 is weaker than that of version 0.6.2 in stress testing.

[skylee-01]

Proposal to improve performance

The performance of version 0.6.3 is weaker than that of version 0.6.2 in stress testing.
Scenario: Agent
Stress Testing Data: Input 50 tokens, output 20 tokens.

Version 0.6.3: 22 QPS, with a significant drop after reaching the maximum value.

Version 0.6.2: 24 QPS, with available performance reaching over 90% at 36 QPS. The decline after reaching the maximum value is gradual.

Under the same conditions, comparing version 0.6.3 with 0.6.2, it was found that the prefill time for version 0.6.3 is about 13ms longer per instance than version 0.6.2. The main reason is a pause of 200-300 microseconds between two blocks.
Conditions for obtaining the data: Using batch_size=20, offline use of LLMEngine with 50 iterations to get the data.

https://github.com/skylee-01/experimental_data/blob/main/nsys_vllm_062.nsys-rep

https://github.com/skylee-01/experimental_data/blob/main/nsys_vllm_063.nsys-rep

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[youkaichao]

you can follow https://docs.vllm.ai/en/latest/getting_started/installation.html#install-the-latest-code to biset all the commits, and find the commit to blame.

and if you can come up with a fix, without affecting the performance of irrelevant usecases, that would be better.

[vrdn-23]

Is this related to #9764

[skylee-01]

Is this related to #9764这与#9764 相关吗？

Thank you for your reply, I will verify it later.