Refine start bench

[yiliu30]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS ABOVE HAVE BEEN CONSIDERED.

Purpose

Test Plan

Test Result

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

[gemini-code-assist]

Summary of Changes

Hello @yiliu30, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on refining the benchmarking process for vLLM, primarily through extensive updates to the start_vllm.sh script. The changes aim to improve the flexibility and scope of performance evaluations by adjusting VLLM runtime parameters, expanding the set of evaluated tasks, and updating model paths. Concurrently, the PR significantly refactors the project's dependency management by removing explicit torch and related library versions from various requirement files, indicating a move towards a more decoupled or externally managed installation of these core components.

Highlights

• Benchmarking Script Refinement: The primary benchmarking script start_vllm.sh has been significantly updated to refine the execution of vLLM benchmarks. This includes adjusting default batch sizes, modifying VLLM environment variables for memory utilization and bucketing, and expanding the range of evaluation tasks performed.
• Dependency Management Overhaul: The project's dependency management has been streamlined across multiple requirements/*.txt files and pyproject.toml. Explicit torch and related torchvision, torchaudio, and other hardware-specific (CPU, CUDA, ROCm, XPU, Neuron, TPU) dependencies have been removed, suggesting a shift towards a more flexible or external installation strategy for these core libraries.
• Expanded Evaluation Tasks: The start_vllm.sh script now includes expanded evaluation tasks, specifically adding piqa, hellaswag, and mmlu to the lm_eval suite, indicating a broader scope for performance assessment.
• VLLM Configuration Optimization: Key VLLM configuration parameters have been adjusted, such as reducing VLLM_GPU_MEMORY_UTILIZATION and increasing CONST_LEN (model length/batched tokens), alongside changes to prompt bucketing and prefill sequence handling, to potentially optimize resource usage and performance.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request refines a benchmark script and updates dependencies. The script changes include adding new configurations, modifying existing ones, and adding more evaluation tasks. The dependency changes remove pinned versions of torch and related packages. My feedback focuses on improving the shell script's maintainability, correctness, and resource management.

[gemini-code-assist]

The batch size for the hellaswag evaluation is hardcoded to 32. This is inconsistent with the other tasks, which use the ${batch_size} variable (set to 64 at the top of the script). This might be unintentional and could lead to misleading benchmark results. Consider using the ${batch_size} variable for consistency.

Suggested change

	--batch_size 32 \
	--batch_size ${batch_size} \

[gemini-code-assist]

The server cleanup logic, including calculating the total benchmark time and killing the vLLM server process, has been commented out. This will cause the vLLM server to remain running after the script completes, which is a resource leak. If this is not intended for debugging, this cleanup logic should be restored.

Suggested change

	# end_time=$(date +%s)
	# echo "Benchmark completed in $((end_time - start_time)) seconds"
	# # Clean up
	# echo "Stopping vLLM server"
	# kill ${pid}
	# echo "Script execution completed"
	# sleep 10
	end_time=$(date +%s)
	echo "Benchmark completed in $((end_time - start_time)) seconds"
	# Clean up
	echo "Stopping vLLM server"
	kill ${pid}
	echo "Script execution completed"
	sleep 10

[gemini-code-assist]

The script assigns values to model_path, v2_model_path, mxfp4_model_path, and nvfp4_model_path multiple times. The last assignment for each variable overwrites the previous ones. This pattern can be confusing and error-prone. To improve clarity and maintainability, it's better to have a single active assignment for each variable and comment out the alternatives. This makes it explicit which path is being used.

[gemini-code-assist]

The unset commands for bucketing environment variables have been commented out. This can lead to non-hermetic script execution, as variables from a previous run might persist and affect the current execution's behavior in unexpected ways. It's safer to explicitly unset these variables to ensure a clean environment for each run.

Suggested change

	#unset VLLM_PROMPT_BS_BUCKET_MIN VLLM_PROMPT_BS_BUCKET_STEP VLLM_PROMPT_BS_BUCKET_MAX
	#unset VLLM_PROMPT_SEQ_BUCKET_MIN VLLM_PROMPT_SEQ_BUCKET_STEP VLLM_PROMPT_SEQ_BUCKET_MAX
	#unset VLLM_DECODE_BS_BUCKET_MIN VLLM_DECODE_BS_BUCKET_STEP VLLM_DECODE_BS_BUCKET_MAX
	#unset VLLM_DECODE_BLOCK_BUCKET_MIN VLLM_DECODE_BLOCK_BUCKET_STEP VLLM_DECODE_BLOCK_BUCKET_MAX
	unset VLLM_PROMPT_BS_BUCKET_MIN VLLM_PROMPT_BS_BUCKET_STEP VLLM_PROMPT_BS_BUCKET_MAX
	unset VLLM_PROMPT_SEQ_BUCKET_MIN VLLM_PROMPT_SEQ_BUCKET_STEP VLLM_PROMPT_SEQ_BUCKET_MAX
	unset VLLM_DECODE_BS_BUCKET_MIN VLLM_DECODE_BS_BUCKET_STEP VLLM_DECODE_BS_BUCKET_MAX
	unset VLLM_DECODE_BLOCK_BUCKET_MIN VLLM_DECODE_BLOCK_BUCKET_STEP VLLM_DECODE_BLOCK_BUCKET_MAX

[gemini-code-assist]

The --enforce-eager option has a trailing space. While this works in the current context, it's a bit unclean and could cause issues if the CMD variable is used in a context where word splitting is handled differently. It's best practice to remove trailing whitespace from string appends for better code style and to prevent potential bugs.

Suggested change

	CMD="$CMD --enforce-eager "
	CMD="$CMD --enforce-eager"