[Misc]add coding benchmark for speculative decoding

[CXIAAAAA]

add likaixin/InstructCoder for speculative decoding benchmark throughput

to run instruct coder benchmark:

VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_USE_V1=1  python3 benchmarks/benchmark_throughput.py --dataset-name=hf --dataset-path=likaixin/InstructCoder --model <you hf model> --input-len 1000 --output-len 100 --num-prompts 2048 --async-engine

to run random benchmark:

VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_USE_V1=1  python3 benchmarks/benchmark_throughput.py --dataset-name=random --model <you hf model> --input-len 1000 --output-len 100 --num-prompts 2048 --async-engine

baseline:
Throughput: 37.29 requests/s, 12226.80 total tokens/s, 7457.64 output tokens/s

ngram proposer: --speculative-model "[ngram]" --ngram_prompt_lookup_min 2 --ngram-prompt-lookup-max 5 --num_speculative_tokens 5
Throughput: 35.28 requests/s, 11569.61 total tokens/s, 7056.79 output tokens/s

benchmark_serving:
server: VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct (--speculative-model "[ngram]" --ngram_prompt_lookup_min 2 --ngram-prompt-lookup-max 5 --num_speculative_tokens 5)
client: python3 benchmarks/benchmark_serving.py --model meta-llama/Meta-Llama-3-8B-Instruct --dataset-name hf --dataset-path likaixin/InstructCoder --num-prompts 2048

baseline:

ngram proposer:

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[youran-qi]

Thank you and LGTM!

For your references, here are the statistics about the input / output lengths (number of tokens) in this dataset. Not sure whether you want to change DEFAULT_OUTPUT_LEN accordingly

	avg	min	max
instruction + input	151	15	837
output	179	9	1317

[LiuXiaoxuanPKU]

LGTM, @ywang96 please also take a look in case I miss anything!

Also, I'm just wondering if it's possible to share some simple benchmark results on the instructcoder dataset, really appreciate it!

[CXIAAAAA]

@LiuXiaoxuanPKU i updated the benchmark, but looks like the throughput regressed a bit. I also shared my command do you think we should use low batch to benchmark or this looks good to you?

[ywang96]

LGTM! I left some nits.

Can you also share a example benchmark command with benchmark_serving.py and its result? Thanks!

[CXIAAAAA]

@ywang96 posted the command and the results in the pr description

[CXIAAAAA]

@JenZhao updated the code

[ywang96]

LGTM! Thanks for the contribution!

Can you fix the pre-commit errors so we can merge it?

[JenZhao]

thank you!

[CXIAAAAA]

@ywang96 look like it is ready to merge

[lihuahua123]

The ngram result seems worse, possibly because you did not use --speculative-disable-by-batch-size?

[ekagra-ranjan]

curious why was this hardcoded to train and not paramterized by args.hf_split as done in other cases? If we are eval a model then should it be tested on non train split?

[CXIAAAAA]

it is hardcoded for now, sure we could add it to args.hf_split.

For train/eval split, everyone's method of split might be slightly different, so this is just an example.

[CXIAAAAA]

The ngram result seems worse, possibly because you did not use --speculative-disable-by-batch-size?

The default batch size is too large, normally we measure in low batch size, e.g. 1 or 32 etc.

[ekagra-ranjan]

but looks like the throughput regressed a bit. I also shared my command do you think we should use low batch to benchmark or this looks good to you?

yup, high bs in this bench was the reason why it regressed. This PR has latest speedup: #18971