[Misc] Add Next Edit Prediction (NEP) datasets support in benchmark_serving.py

[dtransposed]

I am currently using benchmark scripts to profile one of the Next Edit Prediction (NEP) models, zeta on their dataset.

This dataset is functionally similar to the existing likaixin/InstructCoder dataset, in the sense that it can be useful for benchmarking speculative decoding models. However, I decided to create a separate HuggingFaceDataset object, since NEP models have their specific structure (history of edits, prefixes, sometimes suffixes, etc). The code should be extensible to other, similar use cases.

• Fix linting and DCO

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mgoin]

We started moving these benchmark scripts internal to the package so we can have CLI commands. Could you either add it here instead or in both places please? https://github.com/vllm-project/vllm/blob/686623c5e7a0ee0c7679c052ced565dd83055709/vllm/benchmarks/datasets.py

I think the current complication is the vllm bench serve command doesn't have datasets hooked up yet, but I think it should be trivial once I added these #16508

[NickLucche]

Thanks for your PR!
Is Zeta the only NEP dataset available and supported as of now?

[dtransposed]

We started moving these benchmark scripts internal to the package so we can have CLI commands. Could you either add it here instead or in both places please? https://github.com/vllm-project/vllm/blob/686623c5e7a0ee0c7679c052ced565dd83055709/vllm/benchmarks/datasets.py

I think the current complication is the vllm bench serve command doesn't have datasets hooked up yet, but I think it should be trivial once I added these #16508

This is great! I was afraid I would need to keep our separate fork of benchmarks and maintain it!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @dtransposed.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[NickLucche]

LGTM!

[dtransposed]

@NickLucche why cannot I land this? Is this because of the buildkite/ci/pr job?

[dtransposed]

@mgoin @NickLucche just a kind reminder, would love to land it.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @dtransposed.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[dtransposed]

@mgoin @NickLucche kind reminder.

[NickLucche]

Sorry for the late reply!
Can you try to sync with main once again? #17677 just landed so we might be able to merge this without forcing (due to failing tests).

[mgoin]

LGTM thanks

[dtransposed]

@NickLucche on it

[NickLucche]

Looks like pre-commit is failing, please give it another run locally when you find the time!

[dtransposed]

Looks like pre-commit is failing, please give it another run locally when you find the time!

(.venv) (base) damian@damian-ml-machine:~/vllm$ pre-commit run --all-files
yapf.....................................................................Passed
ruff.....................................................................Passed
codespell................................................................Passed
isort....................................................................Passed
clang-format.............................................................Passed
PyMarkdown...............................................................Passed
Lint GitHub Actions workflow files.......................................Passed
pip-compile..............................................................Passed
Run mypy for local Python installation...................................Passed
Lint shell scripts.......................................................Passed
Lint PNG exports from excalidraw.........................................Passed
Check SPDX headers.......................................................Passed
Check for spaces in all filenames........................................Passed
Update Dockerfile dependency graph.......................................Passed
Suggestion...............................................................Passed
- hook id: suggestion
- duration: 0s

:(
Maybe I can just rerun failed jobs (if its possible at some point)

[dtransposed]

@mgoin the failing test seems unrelated ("works locally on my machine" heh), would you mind force landing it?

[mgoin]

Just had to retry the precommit, seems it failed to download the base docker image

[ekagra-ranjan]

Hi folks! @mgoin @NickLucche - I noticed that we have benchmark_dataset.py and datasets.py. What is the difference bw these 2? I am trying to add a dataset so do I need to add to one of them or both of them?

Got the answer from this: #16839 (review)