Feature request: prompt lookup decoding

[kevinhu]

Prompt lookup decoding (PLD) is a variant of speculative decoding that replaces the draft model with a prefix lookup in the current sequence, resulting in a 2-4x throughput boost for input-grounded tasks like summarization and code modification.

Because PLD doesn't require a secondary model, it might be easier to implement in VLLM?

See https://github.com/apoorvumang/prompt-lookup-decoding for details.

[cadedaniel]

#2188 introduces a framework for verifying proposal tokens. Once it's merged then PLD is not very difficult to add.

[trinhdoduyhungss]

Hello @cadedaniel , thank you and the vLLM team for creating a great library. I did see that vLLM supported speculative decoding but I couldn't find any documentation on how to use this feature, nor prompt-lookup-decoding. Can you give me an example of how to use this feature in a simple way?

#2188 introduces a framework for verifying proposal tokens. Once it's merged then PLD is not very difficult to add.

[ynwang007]

Hi,

Do you have an internal benchmark result? What --ngram-prompt-lookup-max and --num_speculative_tokens and batch size value do you recommend to set to achieve optimized result on summarization task? Thanks!

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!