Scope for assisted generation?

[creatorrr]

It seems that assisted generation can further reduce sampling latency. Is there scope for adding support for that in vllm?

Assisted generation docs

[zhuohan123]

Hi! This direction is interesting, but might not be an immediate TODO for vLLM. Added this to the roadmap to keep track.

[creatorrr]

Yeah that makes sense :)

[Rifur13]

Is anybody working on this? I can pick up this task.

[tjtanaa]

Found a github project that claims to be 1.6x faster than Medusa and is loseless.
https://github.com/SafeAILab/EAGLE

[hmellor]

@zhuohan123 will this be added to the new roadmap?

[njhill]

Assisted generation is another name for speculative decoding, the implementation of which is in progress (in fact there are a number of PRs implementing this!)

[hmellor]

Oh I see, I was aware of those PRs but I didn't know this feature request was referring to them. Thanks for clarifying :)

[creatorrr]

Closing this since it’s already in the works in #3250 etc :)