How to do speculative sampling with vllm?

[lsy643]

When I want to use speculative sampling in the vllm, in the generation step, the number of input tokens of each sequence is larger than one, and then error "an illegal memory access was encountered" is reported.

Can you guys suggest a way to support speculative sampling with vllm?

Thanks a lot

[casper-hansen]

I would love to see speculative decoding implemented in vLLM as it could potentially 2x the throughput. It's not possible at the moment, though.

[cadedaniel]

It is currently not possible. Once #2188 is done, you can enable it.